Recent work has focused on using machine learning to automate software engineering processes, such as code completion, code migration, and generating code from natural language description. One of the challenges faced in these tasks is evaluating the quality of the predictions, which is usually done by comparing the prediction to a reference solution. BLEU score has been adopted for programming languages as it can be easily computed for any programming language and even incomplete source code, while enabling fast automated evaluation. However, programming languages are more verbose and have strict syntax when compared to natural languages. This feature causes BLEU to find common n-grams in unrelated programs, which makes distinguishing similar pairs of programs from dissimilar pairs hard. This work presents CrystalBLEU, an evaluation metric based on BLEU, that mitigates the distinguishability problem. Our metric maintains the desirable properties of BLEU, such as handling partial code, applicability to all programming languages, high correlation with human judgement, and efficiency, in addition to reducing the effects of the trivially matched n-grams. We evaluate CrystalBLEU on two datasets from previous work and a new dataset of human-written code. Our results show that CrystalBLEU differentiates similar and unrelated programs better than the original BLEU score and also a variant designed specifically for source code, CodeBLEU.