Learning-based code repair models rely heavily on benchmark datasets for both training and evaluation. CodeXGLUE, one of the most widely used benchmarks in the field, serves as the standard for evaluating many of these models. However, the quality of such benchmarks is often overlooked, which can have a substantial impact on the validity of models evaluation. This case study focuses on the issue of unresolvable bug-fix pairs within CodeXGLUE—pairs that cannot be resolved at the method-level granularity. These pairs are potentially skewing both model evaluation and training outcomes. Given the widespread use of CodeXGLUE, flaws in this benchmark mean that the evaluation of many models could be problematic, raising concerns about the reliability of results across the field. To assess the impact of these unresolvable pairs, we systematically classify and remove them from CodeXGLUE, then use it to reassess the models’ performance. Our findings demonstrate that the presence of unresolvable bug-fix pairs can mislead the evaluation and even result in incorrect ranking of different models performance. Moreover, after removing these pairs from CodeXGLUE, the latest four models show an average performance improvement of 5.3% when evaluated on Defects4J. Ultimately, our case study casts doubt on the reliability of benchmarks with unresolvable pairs for learning-based code repair.