There has been a growing interest in using Large Language Models (LLMs) for code review owing to their proven proficiency in code comprehension. A primary objective of most review tasks is to generate what we call modification-eliciting review comments (MERCs) that explicitly identify modification-required issues in source code. For LLMs to effectively generate MERCs, they often require post-training with a special dataset comprising MERCs, which is, however, not available. As manual annotation of MERCs is too laborious and costly to be a viable solution, the main contribution of this paper is an automated data annotation method MERC-Annot, which can automatically construct a special dataset comprising MERCs (i.e,MERC-dataset) by utilizing LLMs to identify MERCs from existing datasets. Experiments with the commonly used CodeReviewer dataset comprising more than 150K review entries show that MERC-Annot surpasses state-of-the-art methods with impressive performance in identifying MERCs. Validation on a dataset collected from active open-source projects further confirms the robustness and generalizability of MERC-Annot in real-world scenarios. To verify and evaluate the effect of MERC-Annot regarding improving LLMs’ code review ability, we first fine-tune several of the latest LLMs using MERC-dataset to build a new model MERC-Reviewer. We then enhance the model training effect through KTO alignment by feeding the review comments identified as NMERCs (i.e., review comments that do not elicit modifications) to LLMs, resulting in another model MERC-Reviewer-Aligned. The evaluation results show that MERC-Reviewer-Aligned slightly outperforms MERC-Reviewer, while both models have improved significantly compared to the base models in terms of generating MERCs. Human evaluation confirms that both models identify issues more accurately and tend to generate review comments that better describe the issues contained in the code than the base models.