LLM-based automated program repair methods have attracted significant attention for their state-of-the-art performance. However, they were primarily evaluated on a few well-known datasets like Defects4J, raising questions about their effectiveness on new datasets. In this study, we evaluate 11 top-performing LLMs on DEFECTS4J-TRANS, a new dataset derived from transforming Defects4J while maintaining fault semantics. Results from experiments on both Defects4J and DEFECTS4J-TRANS show that all studied LLMs have limited generalizability in APR tasks, with average correct and plausible patches decreasing by 49.48% and 42.90%, respectively, on DEFECTS4J-TRANS. Further investigation into incorporating additional repair-relevant information in repair prompts reveals that, although this information signigicantly enhances the LLMs’ capabilities (increasing correct and plausible patches by up to 136.67% and 121.82%, respectively), performance still falls short of their original dataset results. This indicates that prompt engineering alone is insufficient to substantially enhance LLMs’ repair capabilities. According our study, we also offer several recommendations for future research.