Automatic code summarization, also known as code comment generation, has been proven beneficial for developers to understand better and maintain software projects. However, few research works have investigated the robustness of such models. Robustness requires that the model sustains the quality of the output summaries in the presence of perturbations to the inputs. In this paper, we provide an in-depth study of the robustness of code summarization models. We propose CREATE (Code summaRization modEl’s Adversarial aTtackEr), an approach for performing adversarial attacks against the model. This approach can generate adversarial samples to mislead the model and explore its robustness while ensuring these samples are compilable and semantically similar. We attack mainstream code summarization models with a large-scale available Java dataset to evaluate the effectiveness and efficiency of our approach. The experimental results indicate that CREATE’s attack effectiveness and efficiency surpasses other baselines, causing a decrease in the quality of generated comments by at least 40%. Furthermore, we investigate the magnitude of perturbation caused by CREATE during adversarial attacks, and the results show that the similarity between the adversarial samples generated by CREATE and the input code is approximately 0.8, demonstrating that it induces more minor perturbations compared to other baselines. Finally, we utilize CREATE for adversarial training of the model. Through experimentation, this approach indeed effectively enhances the model’s robustness.