Assessing the Readability of ChatGPT Code Snippet Recommendations: A Comparative Study
To the best of our knowledge, this paper was among the first full research papers worldwide to empirically assess the code readability of Large Language Models (LLMs), such as ChatGPT. At the time of its acceptance (June 2023), empirical evidence on the quality of LLM-generated code was still scarce, and discussions were largely based on anecdotal observations. The main contribution of the paper was to demonstrate that code snippets generated by ChatGPT were often highly didactic, presenting fewer readability warnings when compared to human-written code recommended on Stack Overflow (specially in language standards such as camelCase in Java). In addition, the generated code showed strong semantic adherence to the original programming questions provided by users from 80 countries. At the same time, the study provided a balanced analysis by identifying important limitations of LLMs at that stage. In particular, we observed that the models frequently failed to recommend solutions based on more recent versions of the Java language, such as modern APIs and language features. This limitation was attributed to the broader availability of training data from earlier Java versions, highlighting the impact of training data distribution on the quality of generated code. Over the past two years, this line of research has evolved substantially. Our subsequent studies expanded the evaluation to a broader set of LLMs and moved beyond code recommendation, investigating the ability of LLMs to refactor less readable code into more readable versions based on explicit instructions provided in the input.