Less Effort, More Productivity: Lessons Learned from Developing Millions of Lines of Code with Large Language Model
With the emergence of large language models (LLMs), developers can contribute code more effectively and efficiently by utilizing LLMs to generate code snippets. Thus, a variety of downstream development tasks, such as feature implementation and program repair, can benefit from this emerging technique. However, to the best of our knowledge, existing applications with LLMs are limited to code generation within standalone modules. It remains unclear whether LLMs can be scaled to repository-level generation on large-scale software systems with more than a million lines of code. In this paper, we report our experience with developing a large-scale software system of 1.52 million lines of code (MLoC) with the assistance of state-of-the-art LLMs. We proposed an Expert-AI hybrid development approach. Combined with LLMs, we improve the existing design workflow, including prototyping, diagnosis, and refactoring steps, that can automatically finish coding tasks. This approach combines the strength of complex architectural design from human experts and the automatic code generation from LLMs. We conducted a controlled experiment on PicoScenes, an industrial system that has evolved over a decade with an MLoC codebase. The results suggest that developers save time implementing features by 68.3%. Additionally, we observed a 28.2% reduction in mean cyclomatic complexity. Consequently, the defect rate decreases by 50% and CPU load decreases by 25%. These findings suggest developers can focus on architectural design and spend less effort on implementing trivial modules. Our experience can improve the productivity of expert-led architectural design and LLM-led module implementation to support the maintenance of software systems with MLoC.