Large Language Models (LLMs) represent a leap in artificial intelligence, excelling in tasks using human language(s). Although the main focus of general-purpose LLMs is not code generation, they have shown promising results in the domain. However, the usefulness of LLMs in large-scale software engineering development have not been fully explored yet. In this study we explore the usefulness of LLMs for 214 students working on an academic software engineering project in teams consisting of up to six members. The students were encouraged to integrate LLMs into their development tool-chain.

In this paper, we analyze statistics for the AI generated code, prompts used for code generation and human intervention levels to integrate the code. We also conduct a perception study to gain insights into the perceived usefulness, influencing factors, and future outlook from a student perspective. Our findings suggest that LLMs can play a crucial role in the early stages of software development, especially in generating foundational code structures, and helping with syntax and error debugging. These insights provide us with a framework on how to effectively utilize LLMs as a tool to enhance the productivity of software engineering students, and highlight the necessity of shifting the educational focus toward preparing students for successful human-AI collaboration.