Exploratory Study on LLMs for Test Amplification in Solidity
Large Language Models (LLMs) are an emerging technology that has changed many of our daily activities. In software engineering, we have seen current research on applying LLMs to elicit requirements, generate code, create tests, and perform code reviews, among others. Test amplification is a process to improve a test suite by adding more test cases that increase a specific measured target (e.g., code coverage). In this paper, we conducted an exploratory study to perform test amplification on Solidity contracts by using LLMs. First, we performed a pilot experiment on three different styled prompts to verify which would create better tests. Second, we conducted another pilot on five different LLMs (GPT 4o, GPT O3 Mini, Gemini 2.0 Flash, Claude 3.5, and Claude 3.7 Sonnet) to verify their code coverage outcomes when creating more test cases. Finally, in our main experiment, we used the best-performing prompt and three LLMs from our pilot experiments to amplify the tests in 113 Solidity contract files. Our results show that Claude 3.7 achieved the highest code-coverage metrics, but also generated many more test cases than the other LLMs.
Tue 17 MarDisplayed time zone: Athens change
09:00 - 10:30 | |||
09:00 25mTalk | Ethereum Layer Two Client Similarity: Geth Workshops & Tutorials | ||
09:25 25mTalk | The Silence of the Comments: Patterns and Pitfalls in Smart Contract Code Documentation Workshops & Tutorials Ermanno Sannini University of Sannio, Italy, Lucia Simeone University of Sannio, Corrado Visaggio University of Foggia, Italy, Andrea Di Sorbo University of Sannio | ||
09:50 25mTalk | Exploratory Study on LLMs for Test Amplification in Solidity Workshops & Tutorials Jorden van Handenhoven University of Antwerp, Billy Vanhove University of Antwerp, Mutlu Beyazıt University of Antwerp and Flanders Make vzw, Onur Kilincceker University of Antwerp and Flanders Make vzw, Henrique Rocha Loyola University Maryland, USA, Serge Demeyer University of Antwerp and Flanders Make vzw | ||
10:15 15mTalk | Fine-Tuning and Semantic Prompt Enrichment for LLM-Based Smart Contract Vulnerability Detection Workshops & Tutorials Francesco Salzano University of Molise, Marco Guglielmi University of Molise, Simone Scalabrino University of Molise, Rocco Oliveto University of Molise, Remo Pareschi University of Molise | ||