Coverage Isn’t Enough: SBFL-Driven Insights into Manually Created vs. Automatically Generated Tests
The testing phase is an essential part of software development, but manually creating test cases can be time-consuming. Consequently, there is a growing need for more efficient testing methods. To reduce the burden on developers, various automated test generation tools have been developed, and several studies have been conducted to evaluate the effectiveness of the tests they produce. However, most of these studies focus primarily on coverage metrics, and only a few examine how well the tests support fault localization—particularly using artificial faults introduced through mutation testing. In this study, we compare the SBFL (Spectrum-Based Fault Localization) score and code coverage of automatically generated tests with those of manually created tests. The SBFL score indicates how accurately faults can be localized using SBFL techniques. By employing SBFL score as an evaluation metric—an approach rarely used in prior studies on test generation—we aim to provide new insights into the respective strengths and weaknesses of manually created and automatically generated tests. Our experimental results show that automatically generated tests achieve higher branch coverage than manually created tests, but their SBFL score is lower, especially for code with deeply nested structures. These findings offer guidance on how to effectively combine automatically generated and manually created testing approaches.
| Coverage Isn’t Enough: SBFL-Driven Insights into Manually Created vs. Automatically Generated Tests (simizu-s_profes2025_fin.pdf) | 566KiB |
Wed 3 DecDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
09:00 - 11:00 | Testing and Simulation of Software and Cyber-Physical SystemsShort Papers and Posters / Research Papers / Industry Papers at Sala degli Affreschi (Fresco Room) Chair(s): Javier Gonzalez-Huerta Blekinge Institute of Technology | ||
09:00 15mTalk | Coverage Isn’t Enough: SBFL-Driven Insights into Manually Created vs. Automatically Generated Tests Research Papers File Attached | ||
09:15 15mTalk | A Robust LSTM-based Test Selection Method for Self-Driving Cars Research Papers Ali Gullu University of Tartu, Faiz Ali Shah University of Tartu, Estonia, Dietmar Pfahl University of Tartu | ||
09:30 15mTalk | Generative AI in Simulation-Based Test Environments for Large-Scale Cyber-Physical Systems: An Industrial Study Research Papers Masoud Sadrnezhaad Linköping University, José Antonio Hernández López Universidad de Murcia, Torvald Mårtensson Saab AB, Daniel Varro Linköping University / McGill University DOI Pre-print File Attached | ||
09:45 10mTalk | Obtaining Test Data in the Estonian e-Government System: Challenges and Improvement Potential Industry Papers Maj-Annika Tammisto University of Tartu, Rudolf Ramler Software Competence Center Hagenberg (SCCH), Faiz Ali Shah University of Tartu, Estonia, Dietmar Pfahl University of Tartu | ||
09:55 10mTalk | Software Testing Education and Industry Needs - Report from ENACTEST EU Project Industry Papers Mehrdad Saadatmand RISE Research Institutes of Sweden, Muhammad Abbas Khan RISE Research Institutes of Sweden, Beatriz Marín Universitat Politècnica de València, Ana Paiva INESC TEC, Faculty of Engineering, University of Porto, Nele Van Asch AE nv, Graham Moran NEXO QA, Felix Cammaerts KU Leuven, Monique Snoeck Katholieke Universiteit Leuven, Alexandra Mendes Faculty of Engineering, University of Porto & INESC TEC Pre-print | ||
10:05 10mTalk | STRIPID: Simulation Test Ranking and Interactive Performance Inspection for PID controllers Industry Papers Alejandra Duque-Torres Software Competence Center Hagenberg (SCCH) GmbH, Claus Klammer Software Competence Center Hagenberg, Stefan Fischer Software Competence Center Hagenberg File Attached | ||
10:15 7mTalk | Lab Package Development as a Means for Educating Software Engineering Students Short Papers and Posters Eliisabet Kaasik University of Tartu, Faiz Ali Shah University of Tartu, Estonia, Dietmar Pfahl University of Tartu File Attached | ||
10:22 7mTalk | A Model-Driven Engineering Method for the Development of Digital Twins Short Papers and Posters Emilio Carrión Mercadona Tech, Mercadona - PROS – VRAIN Universitat Politècnica de València, Pedro Valderas Universitat Politècnica de València, Oscar Pastor Universitat Politecnica de Valencia | ||
10:29 7mTalk | From Scenario Selection to Simulation: Safety Testing of an Automated Driving System Short Papers and Posters Fauzia Khan University of Tartu, Estonia, Ali Gullu University of Tartu, Hina Anwar University of Tartu, Dietmar Pfahl University of Tartu | ||
10:36 7mTalk | An Application of Program Mutations For Generating Negative Test Scripts Mimicking Human Errors on Web Applications Short Papers and Posters Tomoya Yamashita Ehime University, Hirohisa Aman Ehime University, Sousuke Amasaki Nanzan University, Tomoyuki Yokogawa Okayama Prefectural University, Minoru Kawahara Ehime University | ||
10:43 7mTalk | Ticket-Augmented Just-in-Time Defect Prediction Short Papers and Posters Emanuele Gentili Mbda Italy S.p.a., Daniele La Prova University of Rome "Tor Vergata", Via del Politecnico 1, Rome, Italy, 00132, Falessi Davide University of Rome Tor Vergata | ||