Machine Learning (ML) is widely adopted in modern software systems, including safety-critical domains such as autonomous cars, medical diagnosis, and aircraft collision avoidance systems. Thus, it is crucial to rigorously test such applications to ensure high dependability. However, standard notions of software quality and reliability become irrelevant when considering ML systems, due to their non-deterministic nature and the lack of a transparent understanding of the models’ semantics. ML is also expected to revolutionize software development. Indeed, ML is being applied for devising novel program analysis and software testing techniques related to malware detection, bug-finding, and type-checking.
DeepTest 2024 will combine academia and industry in a quest for well-founded practical solutions. The aim is to bring together an international group of researchers and practitioners with both ML and SE backgrounds to discuss their research, share datasets, and generally help the field to build momentum. The workshop will consist of invited talks, presentations based on research paper submissions, and one or more panel discussions, where all participants are invited to share their insights and ideas.
Sat 20 AprDisplayed time zone: Lisbon change
09:00 - 10:30 | Opening + TutorialsDeepTest at Eugénio de Andrade Chair(s): Nicolás Cardozo Universidad de los Andes | ||
09:00 30mDay opening | Opening DeepTest | ||
09:30 30mTutorial | Tutorial: TestSpark DeepTest | ||
10:00 30mTutorial | Tutorial: OpenSBT DeepTest Lev Sorokin fortiss GmbH | Technische Universität München Link to publication Authorizer link Pre-print | ||
10:30 - 11:00 | |||
10:30 30mCoffee break | Break Catering | ||
11:00 - 12:30 | Research TalksDeepTest at Eugénio de Andrade Chair(s): Matteo Biagiola Università della Svizzera italiana | ||
11:00 30mPaper | Data vs. Model Machine Learning Fairness Testing: An Empirical Study DeepTest Arumoy Shome Delft University of Technology, Luís Cruz Delft University of Technology, Arie van Deursen Delft University of Technology Pre-print | ||
11:30 30mPaper | Guiding the Search Towards Failure-Inducing Test Inputs Using Support Vector Machines DeepTest Lev Sorokin fortiss GmbH | Technische Universität München, Niklas Kerscher Technische Universität München | Ludwig-Maximilians-Universität München Pre-print | ||
12:00 30mPaper | A Framework for Including Uncertainty in Robustness Evaluation of Bayesian Neural Network Classifiers DeepTest Wasim Essbai Technische Universität Wien, Andrea Bombarda University of Bergamo, Silvia Bonfanti University of Bergamo, Angelo Gargantini University of Bergamo Pre-print | ||
12:30 - 14:00 | |||
12:30 90mLunch | Lunch Catering | ||
14:00 - 15:30 | Keynote + Invited TalkDeepTest at Eugénio de Andrade Chair(s): Foutse Khomh École Polytechnique de Montréal | ||
14:00 45mKeynote | Mobile Application Testing with Large Language Models: Landscape and Vision DeepTest Chunyang Chen Technical University of Munich (TUM) | ||
14:45 45mTalk | A Controlled Experiment of Different Code Representations for Learning-Based Program Repair DeepTest Marjane Namavar , Noor Nashid University of British Columbia, Ali Mesbah University of British Columbia (UBC) Pre-print | ||
15:30 - 16:00 | |||
15:30 30mCoffee break | Break Catering | ||
16:00 - 17:30 | Research Talks + ClosingDeepTest at Eugénio de Andrade Chair(s): Andrea Stocco Technical University of Munich, fortiss | ||
16:00 30mPaper | More is Not Always Better: Exploring Early Repair of DNNs DeepTest Andrei Mancu Technical University of Munich, Thomas Laurent Lero@Trinity College Dublin, Franz Rieger Max Planck Institute for Biological Intelligence and Technical University of Munich, Paolo Arcaini National Institute of Informatics
, Fuyuki Ishikawa National Institute of Informatics, Daniel Rueckert Pre-print | ||
16:30 30mPaper | Federated Repair of Deep Neural Networks DeepTest Davide Li Calsi Politecnico di Milano, Thomas Laurent Lero@Trinity College Dublin, Paolo Arcaini National Institute of Informatics
, Fuyuki Ishikawa National Institute of Informatics Pre-print | ||
17:00 30mDay closing | Closing DeepTest | ||
Accepted Papers
Call for Papers
DeepTest is an interdisciplinary workshop targeting research at the intersection of software engineering and deep learning. This workshop will explore issues related to: - Deep Learning applied to Software Engineering (DL4SE) - Software Engineering applied to Deep Learning (SE4DL)
Although the main focus is on Deep Learning, we also encourage submissions that are more broadly related to Machine Learning, as well as submissions related to (Deep) Reinforcement Learning.
Topics of Interest
We welcome submissions introducing technology (i.e., frameworks, libraries, program analyses and tool evaluation) for testing DL-based applications, and DL-based solutions to solve open research problems (e.g., what is a bug in a DL/RL model). Relevant topics include, but are not limited to:
- High-quality benchmarks for evaluating DL/RL approaches
- Surveys and case studies using DL/RL technology
- Techniques to aid interpretable DL/RL techniques
- Techniques to improve the design of reliable DL/RL models
- DL/RL-aided software development approaches
- DL/RL for fault prediction, localization and repair
- Fuzzing DL/RL systems
- Metamorphic testing as software quality assurance
- Fault Localization and Anomaly Detection
- Use of DL for analyzing natural language-like artefacts such as code, or user reviews
- DL/RL techniques to support automated software testing
- DL/RL to aid program comprehension, program transformation, and program generation
- Safety and security of DL/RL based systems
- New approaches to estimate and measure uncertainty in DL/RL models
Types of Submissions
We accept two types of submissions:
- Full research papers up to 8-page papers (including references) describing original and unpublished results related to the workshop topics;
- Short papers up to 4-page papers (including references) describing preliminary work, new insights in previous work, or demonstrations of testing-related tools and prototypes.
Authors of the previous EMSE special issues associated with DeepTest will also have the possibility to present their work. All submissions must conform to the ICSE 2024 formatting instructions. All submissions must be in PDF. The page limit is strict.
Submissions should use the official “ACM Primary Article Template”, as can be obtained from the ACM Proceedings Template page. LaTeX users should use the sigconf option, as well as the review (to produce line numbers for easy reference by the reviewers). To that end, the following LaTeX code can be placed at the start of the LaTeX document: \documentclass[sigconf,review]{acmart} \acmConference[ICSE 2024]{46th International Conference on Software Engineering}{April 2024}{Lisbon, Portugal}
DeepTest 2024 will employ a double-blind review process. Thus, no submission may reveal its authors’ identities. The authors must make every effort to honor the double-blind review process. In particular, the authors’ names must be omitted from the submission, and references to their prior work should be in the third person.
The official publication date of accepted papers is the date the proceedings are made available in the ACM or IEEE Digital Libraries. This date may be up to two weeks prior to the first day of ICSE 2024. The official publication date affects the deadline for any patent filings related to published work. Purchases of additional pages in the proceedings are not allowed.
If you have any questions or wonder whether your submission is in scope, please do not hesitate to contact the organizers.
Important Dates
- Paper Submission: December 7, 2023 (AoE)
- Acceptance Notification: January 11, 2023 (AoE)
- Camera Ready: January 25, 2024 (AoE)
Submission System
https://easychair.org/my/conference?conf=deeptest2024
Special Issue
Authors of DeepTest 2024 papers are encouraged to submit revised, extended versions of their manuscripts for the special issue in the Empirical Software Engineering (EMSE) journal, edited by Springer (details will follow). The call is also open to non-DeepTest 2024 authors.
Keynote
Mobile Application Testing with Large Language Models: Landscape and Vision
Prof. Chunyang Chen, Technical University of Munich
Abstract
Mobile apps are now indispensable for people’s daily life. To ensure the app quality, automated GUI testing is widely explored for locating bugs. However, there are many issues with current GUI testing tools including low activity coverage, excessive overhead, and missing issues of app usability (e.g., GUI aesthetics or animation) and accessibility (e.g., to the aged and disabled like the blind). The emergence of powerful Large Language Models (LLM) brings an opportunity to overcome these GUI testing issues. In this talk, he is going to introduce his latest works on different aspects of mobile app testing such as boosting GUI testing coverage, testing case generation, and automated visual bug replay, by leveraging LLM methods including GPT-3/4, and ChatGPT. In addition to the academic publications mentioned above, he will also share a landscape of existing works in using LLM in software testing and share the potential future directions in this field.
Bio
Dr Chunyang Chen is a full professor in the School of Computation, Information and Technology, Technical University of Munich, Germany. His main research interest lies in automated software engineering, especially data-driven mobile app development. Besides, he is also interested in Human-Computer Interaction and software security. He has published 100+ research papers in top venues such as ICSE, FSE, ASE, CHI, CSCW with extensive collaboration with industry, including Google, Microsoft, and Meta. His research has won awards including ACM SIGSOFT Early Career Researcher Award, Facebook Research Award, four ACM SIGSOFT Distinguished Paper Awards (ICSE’23/21/20, ASE’18), and multiple best paper/demo awards.
| Sat 20 Apr 2024 Workshop date |
| Thu 25 Jan 2024 Camera Ready |
| Thu 11 Jan 2024 Acceptance Notification |
| Thu 14 Dec 2023 Paper Submission |
