DeepTest 2021

After running as one of the most successful workshops at ICSE 2020, the International Workshop on Testing for Deep Learning and Deep Learning for Testing (DeepTest) returns once more as a co-located workshop at the ACM/IEEE International Conference on Software Engineering (ICSE) in 2021!

DeepTest is a high-quality workshop for research at the intersection of Machine Learning (ML) and software engineering (SE). ML is widely adopted in modern software systems, including safety-critical domains such as autonomous cars, medical diagnosis, and aircraft collision avoidance systems. Thus, it is crucial to rigorously test such applications to ensure high dependability. However, standard notions of software quality and reliability become irrelevant when considering ML systems, due to their non-deterministic nature and the lack of a transparent understanding of the models’ semantics. ML is also expected to revolutionize software development. Indeed, ML is being applied for devising novel program analysis and software testing techniques related to malware detection, fuzzy testing, bug-finding, and type-checking.

The workshop will combine academia and industry in a quest for well-founded practical solutions. The aim is to bring together an international group of researchers and practitioners with both ML and SE backgrounds to discuss their research, share datasets, and generally help the field to build momentum. The workshop will consist of invited talks, presentations based on research paper submissions, and one or more panel discussions, where all participants are invited to share their insights and ideas.

Keynote Speakers

Dr. Amir Ronen (Chief Scientist, SparkBeyond, prev. Stanford University). “Problem Solving Combining Data Science and Web Knowledge”

Abstract: Traditionally, problem-solving using data science is very different from the way we solve problems in everyday life. Both approaches have important advantages. In this talk, I will describe several interrelated efforts to bridge that gap. We will propose web knowledge mining as a way to replace the world knowledge that is often used by people to solve problems. We will then describe several ways to combine web knowledge into an automated data science system and discuss their usefulness and challenges. This is joint work with several colleagues from SparkBeyond including Daniel Carmon, Elad Shaked, and Meir Maor. The talk is mostly high-level. No prior knowledge is assumed.

Biography: Amir Ronen is the Chief Scientist of Spark Beyond. His main interests lie in the border of machine learning and algorithms. He is often fascinated by deep mathematical ideas that have far-reaching practical implications. Amir did his Ph.D. at the Hebrew University under the supervision of Noam Nisan. He was a postdoctoral research fellow at Stanford University and UC Berkeley, a faculty member at the Technion, and a researcher at IBM. Amir received various awards including the Gödel Prize, the Best Paper Prize from the International Joint Conferences Artificial Intelligence (IJCAI) and the Journal of Artificial Intelligence Research (JAIR), and the Wolf prize for Ph.D. Students.

Natalija Gucevska (Facebook London), Prof. Mark Harman (Facebook London, UCL) “Testing Facebook’s WW Simulation System, a Cyber-Cyber Digital Twin of the Facebook WWW Platform”

Abstract: Facebook is building a digital twin of its full platform infrastructure called WW (essentially a realistic simulation, on real WWW infrastructure, that allows it to detect and prevent harmful behaviors on the real platform). WW uses many techniques, including ML, in order to train bots to simulate real user interactions. This talk presents an overview of Facebook’s WW system, its testing (using automated regression, end-to-end and metamorphic testing), and exciting open problems and applications of Software Testing for Smart Simulation systems built using ML.

This talk reports the results of joint work by John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk, Sophia Drossopoulou, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, Simon Lucas, Steve Omohundro, Erik Meijer, Rubmary Rojas, Silvia Sapora and Jie Zhang.

Biography: Natalija Gucevska graduated from the Swiss Institute of Technology in Lausanne (EPFL) with a Master’s degree in Computer Science in 2019. During her master thesis, she worked on detecting outliers in historical financial data in the Swiss bank Swissquote. In 2017, as a member of an interdisciplinary team, she won a gold medal at the 2017 iGEM competition in Synthetic Biology, and the software she developed received a nomination for the best software. Natalija joined Facebook in 2019 as a software engineer where she works on Facebook’s WW Web Enabled Simulation system.

Mark Harman is a full-time Research Scientist at Facebook London, working on Facebook’s Web-Enabled Simulation system WW, together with a London-based Facebook team focussing on AI for scalable software engineering. WW is Facebook’s Cyber-Cyber Digital Twin of its platforms, being built with the long-term aim of measuring, predicting, and optimizing behavior across all Facebook’s platforms. Mark also holds a part-time professorship at UCL and was previously the manager of Facebook’s Sapienz team, which grew out of Majicke, a start-up co-founded by Mark and acquired by Facebook in 2017. The Sapienz tech has been fully deployed as part of Facebook’s overall CI system since 2017 and the Facebook Sapienz continues to develop and extend it. Sapienz has found and helped to fix thousands of bugs before they hit production, on systems of tens of millions of lines of code, used by over 2.6 billion people worldwide every day. In his more purely scientific work, Mark co-founded the field Search-Based Software Engineering (SBSE) and is also known for scientific research on source code analysis, software testing, app store analysis, and empirical software engineering. He received the IEEE Harlan Mills Award and the ACM Outstanding Research Award in 2019 for his work and was awarded a fellowship of the Royal Academy of Engineering in 2020.

Special Issue

Authors of DeepTest 2021 papers are encouraged to submit revised, extended versions of their manuscripts for the special issue Software Testing in the Machine Learning Era of the Empirical Software Engineering (EMSE) journal, edited by Springer. The call is open also to non-DeepTest 2021 authors.

Submission Deadline: December 18th, 2021.

Accepted Papers

Call for Papers

DeepTest is an interdisciplinary workshop targeting research at the intersection of SE and ML. We welcome submissions that investigate:

• how to ensure the quality of ML-based applications, both at a model level and at a system level
• the use of ML to support software engineering tasks, particularly software testing

Relevant topics include, but are not limited to:

Quality

• Quality implication of ML algorithms on large-scale software systems
• Application of classical statistics to ML systems quality
• Training and payload data quality
• Correctness of data abstraction, data trust
• High-quality benchmarks for evaluating ML approaches

Testing and Verification

• Test data synthesis for testing ML systems
• White-box and black-box testing strategies
• ML models for testing programs
• Adversarial machine learning and adversary based learning
• Test coverage
• Vulnerability, sensitivity, and attacks against ML
• Metamorphic testing as software quality assurance
• New abstraction techniques for verification of ML systems
• ML techniques for software verification
• Dev-ops for ML

Fault Localization, Debugging, and Repairing

• Quality Metrics for ML systems, e.g., Correctness, Accuracy, Fairness, Robustness, Explainability
• Sensitivity to data distribution diversity and distribution drift
• Failure explanation and automated debugging techniques
• Runtime monitoring
• Fault Localization and anomaly detection
• Model repairing
• The effect of labeling costs on solution quality (semi-supervised learning)
• ML for fault prediction, localization, and repair
• ML to aid program comprehension, program transformation, and program generation

We accept two types of submissions:

• full research papers up to 8-page papers describing original and unpublished results related to the workshop topics.
• short papers up to 4-page papers describing both preliminary work, new insights in previous work, or demonstrations of testing-related tools and prototypes.

All submissions must conform to the ICSE 2021 formatting instructions. All submissions must be in PDF. The page limit is strict. Submissions must conform to the IEEE formatting instructions IEEE Conference Proceedings Formatting Guidelines (title in 24pt font and full text in 10pt type, LaTeX users must use \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options). DeepTest 2021 will employ a double-blind review process. Thus, no submission may reveal its authors’ identities. The authors must make every effort to honor the double-blind review process. In particular, the authors’ names must be omitted from the submission, and references to their prior work should be in the third person.

If you have any questions or wonder whether your submission is in scope, please do not hesitate to contact the organizers.

Special Issue

Authors of DeepTest 2021 papers are encouraged to submit revised, extended versions of their manuscripts for the special issue Software Testing in the Machine Learning Era of the Empirical Software Engineering (EMSE) journal, edited by Springer. The call is open also to non-DeepTest 2021 authors.

Submission Deadline: December 18th, 2021.

DeepTest 2020

DeepTest 2019

EDSMLS 2020

EDSMLS 2019

EDSMLS 2018