ISSTA 2022
Mon 18 - Fri 22 July 2022 Online

Testing is a promising way to gain trust in a learned action policy π, in particular if π is a neural network. A "bug'' in this context constitutes undesirable or fatal policy behavior, e.g., satisfying a failure condition. But how do we distinguish whether such behavior is due to bad policy decisions, or whether it is actually unavoidable under the given circumstances? This requires knowledge about optimal solutions, which defeats the scalability of testing. Related problems occur in software testing when the correct program output is not known.

Metamorphic testing addresses this issue through metamorphic relations, specifying how a given change to the input should affect the output, thus providing an oracle for the correct output. Yet, how do we obtain such metamorphic relations for action policies? Here, we show that the well explored concept of relaxations in the Artificial Intelligence community can serve this purpose. In particular, if state s’ is a relaxation of state s, i.e., s’ is easier to solve than s, and π fails on easier s’ but does not fail on harder s, then we know that π contains a bug manifested on s’.

We contribute the first exploration of this idea in the context of failure testing of neural network policies π learned by reinforcement learning in simulated environments. We design fuzzing strategies for test-case generation as well as metamorphic oracles leveraging simple, manually designed relaxations. In experiments on three single-agent games, our technology is able to effectively identify true bugs, i.e., avoidable failures of π, which has not been possible until now.

Wed 20 Jul

Displayed time zone: Seoul change

03:00 - 04:00
Session 1-3: Oracles, Models, and Measurement ATechnical Papers at ISSTA 1
03:00
20m
Talk
Using Pre-trained Language Models to Resolve Textual and Semantic Merge Conflicts (Experience Paper)
Technical Papers
Jialu Zhang Yale University, Todd Mytkowicz Microsoft Research, Mike Kaufman Microsoft Corporation, Ruzica Piskac Yale University, Shuvendu K. Lahiri Microsoft Research
DOI
03:20
20m
Talk
Metamorphic Relations via Relaxations: An Approach to Obtain Oracles for Action-Policy Testing
Technical Papers
Hasan Ferit Eniser MPI-SWS, Timo P. Gros Saarland University, Germany, Valentin Wüstholz ConsenSys, Jörg Hoffmann Saarland University and DFKI, Germany, Maria Christakis MPI-SWS
DOI Pre-print
03:40
20m
Talk
An Extensive Study on Pre-trained Models for Program Understanding and Generation
Technical Papers
Zhengran Zeng Southern University of Science and Technology, Hanzhuo Tan Southern University of Science and Technology, The Hong Kong Polytechnic University, Haotian Zhang , Jing Li The Hong Kong Polytech University, Yuqun Zhang Southern University of Science and Technology, Lingming Zhang University of Illinois at Urbana-Champaign
DOI

Fri 22 Jul

Displayed time zone: Seoul change

16:40 - 17:40
Session 3-11: Oracles, Models, and Measurement CTechnical Papers at ISSTA 1
16:40
20m
Talk
An Extensive Study on Pre-trained Models for Program Understanding and Generation
Technical Papers
Zhengran Zeng Southern University of Science and Technology, Hanzhuo Tan Southern University of Science and Technology, The Hong Kong Polytechnic University, Haotian Zhang , Jing Li The Hong Kong Polytech University, Yuqun Zhang Southern University of Science and Technology, Lingming Zhang University of Illinois at Urbana-Champaign
DOI
17:00
20m
Talk
Metamorphic Relations via Relaxations: An Approach to Obtain Oracles for Action-Policy Testing
Technical Papers
Hasan Ferit Eniser MPI-SWS, Timo P. Gros Saarland University, Germany, Valentin Wüstholz ConsenSys, Jörg Hoffmann Saarland University and DFKI, Germany, Maria Christakis MPI-SWS
DOI Pre-print
17:20
20m
Talk
TELL: Log Level Suggestions via Modeling Multi-level Code Block Information
Technical Papers
Jiahao Liu National University of Singapore, Jun Zeng National University of Singapore, Xiang Wang University of Science and Technology of China, Kaihang Ji National University of Singapore, Zhenkai Liang National University of Singapore
DOI