Write a Blog >>
ASE 2021
Mon 15 - Fri 19 November 2021 Australia

This program is tentative and subject to change.

Wed 17 Nov 2021 09:50 - 09:55 at Kangaroo - Learning I
Wed 17 Nov 2021 10:06 - 10:08 at Kangaroo - Tool Demo (2)

Deep learning (DL) training is nondeterministic and such nondeterminism was shown to cause significant variance of model accuracy (up to 10.8%). Such variance may affect the validity of the comparison of newly proposed DL techniques with baselines. To ensure such validity, DL researchers and practitioners must replicate their experiments multiple times with identical settings to quantify the variance of the proposed approaches and baselines. Replicating and measuring DL variances reliably and efficiently is challenging and understudied. We propose a ready-to-deploy framework DEVIATE that (1)measures DL training variance of a DL model with minimal manual efforts, and (2) provides statistical tests of both accuracy and variance. Specifically, DEVIATEautomaticallyanalyzes the DL training code and extracts monitored important metrics (such as accuracy and loss). In addition, DEVIATE performs popular statistical tests and provides users with a report of statistical p-values and effect sizes along with various confidence levels when comparing to selected baselines. We demonstrate the effectiveness of DEVIATE by performing case studies with adversarial training. Specifically, for an adversarial training process that uses the Fast Gradient Signed Method to generate adversarial examples as the training data, DEVIATEmeasures a max difference of accuracy among 8 identical training runs with fixed random seeds to be up to 5.1%.

This program is tentative and subject to change.

Wed 17 Nov

Displayed time zone: Hobart change

09:00 - 10:00
09:00
20m
Talk
DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score
Research Papers
Vincenzo Riccio USI Lugano, Nargiz Humbatova Università della Svizzera Italiana (USI), Gunel Jahangirova USI Lugano, Paolo Tonella USI Lugano
09:20
20m
Talk
Efficient state synchronisation in model-based testing through reinforcement learning
Research Papers
Uraz Cengiz Türker University of Leicester, UK, Robert Hierons University of Sheffield, Mohammad Reza Mousavi King's College London, Ivan Tyukin University of Leicester
09:40
10m
Talk
What do pre-trained code models know about code?
NIER track
Anjan Karmakar Free University of Bozen-Bolzano, Romain Robbes
09:50
5m
Talk
DEVIATE: A Deep Learning Variance Testing Framework
Tool Demonstrations
Viet Hung Pham University of Waterloo, Mijung Kim Purdue University, Lin Tan Purdue University, Yaoliang Yu University of Waterloo, Nachiappan Nagappan Microsoft Research
10:00 - 11:00
10:00
2m
Talk
Shaker: a Tool for Detecting More Flaky Tests Faster
Tool Demonstrations
Marcello Cordeiro Federal University of Pernambuco, Denini Silva Federal University of Pernambuco, Leopoldo Teixeira Federal University of Pernambuco, Breno Miranda Federal University of Pernambuco, Marcelo d'Amorim Federal University of Pernambuco
Link to publication
10:02
2m
Talk
RefactorInsight: Enhancing IDE Representation of Changes in Git with Refactorings Information
Tool Demonstrations
Zarina Kurbatova JetBrains Research, Vladimir Kovalenko JetBrains Research, Ioana Savu Delft University of Technology, Bob Brockbernd Delft University of Technology, Dan Andreescu Delft University of Technology, Matei Anton Delft University of Technology, Roman Venediktov Higher School of Economics, Elena Tikhomirova JetBrains Research, Timofey Bryksin JetBrains Research; HSE University
10:04
2m
Talk
GenTree: Inferring Configuration Interactions using Decision Trees
Tool Demonstrations
KimHao Nguyen University of Nebraska-Lincoln, ThanhVu Nguyen George Mason University
10:06
2m
Talk
DEVIATE: A Deep Learning Variance Testing Framework
Tool Demonstrations
Viet Hung Pham University of Waterloo, Mijung Kim Purdue University, Lin Tan Purdue University, Yaoliang Yu University of Waterloo, Nachiappan Nagappan Microsoft Research
10:08
2m
Talk
FLACK: Localizing Faults in Alloy Models
Tool Demonstrations
Guolong Zheng University of Nebraska Lincoln, ThanhVu Nguyen George Mason University, Simón Gutiérrez Brida University of Rio Cuarto and CONICET, Argentina, Germán Regis Universidad Nacional de Río Cuarto, Marcelo F. Frias Dept. of Software Engineering Instituto Tecnológico de Buenos Aires, Nazareno Aguirre University of Rio Cuarto and CONICET, Argentina, Hamid Bagheri University of Nebraska-Lincoln
10:10
2m
Talk
Scalable Fuzzing of Program Binaries with E9AFL
Tool Demonstrations
Xiang Gao National University of Singapore, Gregory J. Duck National University of Singapore, Abhik Roychoudhury National University of Singapore