Towards Robustness of Deep Program Processing Models -- Detection, Estimation and Enhancement
Deep learning (DL) has recently been widely applied to diverse source code processing tasks in the software engineering (SE) community, which achieves competitive performance (e.g., accuracy). However, the robustness, which requires the model to produce consistent decisions given minorly perturbed code inputs, still lacks systematic investigation as an important quality indicator. This article initiates an early step and proposes a framework CARROT for robustness detection, measurement, and enhancement of DL models for source code processing. We first propose an optimization-based attack technique CARROTA to generate valid adversarial source code examples effectively and efficiently. Based on this, we define the robustness metrics and propose robustness measurement toolkit CARROTM, which employs the worst-case performance approximation under the allowable perturbations. We further propose to improve the robustness of the DL models by adversarial training (CARROTT) with our proposed attack techniques. Our in-depth evaluations on three source code processing tasks (i.e., functionality classification, code clone detection, defect prediction) containing more than 3 million lines of code and the classic or SOTA DL models, including GRU, LSTM, ASTNN, LSCNN, TBCNN, CodeBERT, and CDLH, demonstrate the usefulness of our techniques for ❶ effective and efficient adversarial example detection, ❷ tight robustness estimation, and ❸ effective robustness enhancement.
Slides_PPTX (ASE23_JF_CARROT_zhanghz.pptx) | 5.72MiB |
Slides_PDF (ASE23_JF_CARROT_zhanghz_20230913160949.pdf) | 1.50MiB |
Wed 13 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 12:00 | Program AnalysisResearch Papers / Tool Demonstrations / NIER Track / Journal-first Papers at Room D Chair(s): Domenico Bianculli University of Luxembourg | ||
10:30 12mTalk | An Integrated Program Analysis Framework for Graduate Courses in Programming Languages and Software Engineering Research Papers Prantik Chatterjee Indian Institute Of Technology Kanpur and MathWorks, Pankaj Kumar Kalita IIT Kanpur, Sumit Lahiri Indian Institute Of Technology Kanpur, Sujit Kumar Muduli IIT Kanpur, Vishal Singh Indian Institute of Technology Kanpur, Gourav Takhar Indian Institute of Technology Kanpur, Subhajit Roy IIT Kanpur | ||
10:42 12mTalk | Two Birds with One Stone: Multi-Derivation for Fast Context-Free Language Reachability Analysis Research Papers Chenghang Shi SKLP, Institute of Computing Technology, CAS, Haofeng Li , Yulei Sui University of New South Wales, Sydney, Jie Lu SKLP, Institute of Computing Technology, CAS, Lian Li Institute of Computing Technology at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Jingling Xue UNSW Pre-print File Attached | ||
10:54 12mTalk | NRAgo: Solving SMT(NRA) Formulas with Gradient-based Optimization Tool Demonstrations Minghao Liu Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Kunhang Lv Institute of Software, Chinese Academy of Sciences, Pei Huang Stanford University, Rui Han Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Fuqi Jia Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Yu Zhang Institute of Software, Chinese Academy of Sciences, Feifei Ma Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Jian Zhang Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences File Attached | ||
11:06 12mTalk | Live Programming for Finite Model Finders NIER Track Allison Sullivan University of Texas at Arlington Pre-print File Attached | ||
11:18 12mTalk | Towards Robustness of Deep Program Processing Models -- Detection, Estimation and Enhancement Journal-first Papers Huangzhao Zhang Peking University, Zhiyi Fu Peking University, Ge Li Peking University, Lei Ma University of Alberta, Zhehao Zhao Peking University, Hua'an Yang Peking University, Yizhe Sun Peking University, Yang Liu Nanyang Technological University, Zhi Jin Peking University Link to publication DOI File Attached | ||
11:30 12mTalk | Precise Data-Driven Approximation for Program Analysis via FuzzingRecorded talk Research Papers Nikhil Parasaram University College London; ConsenSys Diligence, Earl T. Barr University College London; Google DeepMind, Sergey Mechtaev University College London, Marcel Böhme MPI-SP; Monash University Pre-print Media Attached | ||
11:42 12mTalk | Contrastive Learning for API Aspect AnalysisRecorded talk Research Papers G. M. Shahariar Ahsanullah University of Science and Technology, Tahmid Hasan Bangladesh University of Engineering and Technology, Anindya Iqbal Bangladesh University of Engineering and Technology Dhaka, Bangladesh, Gias Uddin York University, Canada Pre-print Media Attached |