TCSE logo 
 Sigsoft logo
Sustainability badge

Call for papers

The Software Engineering in Practice (SEIP) track of ICSE is the premier venue for practitioners and researchers to discuss insights, innovations, and solutions to concrete software engineering problems. Practical software development relies on excellent software engineering research.

SEIP provides a unique forum for networking, exchanging ideas, fostering innovations, and forging long-term collaborations to address Software Engineering research that impacts directly on practice. It is one of the most prestigious forums in which to publish work in the Software Engineering literature. SEIP will include participants and speakers from both industrial and academic sectors, drawing together researchers and practitioners eager to communicate and share common interests in software engineering. The track will be composed of invited talks, paper presentations, and interactive sessions, all with a strong focus on software practice.

Please note that the ICSE SEIP track DOES NOT require double-anonymous review. Unlike some other ICSE tracks, it is often important for SEIP paper authors and the organizations involved to be visible to the reviewers in order for the reviewers to fully understand the industrial relevance and context.

Submissions

Link to submission: https://icse2025-seip.hotcrp.com/

Papers address industrially-relevant problems through systematic investigations. Each paper should describe a problem of practical importance, explain how the problem was investigated and in what context, and present evidence for the paper’s conclusions. The submission should have technical and empirical soundness. Other aspects that should also be included where appropriate include: discussing why the resolution of the problem is innovative, (cost-) effective, or efficient; providing a concise explanation of the approach, techniques, and methodologies employed; and explaining the insights or best practices that emerged, tools developed, and/or software processes involved.

Papers will appear in the ICSE SEIP Companion proceedings.

Submission Process

  • All submissions must not exceed 10 pages (for full papers) for the main text, inclusive of all figures, tables, and appendices. 2 pages are allowed for references. All submissions must be in PDF.
  • Submissions must strictly conform to the IEEE conference proceedings formatting instructions specified above. Alterations of spacing, font size, and other changes that deviate from the instructions may result in desk rejection without further review.
  • All submissions must conform to the IEEE conference proceedings template, specified in the IEEE Conference Proceedings Formatting Guidelines (title in 24pt font and full text in 10pt type, LaTeX users must use \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options). Submissions must strictly conform to the IEEE conference proceedings formatting instructions specified above . Alterations of spacing, font size, and other changes that deviate from the instructions may result in desk rejection without further review.
  • By submitting to the ICSE SEIP Track, authors acknowledge that they are aware of and agree to be bound by the ACM Policy and Procedures on Plagiarism and the IEEE Plagiarism FAQ. Papers submitted to ICSE 2025 must not have been published elsewhere and must not be under review or submitted for review elsewhere whilst under consideration for ICSE 2025. Contravention of this concurrent submission policy will be deemed a serious breach of scientific ethics, and appropriate action will be taken in all such cases. To check for double submission and plagiarism issues, the chairs reserve the right to (1) share the list of submissions with the PC Chairs of other conferences with overlapping review periods and (2) use external plagiarism detection software, under contract to the ACM or IEEE, to detect violations of these policies.
  • By submitting to the ICSE Research SEIP, authors acknowledge that they conform to the authorship policy of the ACM, and the authorship policy of the IEEE.
  • If the research involves human participants/subjects, the authors must adhere to the ACM Publications Policy on Research Involving Human Participants and Subjects. Upon submitting, authors will declare their compliance with such a policy. Alleged violations of this policy or any ACM Publications Policy will be investigated by ACM and may result in a full retraction of your paper, in addition to other potential penalties, as per ACM Publications Policy.
  • Please ensure that you and your co-authors obtain an ORCID ID, so you can complete the publishing process for your accepted paper. ACM and IEEE have been involved in ORCID and may collect ORCID IDs from all published authors. We are committed to improve author discoverability, ensure proper attribution and contribute to ongoing community efforts around name normalization; your ORCID ID will help in these efforts.

Evaluation

All SEIP submissions will be reviewed by members of the SEIP Program Committee. Submissions will be evaluated based on relevance to industrial practice, the significance of contribution, and the quality of presentation.

By submitting to this track, authors acknowledge that they are aware of and agree to be bound by the ACM Policy and Procedures on Plagiarism and the IEEE Plagiarism FAQ. In particular, papers submitted to ICSE 2025 SEIP must not have been published elsewhere and must not be under review or submitted for review elsewhere while under consideration for ICSE SEIP 2025. Contravention of this concurrent submission policy will be deemed a serious breach of scientific ethics, and appropriate action will be taken in all such cases. To check for double submission and plagiarism issues, the chairs reserve the right to (1) share the list of submissions with the PC Chairs of other conferences with overlapping review periods and (2) use external plagiarism detection software, under contract to the ACM or IEEE, to detect violations of these policies.

By submitting to this track, authors acknowledge that they conform to the authorship policy of the ACM, and the authorship policy of the IEEE.

Important Dates

  • Submission deadline: October 10th, 2024
  • Acceptance notification: December 15th, 2024
  • Camera ready: January 10, 2025

Conference Attendance Expectation

If a submission is accepted, at least one author of the paper is required to register for and attend the conference to present the paper. Virtual attendance will be an option.

Contact

If there are queries regarding the CFP, please contact the ICSE SEIP 2025 chairs.

Dates
Wed 30 Apr 2025
Thu 1 May 2025
Fri 2 May 2025
Tracks
ICSE Demonstrations
ICSE Journal-first Papers
ICSE New Ideas and Emerging Results (NIER)
ICSE Research Track
ICSE SE In Practice (SEIP)
ICSE SE in Society (SEIS)

This program is tentative and subject to change.

You're viewing the program in a time zone which is different from your device's time zone change time zone

Wed 30 Apr

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
Human and Social Process 1SE In Practice (SEIP) / New Ideas and Emerging Results (NIER) / Journal-first Papers at 207
Chair(s): Hausi Müller University of Victoria
11:45
15m
Talk
Identifying Factors Contributing to ``Bad Days'' for Software Developers: A Mixed-Methods Study
SE In Practice (SEIP)
Ike Obi Purdue University, West Lafayette, Jenna L. Butler Microsoft Research, Sankeerti Haniyur Microsoft Corporation, Brian Hassan Microsoft Corporation, Margaret-Anne Storey University of Victoria, Brendan Murphy Microsoft Corporation
12:00
15m
Talk
Time Warp: The Gap Between Developers’ Ideal vs Actual Workweeks in an AI-Driven EraAward Winner
SE In Practice (SEIP)
Sukrit Kumar Georgia Institute of Technology, Drishti Goel Microsoft, Thomas Zimmermann University of California, Irvine, Brian Houck Microsoft Research, B. Ashok Microsoft Research. India, Chetan Bansal Microsoft Research
12:15
15m
Talk
Wearables to measure developer experience at work
SE In Practice (SEIP)
Charlotte Brandebusemeyer Hasso Plattner Institute, University of Potsdam, Tobias Schimmer SAP Labs, Bert Arnrich Hasso Plattner Institute, University of Potsdam
11:00 - 12:30
11:15
15m
Talk
KuiTest: Leveraging Knowledge in the Wild as GUI Testing Oracle for Mobile Apps
SE In Practice (SEIP)
Yongxiang Hu Fudan University, Yu Zhang Meituan, Xuan Wang Fudan University, Yingjie Liu School of Computer Science, Fudan University, Shiyu Guo Meituan, Chaoyi Chen Meituan, Xin Wang Fudan University, Yangfan Zhou Fudan University
11:30
15m
Talk
GUIWatcher: Automatically Detecting GUI Lags by Analyzing Mobile Application Screencasts
SE In Practice (SEIP)
Wei Liu Concordia University, Montreal, Canada, Feng Lin Concordia University, Linqiang Guo Concordia University, Tse-Hsun (Peter) Chen Concordia University, Ahmed E. Hassan Queen’s University
12:00
15m
Talk
Agent for User: Testing Multi-User Interactive Features in TikTok
SE In Practice (SEIP)
Sidong Feng Monash University, Changhao Du Jilin University, huaxiao liu Jilin University, Qingnan Wang Jilin University, Zhengwei Lv ByteDance, Gang Huo ByteDance, Xu Yang ByteDance, Chunyang Chen TU Munich
11:00 - 12:30
AI for Testing and QA 1Research Track / SE In Practice (SEIP) at 214
Chair(s): Jieshan Chen CSIRO's Data61
12:15
15m
Talk
DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production
SE In Practice (SEIP)
Xiaoyun Liang ByteDance, Jingyi Ren ByteDance, Jiayi Qi ByteDance, Chao Peng ByteDance, Bo Jiang Bytedance Network Technology
11:00 - 12:30
SE for AI 1New Ideas and Emerging Results (NIER) / SE In Practice (SEIP) / Research Track at 215
Chair(s): Houari Sahraoui DIRO, Université de Montréal
12:00
15m
Talk
Evaluation of Tools and Frameworks for Machine Learning Model ServingSE for AI
SE In Practice (SEIP)
Niklas Beck Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, Benny Stein Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, Dennis Wegener T-Systems International GmbH, Lennard Helmer Fraunhofer Institute for Intelligent Analysis and Information Systems
12:15
15m
Talk
Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation ModelsSE for AI
SE In Practice (SEIP)
Kirill Vasilevski Huawei Canada, Dayi Lin Centre for Software Excellence, Huawei Canada, Ahmed E. Hassan Queen’s University
15:30 - 16:00
15:30
30m
Poster
FlatD: Protecting Deep Neural Network Program from Reversing Attacks
SE In Practice (SEIP)
Jinquan Zhang The Pennsylvania State University, Zihao Wang Penn State University, Pei Wang Independent Researcher, Rui Zhong Palo Alto Networks, Dinghao Wu Pennsylvania State University
16:00 - 17:30
17:15
15m
Talk
Safe Validation of Pricing Agreements
SE In Practice (SEIP)
John C. Kolesar Yale University, Tancrède Lepoint Amazon Web Services, Martin Schäf Amazon Web Services, Willem Visser Amazon Web Services
16:00 - 17:45
Human and Social 1SE in Society (SEIS) / SE In Practice (SEIP) at 206 plus 208
Chair(s): Yvonne Dittrich IT University of Copenhagen, Denmark
16:00
15m
Talk
Systematizing Inclusive Design in MOSIP: An Experience Report
SE In Practice (SEIP)
Soumiki Chattopadhyay Oregon State University, Amreeta Chatterjee Oregon State University, Puja Agarwal Oregon State University, Bianca Trinkenreich Colorado State University, Swarathmika Kumar MOSIP-IIIT Bangalore, Rohit Ranjan Rai MOSIP-IIIT Bangalore, Resham Chugani MOSIP-IIIT Bangalore, Pragya Kumari MOSIP-IIIT Bangalore, Margaret Burnett Oregon State University, Anita Sarma Oregon State University
16:00 - 17:30
AI for Testing and QA 2Research Track / SE In Practice (SEIP) at 214
Chair(s): Michael Pradel University of Stuttgart
17:15
15m
Talk
Improving Code Performance Using LLMs in Zero-Shot: RAPGen
SE In Practice (SEIP)
Spandan Garg Microsoft Corporation, Roshanak Zilouchian Moghaddam Microsoft, Neel Sundaresan Microsoft
16:00 - 17:30
Analysis 1Research Track / SE In Practice (SEIP) at 215
Chair(s): Antonio Filieri AWS and Imperial College London
17:15
15m
Talk
ArkAnalyzer: The Static Analysis Framework for OpenHarmony
SE In Practice (SEIP)
chenhaonan Beihang University, Daihang Chen Beihang University, Yizhuo Yang Beihang University, Lingyun Xu Huawei, Liang Gao Huawei, Mingyi Zhou Beihang University, Chunming Hu Beihang University, Li Li Beihang University

Thu 1 May

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
12:00
15m
Talk
AI-Assisted SQL Authoring at Industry ScaleSE for AI
SE In Practice (SEIP)
Chandra Sekhar Maddila Meta Platforms, Inc., Negar Ghorbani Meta Platforms Inc., Kosay Jabre Meta Platforms, Inc., Vijayaraghavan Murali Meta Platforms Inc., Edwin Kim Meta Platforms, Inc., Parth Thakkar Meta Platforms, Inc., Nikolay Pavlovich Laptev Meta Platforms, Inc., Olivia Harman Meta Platforms, Inc., Diana Hsu Meta Platforms, Inc., Rui Abreu Meta, Peter C Rigby Meta / Concordia University
12:15
15m
Talk
Automating ML Model Development at ScaleSE for AI
SE In Practice (SEIP)
Kaiyuan Wang Google, Yang Li Google Inc, Junyang Shen Google Inc, Kaikai Sheng Google Inc, Yiwei You Google Inc, Jiaqi Zhang Google Inc, Srikar Ayyalasomayajula Google Inc, Julian Grady Google Inc, Martin Wicke Google Inc
11:00 - 12:30
Analysis 2SE In Practice (SEIP) / Journal-first Papers / Demonstrations at 205
Chair(s): Mahmoud Alfadel University of Calgary
11:15
15m
Talk
Towards Better Static Analysis Bug Reports in the Clang Static Analyzer
SE In Practice (SEIP)
Kristóf Umann Eötvös Loránd University, Faculty of Informatics, Dept. of Programming Languages and Compilers, Zoltán Porkoláb Ericsson
11:45
15m
Talk
LLM Driven Smart Assistant for Data Mapping
SE In Practice (SEIP)
Arihant Bedagkar Tata Consultancy Services, Sayandeep Mitra Tata Consultancy Services, Raveendra Kumar Medicherla TCS Research, Tata Consultancy Services, Ravindra Naik TCS Research, TRDDC, India, Samiran Pal Tata Consultancy Services
12:00
15m
Talk
On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure CategoriesArtifact-AvailableArtifact-FunctionalArtifact-Reusable
SE In Practice (SEIP)
Henri Aïdasso École de technologie supérieure (ÉTS), Francis Bordeleau École de Technologie Supérieure (ETS), Ali Tizghadam TELUS
Pre-print
11:00 - 12:30
Security and Analysis 1Research Track / SE In Practice (SEIP) at 210
11:45
15m
Talk
Detecting Python Malware in the Software Supply Chain with Program AnalysisArtifact-AvailableArtifact-FunctionalArtifact-ReusableSecurity
SE In Practice (SEIP)
Ridwan Salihin Shariffdeen SonarSource SA, Behnaz Hassanshahi Oracle Labs, Australia, Martin Mirchev National University of Singapore, Ali El Husseini National University of Singapore, Abhik Roychoudhury National University of Singapore
11:00 - 12:30
AI for Design and ArchitectureDemonstrations / SE In Practice (SEIP) / Research Track at 211
11:45
15m
Talk
aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing
SE In Practice (SEIP)
Siyuan Jiang , Jia Li Peking University, He Zong aiXcoder, Huanyu Liu Peking University, Hao Zhu Peking University, Shukai Hu aiXcoder, Erlu Li aiXcoder, Jiazheng Ding aiXcoder, Ge Li Peking University
Pre-print
12:00
15m
Talk
Leveraging MLOps: Developing a Sequential Classification System for RFQ Documents in Electrical Engineering
SE In Practice (SEIP)
Claudio Martens Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), Hammam Abdelwahab Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), Katharina Beckh Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), Birgit Kirsch Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), Vishwani Gupta Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), Dennis Wegener Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), Steffen Hoh Schneider Electric
12:15
15m
Talk
On Mitigating Code LLM Hallucinations with API Documentation
SE In Practice (SEIP)
Nihal Jain Amazon Web Services, Robert Kwiatkowski , Baishakhi Ray Columbia University, New York;, Murali Krishna Ramanathan AWS AI Labs, Varun Kumar AWS AI Labs
11:00 - 12:30
11:45
15m
Talk
SEMANTIC CODE FINDER: An Efficient Semantic Search Framework for Large-Scale Codebases
SE In Practice (SEIP)
daeha ryu Innovation Center, Samsung Electronics, Seokjun Ko Samsung Electronics Co., Eunbi Jang Innovation Center, Samsung Electronics, jinyoung park Innovation Center, Samsung Electronics, myunggwan kim Innovation Center, Samsung Electronics, changseo park Innovation Center, Samsung Electronics
12:00
15m
Talk
Time to Retrain? Detecting Concept Drifts in Machine Learning Systems
SE In Practice (SEIP)
Tri Minh-Triet Pham Concordia University, Karthikeyan Premkumar Ericsson, Mohamed Naili Ericsson, Jinqiu Yang Concordia University
12:15
15m
Talk
UML Sequence Diagram Generation: A Multi-Model, Multi-Domain Evaluation
SE In Practice (SEIP)
Chi Xiao Ericsson AB, Daniel Ståhl Ericsson AB, Jan Bosch Chalmers University of Technology
11:00 - 12:30
12:00
15m
Talk
NICE: Non-Functional Requirements Identification, Classification, and Explanation Using Small Language ModelsArtifact-AvailableAward Winner
SE In Practice (SEIP)
Gokul Rejithkumar TCS Research, Preethu Rose Anish TCS Research
13:00 - 13:30
13:00
30m
Poster
FlatD: Protecting Deep Neural Network Program from Reversing Attacks
SE In Practice (SEIP)
Jinquan Zhang The Pennsylvania State University, Zihao Wang Penn State University, Pei Wang Independent Researcher, Rui Zhong Palo Alto Networks, Dinghao Wu Pennsylvania State University
13:00
30m
Talk
Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry
SE In Practice (SEIP)
Martin Obaidi Leibniz Universität Hannover, Nicolas Voß Graphmasters GmbH, Hannah Deters Leibniz University Hannover, Jakob Droste Leibniz Universität Hannover, Marc Herrmann Leibniz Universität Hannover, Jannik Fischbach Netlight Consulting GmbH and fortiss GmbH, Kurt Schneider Leibniz Universität Hannover, Software Engineering Group

Fri 2 May

Displayed time zone: Eastern Time (US & Canada) change

10:30 - 11:00
10:30
30m
Talk
Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry
SE In Practice (SEIP)
Martin Obaidi Leibniz Universität Hannover, Nicolas Voß Graphmasters GmbH, Hannah Deters Leibniz University Hannover, Jakob Droste Leibniz Universität Hannover, Marc Herrmann Leibniz Universität Hannover, Jannik Fischbach Netlight Consulting GmbH and fortiss GmbH, Kurt Schneider Leibniz Universität Hannover, Software Engineering Group
11:00 - 12:30
11:15
15m
Talk
Soft Skills in Software Engineering: Insights from the Trenches
SE In Practice (SEIP)
Sanna Malinen University of Canterbury, Matthias Galster University of Canterbury, Antonija Mitrovic University of Canterbury, New Zealand, Sreedevi Sankara Iyer University of Canterbury, Pasan Peiris University of Canterbury, New Zealand, April Clarke University of Canterbury
11:00 - 12:30
Human and Social using AI 2Research Track / SE In Practice (SEIP) / Demonstrations at 207
11:00
15m
Talk
Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation ModelsArtifact-AvailableArtifact-FunctionalArtifact-Reusable
SE In Practice (SEIP)
Hao Li Queen's University, Cor-Paul Bezemer University of Alberta, Ahmed E. Hassan Queen’s University
11:30
15m
Talk
Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplace
SE In Practice (SEIP)
Jenna L. Butler Microsoft Research, Jina Suh Microsoft Research, Sankeerti Haniyur Microsoft Corporation, Constance Hadley Institute for Work Life
11:45
15m
Talk
Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company
SE In Practice (SEIP)
Guilherme Vaz Pereira School of Technology, PUCRS, Brazil, Victoria Jackson University of California, Irvine, Rafael Prikladnicki School of Technology at PUCRS University, Andre van der Hoek University of California, Irvine, Luciane Fortes Globo, Carolina Araújo Globo, André Coelho Globo, Ligia Chelli Globo, Diego Ramos Globo
Pre-print
12:00
15m
Talk
Human-In-the-Loop Software Development Agents
SE In Practice (SEIP)
Wannita Takerngsaksiri Monash University, Jirat Pasuksmit Atlassian, Patanamon Thongtanunam University of Melbourne, Kla Tantithamthavorn Monash University, Ruixiong Zhang Atlassian, Fan Jiang Atlassian, Jing Li Atlassian, Evan Cook Atlassian, Kun Chen Atlassian, Ming Wu Atlassian
11:00 - 12:30
12:00
15m
Talk
Video Game Procedural Content Generation Through Software Transplantation
SE In Practice (SEIP)
Mar Zamorano López University College London, Daniel Blasco SVIT Research Group. Universidad San Jorge, Carlos Cetina , Federica Sarro University College London
11:00 - 12:30
11:15
15m
Talk
Evaluating Agent-based Program Repair at Google
SE In Practice (SEIP)
Patrick Rondon Google, Renyao Wei Google, José Pablo Cambronero Google, USA, Jürgen Cito TU Wien, Aaron Sun Google, Siddhant Sanyam Google, Michele Tufano Google, Satish Chandra Google, Inc
11:30
15m
Talk
Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and DatasetArtifact-AvailableArtifact-FunctionalArtifact-Reusable
SE In Practice (SEIP)
Mohammad Saiful Islam Toronto Metropolitan University, Toronto, Canada, Mohamed Sami Rakha Toronto Metropolitan University, Toronto, Canada, William Pourmajidi Toronto Metropolitan University, Toronto, Canada, Janakan Sivaloganathan Toronto Metropolitan University, Toronto, Canada, John Steinbacher IBM, Andriy Miranskyy Toronto Metropolitan University (formerly Ryerson University)
Pre-print
11:45
15m
Talk
Crash Report Prioritization for Large-Scale Scheduled Launches
SE In Practice (SEIP)
Nimmi Rashinika Weeraddana University of Waterloo, Sarra Habchi Ubisoft Montréal, Shane McIntosh University of Waterloo
12:00
15m
Talk
LogLM: From Task-based to Instruction-based Automated Log Analysis
SE In Practice (SEIP)
Yilun Liu Huawei co. LTD, Yuhe Ji Huawei co. LTD, Shimin Tao University of Science and Technology of China; Huawei co. LTD, Minggui He Huawei co. LTD, Weibin Meng Huawei co. LTD, Shenglin Zhang Nankai University, Yongqian Sun Nankai University, Yuming Xie Huawei co. LTD, Boxing Chen Huawei Canada, Hao Yang Huawei co. LTD
Pre-print
11:00 - 12:30
AI for Testing and QA 5SE In Practice (SEIP) at 214
11:00
15m
Talk
ASTER: Natural and Multi-language Unit Test Generation with LLMsAward Winner
SE In Practice (SEIP)
Rangeet Pan IBM Research, Myeongsoo Kim Georgia Institute of Technology, Rahul Krishna IBM Research, Raju Pavuluri IBM T.J. Watson Research Center, Saurabh Sinha IBM Research
Pre-print
11:15
15m
Talk
Automated Code Review In Practice
SE In Practice (SEIP)
Umut Cihan Bilkent University, Vahid Haratian Bilkent Univeristy, Arda İçöz Bilkent University, Mert Kaan Gül Beko, Ömercan Devran Beko, Emircan Furkan Bayendur Beko, Baykal Mehmet Ucar Beko, Eray Tüzün Bilkent University
11:30
15m
Talk
CI at Scale: Lean, Green, and Fast
SE In Practice (SEIP)
Dhruva Juloori Uber Technologies, Inc, Zhongpeng Lin Uber Technologies Inc., Matthew Williams Uber Technologies, Inc, Eddy Shin Uber Technologies, Inc, Sonal Mahajan Uber Technologies Inc.
11:45
15m
Talk
Moving Faster and Reducing Risk: Using LLMs in Release DeploymentAward Winner
SE In Practice (SEIP)
Rui Abreu Meta, Vijayaraghavan Murali Meta Platforms Inc., Peter C Rigby Meta / Concordia University, Chandra Sekhar Maddila Meta Platforms, Inc., Weiyan Sun Meta Platforms, Inc., Jun Ge Meta Platforms, Inc., Kaavya Chinniah Meta Platforms, Inc., Audris Mockus The University of Tennessee, Megh Mehta Meta Platforms, Inc., Nachiappan Nagappan Meta Platforms, Inc.
12:00
15m
Talk
Prioritizing Large-scale Natural Language Test Cases at OPPO
SE In Practice (SEIP)
Haoran Xu , Chen Zhi Zhejiang University, Tianyu Xiang Guangdong Oppo Mobile Telecommunications Corp., Ltd., Zixuan Wu Zhejiang University, Gaorong Zhang Zhejiang University, Xinkui Zhao Zhejiang University, Jianwei Yin Zhejiang University, Shuiguang Deng Zhejiang University; Alibaba-Zhejiang University Joint Institute of Frontier Technologies
12:15
15m
Talk
Search+LLM-based Testing for ARM SimulatorsArtifact-AvailableArtifact-FunctionalArtifact-Reusable
SE In Practice (SEIP)
Bobby Bruce University of California at Davis, USA, Aidan Dakhama King's College London, Karine Even-Mendoza King’s College London, William B. Langdon University College London, Hector Menendez King’s College London, Justyna Petke University College London
11:00 - 12:30
12:00
15m
Talk
How is Google using AI for internal code migrations?
SE In Practice (SEIP)
Stoyan Nikolov Google, Inc., Daniele Codecasa Google, Inc., Anna Sjovall Google, Inc., Maxim Tabachnyk Google, Siddharth Taneja Google, Inc., Celal Ziftci Google, Satish Chandra Google, Inc
14:00 - 15:30
15:00
15m
Talk
BSODiag: A Global Diagnosis Framework for Batch Servers Outage in Large-scale Cloud Infrastructure Systems
SE In Practice (SEIP)
Tao Duan Xi'an Jiaotong University, Runqing Chen Alibaba, Pinghui Wang Xi'an Jiaotong University, Junzhou Zhao Xi'an Jiaotong University, Jiongzhou Liu Alibaba, Shujie Han Northwestern Polytechnical University, Yi Liu Alibaba, Fan Xu Alibaba
15:15
15m
Talk
On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?Artifact-Available
SE In Practice (SEIP)
Matteo Esposito University of Oulu, Francesco Palagiano Multitel di Lerede Alessandro & C. s.a.s., Valentina Lenarduzzi University of Oulu, Davide Taibi University of Oulu
DOI Pre-print
14:00 - 15:30
14:00
15m
Talk
Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products
SE In Practice (SEIP)
Nadia Nahar Carnegie Mellon University, Christian Kästner Carnegie Mellon University, Jenna L. Butler Microsoft Research, Chris Parnin Microsoft, Thomas Zimmermann University of California, Irvine, Christian Bird Microsoft Research
14:00 - 15:30
15:00
15m
Talk
Testing False Recalls in E-commerce Apps: a User-perspective Blackbox Approach
SE In Practice (SEIP)
Shengnan Wu School of Computer Science, Fudan University, Yongxiang Hu Fudan University, Jiazhen Gu Fudan University, China, Penglei Mao School of Computer Science, Fudan University, Jin Meng Meituan Inc., Liujie Fan Meituan Inc., Zhongshi Luan Meituan Inc., Xin Wang Fudan University, Yangfan Zhou Fudan University
14:00 - 15:30
Security and Analysis 3Research Track / SE In Practice (SEIP) at 210
15:00
15m
Talk
A scalable, effective and simple Vulnerability Tracking approach for heterogeneous SAST setups based on Scope+OffsetSecurity
SE In Practice (SEIP)
James Johnson --, Julian Thome GitLab Inc., Lucas Charles GitLab Inc., Hua Yan GitLab Inc., Jason Leasure GitLab Inc.
Pre-print
15:15
15m
Talk
''ImmediateShortTerm3MthsAfterThatLOL'': Developer Secure-Coding Sentiment, Practice and Culture in OrganisationsArtifact-AvailableArtifact-FunctionalArtifact-ReusableSecurity
SE In Practice (SEIP)
Ita Ryan University College Cork, Utz Roedig University College Cork, Klaas-Jan Stol Lero; University College Cork; SINTEF Digital
16:00 - 17:30
16:00
15m
Talk
Full Line Code Completion: Bringing AI to Desktop
SE In Practice (SEIP)
Anton Semenkin JetBrains, Vitaliy Bibaev JetBrains, Yaroslav Sokolov JetBrains, Kirill Krylov JetBrains, Alexey Kalina JetBrains, Anna Khannanova JetBrains, Danila Savenkov JetBrains, Darya Rovdo JetBrains, Igor Davidenko JetBrains, Kirill Karnaukhov JetBrains, Maxim Vakhrushev JetBrains, Mikhail Kostyukov JetBrains, Mikhail Podvitskii JetBrains, Petr Surkov JetBrains, Yaroslav Golubev JetBrains Research, Nikita Povarov JetBrains, Timofey Bryksin JetBrains Research
Pre-print
16:00 - 17:30
16:30
15m
Talk
An Empirical Study on Decision-Making Aspects in Responsible Software Engineering for AIArtifact-Available
SE In Practice (SEIP)
Lekshmi Murali Rani Chalmers University of Technology and University of Gothenburg, Sweden, Faezeh Mohammadi Chalmers University of Technology and University of Gothenburg, Sweden, Robert Feldt Chalmers University of Technology, Sweden, Richard Berntsson Svensson Chalmers | University of Gothenburg
16:00 - 17:30
17:00
15m
Talk
Sunflower: Enhancing Linux Kernel Fuzzing via Exploit-Driven Seed GenerationArtifact-AvailableArtifact-FunctionalArtifact-ReusableSecurity
SE In Practice (SEIP)
Qiang Zhang Hunan University, Yuheng Shen Tsinghua University, Jianzhong Liu Tsinghua University, Yiru Xu Tsinghua University, Heyuan Shi Central South University, Yu Jiang Tsinghua University, Wanli Chang College of Computer Science and Electronic Engineering, Hunan University
16:00 - 17:30
16:30
15m
Talk
Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings
SE In Practice (SEIP)
Petr Tsvetkov JetBrains Research, Aleksandra Eliseeva JetBrains Research, Danny Dig University of Colorado Boulder, JetBrains Research, Alexander Bezzubov JetBrains, Yaroslav Golubev JetBrains Research, Timofey Bryksin JetBrains Research, Yaroslav Zharov JetBrains Research
Pre-print
16:45
15m
Talk
Enhancing Differential Testing: LLM-Powered Automation in Release Engineering
SE In Practice (SEIP)
Ajay Krishna Vajjala George Mason University, Arun Krishna Vajjala George Mason University, Carmen Badea Microsoft Research, Christian Bird Microsoft Research, Robert DeLine Microsoft Research, Jason Entenmann Microsoft Research, Nicole Forsgren Microsoft Research, Aliaksandr Hramadski Microsoft, Sandeepan Sanyal Microsoft, Oleg Surmachev Microsoft, Thomas Zimmermann University of California, Irvine, Haris Mohammad Microsoft, Jade D'Souza Microsoft, Mikhail Demyanyuk Microsoft
17:00
15m
Talk
How much does AI impact development speed? An enterprise-based randomized controlled trial
SE In Practice (SEIP)
Elise Paradis Google, Inc, Kate Grey Google, Quinn Madison Google, Daye Nam Google, Andrew Macvean Google, Inc., Nan Zhang Google, Ben Ferrari-Church Google, Satish Chandra Google, Inc
16:00 - 17:30
SE for AI with Quality 3Research Track / SE In Practice (SEIP) at 215
17:00
15m
Talk
Test Input Validation for Vision-based DL Systems: An Active Learning ApproachArtifact-AvailableArtifact-FunctionalArtifact-ReusableSE for AI
SE In Practice (SEIP)
Delaram Ghobari University of Ottawa, Mohammad Hossein Amini University of Ottawa, Dai Quoc Tran SmartInsideAI Company Ltd. and Sungkyunkwan University, Seunghee Park SmartInsideAI Company Ltd. and Sungkyunkwan University, Shiva Nejati University of Ottawa, Mehrdad Sabetzadeh University of Ottawa
Pre-print

Accepted Papers

Title
Agent for User: Testing Multi-User Interactive Features in TikTok
SE In Practice (SEIP)
AI-Assisted SQL Authoring at Industry ScaleSE for AI
SE In Practice (SEIP)
aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing
SE In Practice (SEIP)
Pre-print
An Empirical Study on Decision-Making Aspects in Responsible Software Engineering for AIArtifact-Available
SE In Practice (SEIP)
Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and DatasetArtifact-AvailableArtifact-FunctionalArtifact-Reusable
SE In Practice (SEIP)
Pre-print
ArkAnalyzer: The Static Analysis Framework for OpenHarmony
SE In Practice (SEIP)
A scalable, effective and simple Vulnerability Tracking approach for heterogeneous SAST setups based on Scope+OffsetSecurity
SE In Practice (SEIP)
Pre-print
ASTER: Natural and Multi-language Unit Test Generation with LLMsAward Winner
SE In Practice (SEIP)
Pre-print
Automated Code Review In Practice
SE In Practice (SEIP)
Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry
SE In Practice (SEIP)
Automating ML Model Development at ScaleSE for AI
SE In Practice (SEIP)
Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products
SE In Practice (SEIP)
BSODiag: A Global Diagnosis Framework for Batch Servers Outage in Large-scale Cloud Infrastructure Systems
SE In Practice (SEIP)
CI at Scale: Lean, Green, and Fast
SE In Practice (SEIP)
Crash Report Prioritization for Large-Scale Scheduled Launches
SE In Practice (SEIP)
Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplace
SE In Practice (SEIP)
Detecting Python Malware in the Software Supply Chain with Program AnalysisArtifact-AvailableArtifact-FunctionalArtifact-ReusableSecurity
SE In Practice (SEIP)
DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production
SE In Practice (SEIP)
Enhancing Differential Testing: LLM-Powered Automation in Release Engineering
SE In Practice (SEIP)
Evaluating Agent-based Program Repair at Google
SE In Practice (SEIP)
Evaluation of Tools and Frameworks for Machine Learning Model ServingSE for AI
SE In Practice (SEIP)
Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company
SE In Practice (SEIP)
Pre-print
FlatD: Protecting Deep Neural Network Program from Reversing Attacks
SE In Practice (SEIP)
Full Line Code Completion: Bringing AI to Desktop
SE In Practice (SEIP)
Pre-print
GUIWatcher: Automatically Detecting GUI Lags by Analyzing Mobile Application Screencasts
SE In Practice (SEIP)
How is Google using AI for internal code migrations?
SE In Practice (SEIP)
How much does AI impact development speed? An enterprise-based randomized controlled trial
SE In Practice (SEIP)
Human-In-the-Loop Software Development Agents
SE In Practice (SEIP)
Identifying Factors Contributing to ``Bad Days'' for Software Developers: A Mixed-Methods Study
SE In Practice (SEIP)
''ImmediateShortTerm3MthsAfterThatLOL'': Developer Secure-Coding Sentiment, Practice and Culture in OrganisationsArtifact-AvailableArtifact-FunctionalArtifact-ReusableSecurity
SE In Practice (SEIP)
Improving Code Performance Using LLMs in Zero-Shot: RAPGen
SE In Practice (SEIP)
KuiTest: Leveraging Knowledge in the Wild as GUI Testing Oracle for Mobile Apps
SE In Practice (SEIP)
Leveraging MLOps: Developing a Sequential Classification System for RFQ Documents in Electrical Engineering
SE In Practice (SEIP)
LLM Driven Smart Assistant for Data Mapping
SE In Practice (SEIP)
LogLM: From Task-based to Instruction-based Automated Log Analysis
SE In Practice (SEIP)
Pre-print
Moving Faster and Reducing Risk: Using LLMs in Release DeploymentAward Winner
SE In Practice (SEIP)
NICE: Non-Functional Requirements Identification, Classification, and Explanation Using Small Language ModelsArtifact-AvailableAward Winner
SE In Practice (SEIP)
On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?Artifact-Available
SE In Practice (SEIP)
DOI Pre-print
On Mitigating Code LLM Hallucinations with API Documentation
SE In Practice (SEIP)
On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure CategoriesArtifact-AvailableArtifact-FunctionalArtifact-Reusable
SE In Practice (SEIP)
Pre-print
Prioritizing Large-scale Natural Language Test Cases at OPPO
SE In Practice (SEIP)
Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation ModelsSE for AI
SE In Practice (SEIP)
Safe Validation of Pricing Agreements
SE In Practice (SEIP)
Search+LLM-based Testing for ARM SimulatorsArtifact-AvailableArtifact-FunctionalArtifact-Reusable
SE In Practice (SEIP)
SEMANTIC CODE FINDER: An Efficient Semantic Search Framework for Large-Scale Codebases
SE In Practice (SEIP)
Soft Skills in Software Engineering: Insights from the Trenches
SE In Practice (SEIP)
Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation ModelsArtifact-AvailableArtifact-FunctionalArtifact-Reusable
SE In Practice (SEIP)
Sunflower: Enhancing Linux Kernel Fuzzing via Exploit-Driven Seed GenerationArtifact-AvailableArtifact-FunctionalArtifact-ReusableSecurity
SE In Practice (SEIP)
Systematizing Inclusive Design in MOSIP: An Experience Report
SE In Practice (SEIP)
Testing False Recalls in E-commerce Apps: a User-perspective Blackbox Approach
SE In Practice (SEIP)
Test Input Validation for Vision-based DL Systems: An Active Learning ApproachArtifact-AvailableArtifact-FunctionalArtifact-ReusableSE for AI
SE In Practice (SEIP)
Pre-print
Time to Retrain? Detecting Concept Drifts in Machine Learning Systems
SE In Practice (SEIP)
Time Warp: The Gap Between Developers’ Ideal vs Actual Workweeks in an AI-Driven EraAward Winner
SE In Practice (SEIP)
Towards Better Static Analysis Bug Reports in the Clang Static Analyzer
SE In Practice (SEIP)
Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings
SE In Practice (SEIP)
Pre-print
UML Sequence Diagram Generation: A Multi-Model, Multi-Domain Evaluation
SE In Practice (SEIP)
Video Game Procedural Content Generation Through Software Transplantation
SE In Practice (SEIP)
Wearables to measure developer experience at work
SE In Practice (SEIP)

The following papers have been accepted in the ICSE 2025 SEIP Track. The papers are will be published by the IEEE and appear in the IEEE and ACM digital libraries, subject to an author submitting their camera-ready and copyright forms, and registering to attend the conference. (Authors are required to present the papers at the conference, otherwise they will be withdrawn).

John Kolesar, Tancrède Lepoint, Martin Schäf, Willem Visser, "Safe Validation of Pricing Agreements"

Abstract: Pricing agreements at AWS define how customers are billed for usage of services and resources. A pricing agreement consists of a complex sequence of terms that can include free tiers, savings plans, credits, volume discounts, and other similar features. To ensure that pricing agreements reflect the customers' intentions, we employ a protocol that runs a set of validations that check all pricing agreements for irregularities. Since pricing agreements are sensitive, we want to limit the amount of information available to actors involved in the validation protocol. Actors who repair validation errors should not be able to obtain information that is not necessary for the repair. Personal information for individual customers should remain private, and we also want to hide statistical properties of the set of pricing agreements, such as the average number of credits awarded to customers. We introduce Parrot, a protocol for validation of pricing agreements that prevents information leakage. Parrot involves three categories of actors: the auditor who writes a query to check a validity condition, the fixers who repair anonymized invalid pricing agreements, and the owners who have permission to view non-anonymized pricing agreements. The protocol includes safeguards to prohibit parties other than the owners from learning sensitive information about individual customers or statistical properties of the full set of pricing agreements.

 Tags: "Business", "Analysis", "Security"  
 
Qiang Zhang, Yuheng Shen, Jianzhong Liu, Yiru Xu, Heyuan Shi, Yu Jiang, Wanli Chang, "Sunflower: Enhancing Linux Kernel Fuzzing via Exploit-Driven Seed Generation"

Abstract: Abstract—The Linux kernel is the foundation of billions of contemporary computing systems, and ensuring its security and integrity is a necessity. Despite the Linux kernel’s pivotal role, guaranteeing its security is a difficult task due to its complex code logic. This leads to new vulnerabilities being frequently introduced, and malicious exploits can result in severe conse- quences like Denial of Service (DoS) or Remote Code Execu- tion (RCE). Fuzz testing (fuzzing), particularly Syzkaller, has been instrumental in detecting vulnerabilities within the kernel. However, Syzkaller’s effectiveness is hindered due to limitations in system call descriptions and initial seeds. In this paper, we propose SUNFLOWER, an initial corpus generator that leverages existing exploits and proof-of-concept examples. SUNFLOWER is specifically designed to meet the critical requirements of industry deployments by facilitating the construction of a high-quality seed corpus based on bugs found in the wild. By collecting and analyzing numerous real-world exploits responsible for kernel vulnerabilities, the tool extracts essential system call sequences while also rectifying execution dependency errors. This approach addresses a pressing industry need for more effective vulnerabil- ity assessment and exploit development, making it an invaluable asset for cybersecurity professionals. The evaluation shows that, with the help of SUNFLOWER, we find a total number of 25 previously unknown vulnerabilities within the extensively tested Linux Kernel, while by augmenting Syzkaller with SUNFLOWER, we achieve a 9.5% and 10.8% improvement on code coverage compared with the Syzkaller and Moonshine.

 Tags: "Security", "Testing and Quality"  
 
Ridwan Shariffdeen, Behnaz Hassanshahi, Martin Mirchev, Ali El Husseini, Abhik Roychoudhury, "Detecting Python Malware in the Software Supply Chain with Program Analysis"

Abstract: The frequency of supply-chain attacks has reached unprecedented levels, amounting to a growing concern about the security of open-source software. Existing state-of-the-art techniques often generate a high number of false positives and false negatives. For an effective detection tool, it is crucial to strike a balance between these results. In this paper, we address the problem of software supply chain protection through program analysis. We present HERCULE, an inter-package analysis tool to detect malicious packages in the Python ecosystem. We enhance state-of-the-art approaches with the primary goal of reducing false positives. Key technical contributions include improving the accuracy of pattern-based malware detection and employing program dependency analysis to identify malicious packages in the development environment. Extensive evaluation against multiple benchmarks including Backstabber’s Knife Collection and MalOSS demonstrates that HERCULE outperforms existing state-of-the-art techniques with 0.866 f1-score. Additionally, HERCULE detected new malicious packages which the PyPI security team removed, showing its practical value.

 Tags: "Security", "Analysis"  
 
Dhruva Juloori, Zhongpeng Lin, Matthew Williams, Eddy Shin, Sonal Mahajan, "CI at Scale: Lean, Green, and Fast"

Abstract: Maintaining a "green" mainline branch—where all builds pass successfully—is crucial but challenging in fast-paced, large-scale software development environments, particularly with concurrent code changes in large monorepos. SubmitQueue, a system designed to address these challenges, speculatively executes builds and only lands changes with successful outcomes. However, despite its effectiveness, the system faces inefficiencies in resource utilization, leading to a high rate of premature build aborts and delays in landing smaller changes blocked by larger conflicting ones. This paper introduces enhancements to SubmitQueue, focusing on optimizing resource usage and improving build prioritization. Central to this is our innovative probabilistic model, which distinguishes between changes with shorter and longer build times to prioritize builds for more efficient scheduling. By leveraging a machine learning model to predict build times and incorporating this into the probabilistic framework, we expedite the landing of smaller changes blocked by conflicting larger time-consuming changes. Additionally, introducing a concept of speculation threshold ensures that only the most likely builds are executed, reducing unnecessary resource consumption. After implementing these enhancements across Uber's major monorepos (Go, iOS, and Android), we observed a reduction in Continuous Integration (CI) resource usage by approximately 53\%, CPU usage by 44\%, and P95 waiting times by 37\%. These improvements highlight the enhanced efficiency of SubmitQueue in managing large-scale software changes while maintaining a green mainline.

 Tags: "Testing and Quality", "AI for SE", "Process"  
 
Shengnan Wu, Yongxiang Hu, Jiazhen Gu, Penglei Mao, Jin Meng, Liujie Fan, Zhongshi Luan, Xin Wang, Yangfan Zhou, "Testing False Recalls in E-commerce Apps: a User-perspective Blackbox Approach"

Abstract: Search components are essential in e-commerce apps, allowing users to find products and services. However, they often suffer from bugs, leading to false recalls, \textit{i.e.}, irrelevant search results. Detecting false recalls automatically is challenging. As users and shop owners adopt ambiguous natural language to describe their purchasing intentions and products, precise relevance determination becomes difficult. We propose \textbf{f}alse \textbf{r}ecall \textbf{H}ound (frHound), a black box testing approach targeting false recalls. The core idea of \MIG is to mimic users' online purchasing behavior. Specifically, frHound first designs 37 features to align with how users process information during online shopping, explored by a comprehensive user study. Then, frHound uses an outlier detection technique to identify the most divergent search results, similar to how general users make purchasing decisions during online shopping. Those divergent search results are likely false recalls, as most search results are relevant during e-commerce searches. Experiments with real industry data show frHound reduces human labor, time, and financial costs associated with discovering false recalls by 36.74 times. In a seven-month trial with \textit{M-app}, a popular Chinese e-commerce platform, frHound identified 1282 false recalls, improving user satisfaction and reducing false recall discovery costs.

 Tags: "User experience", "Business"  
 
daeha ryu, seokjun ko, eunbi jang, jinyoung park, myunggwan kim, changseo park, "SEMANTIC CODE FINDER: An Efficient Semantic Search Framework for Large-Scale Codebases"

Abstract: We present SEMANTIC CODE FINDER, a framework for semantic code search that delivers high-level search performance and supports multiple programming languages. Leveraging code summaries, it enables meaningful semantic code search by extracting the core content of code methods and using this information for search queries. Evaluated on large-scale codebases, SEMANTIC CODE FINDER demonstrates its effectiveness in outperforming existing open-source code search tools, achieving higher recall and precision rates. It delivers superior search performance across Java, Python, and C++. Notably, SEMANTIC CODE FINDER outperforms CodeMatcher, a previously successful semantic code search tool, by approximately 41\% in terms of MRR. Moreover, it shows consistent performance across Java, Python, and C++ languages, highlighting its robustness and effectiveness. Currently, it is being used as a code search service for a significant amount of source code within Samsung Electronics, meeting the needs of its developers.

 Tags: "Analysis", "Prog Comprehension/Reeng/Maint", "AI for SE"  
 
Yongxiang Hu, Yu Zhang, Xuan Wang, Yingjie Liu, Shiyu Guo, Chaoyi Chen, Xin Wang, Yangfan Zhou, "KuiTest: Leveraging Knowledge in the Wild as GUI Testing Oracle for Mobile Apps"

Abstract: In industrial practice, UI (User Interface) functional bugs typically manifest as inconsistent UI input and corresponding response. Such bugs can deteriorate user experiences and are, therefore, a major target of industrial testing practice. For a long time, testing for UI functional bugs has relied on rule-based methods, which are labor-intensive for rule development and maintenance. Given that the UI functional bugs typically manifest where an app’s response deviates from the user’s expectations, we proposed the key point of reducing human efforts lies in simulating human expectations. Due to the vast in-the-wild knowledge of large language models (LLMs), they are well-suited for this simulation. By leveraging LLMs as UI testing oracle, we proposed KuiTest, the first rule-free UI functional testing tool we designed for Company M, one of the largest E-commerce app providers serving over 600 million users. KuiTest can automatically predict the effect of UI inputs and verify the post-interaction UI response. We evaluate the design of KuiTest via a set of ablation experiments. Moreover, real-world deployments demonstrate that KuiTest can effectively detect previously unknown UI functional bugs and significantly improve the efficiency of GUI testing.

 Tags: "User experience", "AI for SE", "Testing and Quality"  
 
Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, Eray Tuzun, "Automated Code Review In Practice"

Abstract: Context: Code review is a widespread practice among practitioners to improve software quality and transfer knowledge. It is often perceived as time-consuming due to the need for manual effort and potential delays in the development process. Several AI-assisted code review tools (Qodo, GitHub Copilot, Coderabbit, etc.) provide automated code reviews using large language models (LLMs). The overall effects of such tools in the industry setting are yet to be examined. Objective: This study examines the impact of LLM-based automated code review tools in an industry setting. Method: The study was conducted within an industrial software development environment that adopted an AI-assisted code review tool (based on open-source Qodo PR Agent). Approximately 238 practitioners across ten projects had access to the tool. We focused our analysis on three projects, encompassing 4,335 pull requests, of which 1,568 underwent automated reviews. Our data collection comprised three primary sources: (1) a quantitative analysis of pull request data, including comment labels indicating whether developers acted on the automated comments, (2) surveys sent to developers regarding their experience with the reviews on individual pull requests, and (3) a broader survey of 22 practitioners capturing their general opinions on automated code reviews. Results: \%73.8 of automated code review comments were labeled as resolved. However, the overall average pull request closure duration increased from five hours 52 minutes to eight hours 20 minutes, with varying trends observed across different projects. According to survey responses, most practitioners observed a minor improvement in code quality as a result of automated code reviews. Conclusion: Our findings indicate that the tool was useful for software development. Additionally, developers highlighted several advantages, such as accelerated bug detection, increased awareness of code quality, and the promotion of best practices. However, it also led to longer pull request closure times. Developers noted several disadvantages, including faulty reviews, unnecessary corrections, and overly frequent or irrelevant comments. Based on our findings, we discussed how practitioners can more effectively utilize automated code review technologies.

 Tags: "Testing and Quality", "AI for SE"  
 
Anton Semenkin, Vitaliy Bibaev, Yaroslav Sokolov, Kirill Krylov, Alexey Kalina, Anna Khannanova, Danila Savenkov, Darya Rovdo, Igor Davidenko, Kirill Karnaukhov, Maxim Vakhrushev, Mikhail Kostyukov, Mikhail Podvitskii, Petr Surkov, Yaroslav Golubev, Nikita Povarov, Timofey Bryksin, "Full Line Code Completion: Bringing AI to Desktop"

Abstract: In recent years, several industrial solutions for the problem of multi-token code completion appeared, each making a great advance in the area but mostly focusing on cloud-based runtime and avoiding working on the end user's device. In this work, we describe our approach for building a multi-token code completion feature for the JetBrains' IntelliJ Platform, which we call \textit{Full Line Code Completion}. The feature suggests only syntactically correct code and works fully locally, \textit{i.e.}, data querying and the generation of suggestions happens on the end user's machine. We share important time and memory-consumption restrictions, as well as design principles that a code completion engine should satisfy. Working entirely on the end user's device, our code completion engine enriches user experience while being not only fast and compact but also secure. We share a number of useful techniques to meet the stated development constraints and also describe offline and online evaluation pipelines that allowed us to make better decisions. Our online evaluation shows that the usage of the tool leads to 1.3 times more Python code in the IDE being produced by code completion. The described solution was initially started with a help of researchers and was then bundled into all JetBrains IDEs where it is now used by millions of users. Thus, we believe that this work is useful for bridging academia and industry, providing researchers with the knowledge of what happens when complex research-based solutions are integrated into real products.

 Tags: "IDEs"  
 
Matteo Esposito, Francesco Palagiano, Valentina Lenarduzzi, Davide Taibi, "On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?"

Abstract: \textit{Context}. The security of critical infrastructure has been a fundamental concern since the advent of computers, and this concern has only intensified in today’s cyber warfare landscape. Protecting mission-critical systems (MCSs), including essential assets like healthcare, telecommunications, and military coordination, is vital for national security. These systems require prompt and comprehensive governance to ensure their resilience, yet recent events have shown that meeting these demands is increasingly challenging. \textit{Aim}. Building on prior research that demonstrated the potential of GAI, particularly Large Language Models (LLMs), in improving risk analysis tasks, we aim to explore practitioners' perspectives, specifically developers and security personnel, on using generative AI (GAI) in the governance of IT MCSs seeking to provide insights and recommendations for various stakeholders, including researchers, practitioners, and policymakers. \textit{Method.} We designed a survey to collect practical experiences, concerns, and expectations of practitioners who develop and implement security solutions in the context of MCSs. Analyzing this data will help identify key trends, challenges, and opportunities for introducing GAIs in this niche domain. \textit{Conclusions and Future Works}. Our findings highlight that the safe use of LLMs in MCS governance requires interdisciplinary collaboration. Researchers should focus on designing regulation-oriented models and focus on accountability; practitioners emphasize data protection and transparency, while policymakers must establish a unified AI framework with global benchmarks to ensure ethical and secure LLMs-based MCS governance.

 Tags: "SE for AI", "Security", "Real-Time"  
 
Gokul Rejithkumar, Preethu Rose Anish, "NICE: Non-Functional Requirements Identification, Classification, and Explanation Using Small Language Models"

Abstract: Accurate identification and classification of Non-Functional Requirements (NFRs) is essential for informed architectural decision-making and maintaining software quality. Numerous language model-based techniques have been proposed for NFR identification and classification. However, understanding the reasoning behind the classification outputs of these techniques remains challenging. Rationales for the classification outputs of language models enhance comprehension, aid in debugging the models, and build confidence in the classification outputs. In this paper, we present NICE, a tool for NFR Identification, Classification, and Explanation. Using an industrial requirements dataset, we generated explanations in natural language using the GPT-4o large language model (LLM). We then fine-tuned small language models (SLMs), including T5, Llama 3.1, and Phi 3, with these LLM-generated explanations to identify and classify NFRs and to explain their classification outputs. We evaluated NICE using standard evaluation metrics such as F1-score and human evaluation to assess the quality of the generated explanations. Among the models tested, T5 produced explanations of quality comparable to Llama 3.1 and Phi 3 but achieved the highest average F1-score of 0.90 in multi-label NFR classification on the industrial requirements dataset. Furthermore, to evaluate the effectiveness of NICE, a survey was conducted with 20 requirements analysts and software developers. NICE is currently deployed as a part of the Knowledge-assisted Requirements Evolution (K-RE) framework developed by a large IT vendor organization.

 Tags: "Requirements", "AI for SE"  
 
Jinquan Zhang, Zihao Wang, Pei Wang, Rui Zhong, Dinghao Wu, "FlatD: Protecting Deep Neural Network Program from Reversing Attacks"

Abstract: The emergence of deep learning (DL) compilers provides automated optimization and compilation across DL frameworks and hardware platforms, which enhances the performance of AI service and primarily benefits the deployment to edge devices and low-power processors. However, DNN programs generated from DL compilers introduce a new attack interface. They are targeted by new model extraction attacks that can fully or partially rebuild the DNN model by reversing the DNN programs. Unfortunately, no defense countermeasure is designed to hinder this kind of attack. To address the issue, we investigate all the state-of-the-art reversing-based model extraction attacks and identify an essential component shared across the frameworks. Based on this observation, we propose FlatD, the first defense framework for DNN programs toward reversing-based model extraction attacks. FlatD manipulates and conceals the original control flow graph (CFG) of DNN programs based on control flow flattening (CFF). Unlike traditional CFF, FlatD ensures the DNN programs are challenging for attackers to recover their CFG and gain necessary information statically. Our evaluation shows that, compared to the traditional CFF (O-LLVM), FlatD provides more effective and stealthy protection to DNN programs with similar performance and less scale.

 Tags: "SE for AI", "Security", "Analysis"  
 
Kristóf Umann, Zoltán Porkoláb, "Towards Better Static Analysis Bug Reports in the Clang Static Analyzer"

Abstract: Static analysis is a method increasingly used for finding bugs and other deviations of software systems. While it fits well for the modern development environment and is capable to catch issues earlier than the manual code review or the various testing methodologies, human experts are still deeply involved in the evaluation of whether the tool reported a true finding or a false positive. This creates a serious bottleneck in the development flow. While many efforts have been made to improve the accuracy of the tools, little attention has been paid to the quality of the reports. Improving the report quality could improve the decision on possible false positives, shorten the bug fixing time and enhance the trust in static analysis tools. In this paper we report our research to find the most important attributes for generating clear and concise bug reports for the Clang Static Analyzer tool. With the help of experienced developers, we evaluated several test cases synthetized from real-world examples and analyzed how they rated the report elements according to the aspects of understanding. The results show that current reports from the Clang Static Analyzer, one of the most popular static analysis tools, can flood the developers with unimportant information while some of the report generation steps may eliminate relevant code parts. Our measurement methodology and results could be used to improve bug report quality, and therefore enhance the application of the tools. Despite our study focusing on one specific tool, the lessons learned could be used for a research targeting a wider range of static analyzers. Based on our findings, we made suggestions to the developers of the Clang Static Analyzer and an improvement to the bug report generation was made and is already available in version 19.0.0.

 Tags: "Analysis", "Testing and Quality", "IDEs"  
 
Chi Xiao, Daniel Ståhl, Jan Bosch, "UML Sequence Diagram Generation: A Multi-Model, Multi-Domain Evaluation"

Abstract: The automation of UML sequence diagram generation has posed a persistent challenge in software engineering, with existing approaches relying heavily on manual processes. Recent advancements in natural language processing, particularly through large language models, offer promising solutions for automating this task. This paper investigates the use of large language models in automating the generation of UML sequence diagrams from natural language requirements. We evaluate three state-of-the-art large language models, GPT 4o, Mixtral 8x7B, and Llama 3.1 8B, across multiple datasets, including both public and proprietary requirements, to assess their performance in terms of correctness, completeness, clarity, and readability. The results indicate GPT 4o consistently outperforms the other models in most metrics. Our findings highlight the potential of large language models to streamline requirements engineering by reducing manual effort, although further refinement is needed to enhance their performance in complex scenarios. This study provides key insights into the strengths and limitations of these models, and offers practical guidance for their application, advancing the understanding of how large language models can support automation in software engineering tasks.

 Tags: "AI for SE", "Analysis", "Design/Architecture"  
 
Claudio Martens, Hammam Abdelwahab, Katharina Beckh, Birgit Kirsch, Vishwani Gupta, Dennis Wegener, Steffen Hoh, "Leveraging MLOps: Developing a Sequential Classification System for RFQ Documents in Electrical Engineering"

Abstract: Vendors participating in tenders face significant challenges in creating timely and accurate order quotations from Request for Order (RFQ) documents. The success of their bids is heavily dependent on the speed and precision of these quotations. A key bottleneck in this process is the time-consuming task of identifying relevant products from the product catalogue that align with the RFQ descriptions. We propose the implementation of an automatic classification system that utilizes a context-aware language model specifically designed for the electrical engineering domain. Our approach aims to streamline the identification of relevant products, thereby enhancing the efficiency and accuracy of the quotation process. However, an effective solution must be scalable and easily adjustable. Thus, we present a machine learning operations (MLOps) architecture that facilitates automated training and deployment. We pay particular attention to automated pipelines, which are essential for the operation of a maintainable ML solution. Furthermore, we outline best practices for creating production-ready pipelines and encapsulating data science efforts. Schneider Electric currently operates the solution presented in this paper.

 Tags: "Business", "AI for SE", "Design/Architecture"  
 
Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, Saurabh Sinha, "ASTER: Natural and Multi-language Unit Test Generation with LLMs"

Abstract: Implementing automated unit tests is an important but time consuming activity in software development. To support developers in this task, software engineering research over the past few decades has developed many techniques for automating unit test generation. However, despite this effort, usable tools exist for very few programming languages. Moreover, studies have found that automatically generated tests suffer poor readability and often do not resemble developer-written tests. In this work, we present a rigorous investigation of how large language models (LLMs) can help bridge the gap. We describe a generic pipeline that incorporates static analysis to guide LLMs in generating compilable and high-coverage test cases. We illustrate how the pipeline can be applied to different programming languages, specifically Java and Python, and to complex software requiring environment mocking. We conducted an empirical study to assess the quality of the generated tests in terms of code coverage and test naturalness---evaluating them on standard as well as enterprise Java applications and a large Python benchmark. Our results demonstrate that LLM-based test generation, when guided by static analysis, can be competitive with, and even outperform, state-of-the-art test-generation techniques in coverage achieved while also producing considerably more natural test cases that developers find easy to read and understand. We also present the results of a user study, conducted with 161 professional developers, that highlights the naturalness characteristics of the tests generated by our approach.

 Tags: "Testing and Quality", "AI for SE"  
 
Chandra Maddila, Negar Ghorbani, Kosay Jabre, Vijayaraghavan Murali, Edwin Kim, Parth Thakkar, Nikolay Pavlovich Laptev, Olivia Harman, Diana Hsu, Rui Abreu, Peter Rigby, "AI-Assisted SQL Authoring at Industry Scale"

Abstract: SqlCompose brings generative AI into the data analytics domain. SQL is declarative, has formal table schemas, and is often written in a non-linear manner. We address each of these challenges and develop a set of models that shows the importance of each problem. We first develop an internal SQL benchmark to perform offline tests at Meta. We evaluate how well the Public Llama model performs. We attain a BLEU score of 53% and 24% for single- and multi-line predictions, respectively. This performance is consistent with prior works on imperative languages. We then fine-tune Llama on our internal data and database schemas. SC-Schema substantially outperforms Llama by 16 percentage points on BLEU score. SQL is often written with multiple sub queries and in a non-sequential manner. We develop SC-FIM which is aware of the context before and after the line(s) that need to be completed. This fill-in-the-middle model outperform SC-FIM by 35 percentage points. We also measure how often the models get the correct table names, and SC-FIM is able to do this 75% of the time a major improvement over the other two models. Aside from our scientific research, we also roll out SC-FIM at Meta. SqlCompose is used on a weekly basis by over 10k users including data scientists and software engineers, less than 1% of users have disabled SqlCompose. We use the feedback from users to improve SqlCompose. Interesting positive themes include completing tedious or repetitive SQL clauses, suggesting boilerplate coding, and help in eliminate the need to remember difficult SQL syntax. The most significant negative themes was table and column name hallucinations, which has been reduced with the release of SC-FIM. The SqlCompose models consistently outperform public and internal LLMs despite their smaller size (7 bn and 13 bn), which provides early indications that smaller specialist models can outperform larger general purpose models.

 Tags: "Databases", "AI for SE"  
 
Rui Abreu, Vijayaraghavan Murali, Peter Rigby, Chandra Maddila, Weiyan Sun, Jun Ge, Kaavya Chinniah, Audris Mockus, Megh Mehta, Nachiappan Nagappan, "Moving Faster and Reducing Risk: Using LLMs in Release Deployment"

Abstract: Release engineering has traditionally focused on continuously delivering features and bug fixes to users, but at a certain scale, it becomes impossible for a release engineering team to determine what should be released. At Meta’s scale, the responsibility appropriately and necessarily falls back on the engineer writing and reviewing the code. To address this challenge, we developed models of diff risk scores (DRS) to determine how likely a diff is to cause a SEV, i.e., a severe fault that impacts end-users. Assuming that SEVs are only caused by diffs, a naive model could randomly gate X% of diffs from landing, which would automatically catch X% of SEVs on average. However, we aimed to build a model that can capture Y% of SEVs by gating X% of diffs, where Y >> X. By training the model on historical data on diffs that have caused SEVs in the past, we can predict the riskiness of an outgoing diff to cause a SEV. Diffs that are beyond a particular threshold of risk can then be gated. We have four types of gating: no gating (green), weekend gating (weekend), medium impact on end-users (yellow), and high impact on end-users (red). The input parameter for our models is the level of gating, and the outcome measure is the number of captured SEVs, i.e., the number of gated diffs that would have led to a SEV. Our research approaches include a logistic regression model, a BERT-based model, and generative LLMs. Our baseline regression model captures 18.7%, 27.9%, and 84.6% of SEVs while respectively gating the top 5% (weekend), 10% (yellow), and 50% (red) of risky diffs. The BERT-based model, StarBERT, only captures 0.61×, 0.85×, and 0.81× as many SEVs as the logistic regression for the weekend, yellow, and red gating zones, respectively. The generative LLMs, iCodeLlama-34B and iDiffLlama-13B, when risk-aligned, capture more SEVs than the logistic regression model in production: 1.40×, 1.52×, 1.05×, respectively.

 Tags: "Testing and Quality", "AI for SE"  
 
James Johnson, Julian Thome, Lucas Charles, Hua Yan, Jason Leasure, "A scalable, effective and simple Vulnerability Tracking approach for heterogeneous SAST setups based on Scope+Offset"

Abstract: Managing software projects using Source Control Management (SCM) systems like Git, combined with automated security testing in Continuous Integration and Continuous Delivery (CI/CD) processes, is a best practice in today’s software industry. These processes continuously monitor code changes to detect security vulnerabilities as early as possible. Security testing often involves multiple Static Application Security Testing (SAST) tools, each specialized in detecting specific vulnerabilities, such as hardcoded passwords or insecure data flows. A heterogeneous SAST setup, using multiple tools, helps minimize the software’s attack surface. The security findings from these tools undergo Vulnerability Management, a semi-manual process of understanding, categorizing, storing, and acting on them. Code volatility, i.e., the constant change of the project’s source code, as well as double reporting, i.e., the overlap of findings reported by multiple tools, are potential sources of duplication imposing futile auditing effort on the analyst. Vulnerability Tracking is an automated process that helps deduplicating and tracking vulnerabilities throughout the lifetime of a software project. We propose a scalable Vulnerability Tracking approach called Scope+Offset for heterogeneous SAST setups that reduces the noise introduced by code volatility as well as code duplication. Our proposed, fully automated method proved to be highly effective in an industrial setting, reducing the negative effect of duplication by approximately 30% which directly translates to a reduction in futile auditing time while inducing a negligible performance overhead. Since its product integration into GitLab in 2022, Scope+Offset provided vulnerability tracking to the thousands of security scans running on the GitLab DevSecOps platform every day where the GitLab DevSecOps platform can be considered as a heterogeneous SAST setup as it includes a variety of different SAST tools.

 Tags: "Security", "Process", "MSR"  
 
Soumiki Chattopadhyay, Amreeta Chatterjee, Puja Agarwal, Bianca Trinkenreich, Swarathmika Kumar, Rohit Ranjan Rai, Resham Chugani, Pragya Kumari, Margaret Burnett, Anita Sarma, "Systematizing Inclusive Design in MOSIP: An Experience Report"

Abstract: The GenderMag method has been successfully used by software teams to improve inclusivity in their software products across various domains. Given the success of this method, here we investigate how GenderMag can be systematically adopted in an organization. It is a conceptual replication of our prior work that identified a set of practices and pitfalls synthesized across different USA-based teams. Through Action Research, we trace the 3+ years long journey of GenderMag adoption in the MOSIP organization; starting from the initial ‘unfreeze’ stage to the institutionalization (‘re-freeze’) of GenderMag in the organization’s processes. Our findings identify: (1) which practices from the prior work could be generalized and how some of them had to be modified to fit MOSIP organization’s context (Digital Public Goods, open-source product, and fully remote work environment), and (2) the pitfalls that occurred.

 Tags: "Human/Social", "User experience"  
 
Ita Ryan, Utz Roedig, Klaas-Jan Stol, "'ImmediateShortTerm3MthsAfterThatLOL'': Developer Secure-Coding Sentiment, Practice and Culture in Organisations"

Abstract: As almost all areas of human endeavour undergo rapid digital transformation, secure coding is increasingly important to personal, commercial and national security. Yet studies have shown that software developers do not always prioritise or even understand security. Our large survey of organically sourced coders (n=863) examines how software developers currently experience secure coding in the workplace. We found that developers express an interest in secure coding, display basic security knowledge, and turn to their managers and teams first for help with security concerns. We found that developer secure coding traits and security practice do not correlate with organisational statistics such as size, but do correlate weakly with measures of security culture, and in some cases with practice, indicating that organisational security support goes hand-in-hand with secure development. Investigating the effects of code breaches, we found that for almost half of cases, code security does not increase, or increases only for a short time.

 Tags: "Security", "Human/Social", "Process"  
 
Yilun Liu, Yuhe Ji, Shimin Tao, Minggui He, Weibin Meng, Shenglin Zhang, Yongqian Sun, Yuming Xie, Boxing Chen, Hao Yang, "LogLM: From Task-based to Instruction-based Automated Log Analysis"

Abstract: Automatic log analysis is essential for the efficient Operation and Maintenance (O\&M) of software systems, providing critical insights into system behaviors. However, existing approaches mostly treat log analysis as training a model to perform an isolated task, using task-specific log-label pairs. These task-based approaches are inflexible in generalizing to complex scenarios, depend on task-specific training data, and cost significantly when deploying multiple models. In this paper, we propose an instruction-based training approach that transforms log-label pairs from multiple tasks and domains into a unified format of instruction-response pairs. Our trained model, LogLM, can follow complex user instructions and generalize better across different tasks, thereby increasing flexibility and reducing the dependence on task-specific training data. By integrating major log analysis tasks into a single model, our approach also relieves model deployment burden. Experimentally, LogLM outperforms existing approaches across five log analysis capabilities, and exhibits strong generalization abilities on complex instructions and unseen tasks. In addition, LogLM has been deployed on an O\&M platform of Huawei, managing 11165 devices and processing an average of 209 user instructions per day.

 Tags: "Analysis", "AI for SE"  
 
Niklas Beck, Benny Stein, Dennis Wegener, Lennard Helmer, "Evaluation of Tools and Frameworks for Machine Learning Model Serving"

Abstract: Machine learning (ML) models are ubiquitous, as ML expands into numerous application domains. Despite this growth, the software engineering of ML systems remains complex, particularly in production environments. Serving ML models for inference is a critical part, as it provides the interface between the model and the surrounding components. Today, a variety of open source tools and frameworks for model serving exists which promise ease of use and performance. However, they differ in terms of usability, flexibility, scalability, and their overall performance. In this work, we systematically evaluate several popular model serving tools and frameworks in the context of a natural language processing scenario. In detail, we analyze their features and capabilities, conduct runtime experiments, and report on our experiences from various real-world ML projects. Our evaluation results provide valuable insights and considerations for ML engineers and other practitioners seeking for effective serving environments that seamlessly integrate with the existing ML tech stack, simplifying and accelerating the process of serving ML models in production.

 Tags: "SE for AI"  
 
Mar Zamorano, Daniel Blasco, Carlos Cetina, Federica Sarro, "Video Game Procedural Content Generation Through Software Transplantation"

Abstract: Software transplantation generates new piece of software by reusing existing parts from a given piece of software (i.e., host) to enhance other parts of a same software or different software (i.e., donor). To date, its use has been proved beneficial for traditional software systems. In this paper, we argue that software transplantation can be used for automatically producing video game procedural content. We assessed the feasibility of our idea by realising the first search-based algorithm for procedural content transplantation and empirically evaluating it in an industrial case study in collaboration with the developers of the commercial video game Kromaia. Specifically, our proposed approach, dubbed IMHOTEP, enables developers to choose what video-game content to transplant and where, and automatically searches for an appropriate solution to integrate the organ into the host. Such a search is performed by using an evolutionary algorithm guided by a simulation-based fitness function, which is novel w.r.t previous transplantation work generally guided by test-suite compliance. We empirically evaluate the effectiveness of IMHOTEP to transplant procedural content, specifically non-playable characters, for the commercial video game Kromaia and benchmarked it against a state-of-the-art approach in search-based procedural content generation, as well as a variant of IMHOTEP itself guided by a test-suite-based fitness function. We found that using IMHOTEP, Kromaia developers were able to transplant 129 distinct organs taken from the game’s scenarios into five different hosts, thus generating a total of 645 new transplanted non-playable characters for this game. Moreover, we found that the game content generated by using IMHOTEP was 1.5 times superior than the one obtained by using its test-suite-based variant, and 2.5 times superior than the one generated by the state-of-the-art benchmark. Furthermore, the transplants generated by IMHOTEP have also unveiled organ interactions that had not been previously identified in the literature. Finally, a focus group with game developers indicated their satisfaction with the content generated by IMHOTEP and willingness to use it in their game development activity. The positive results obtained by IMHOTEP, prove the viability of Procedural Content Transplantation and open up new research avenues for automated video-game content generation as well as for software transplantation.

 Tags: "Design/Architecture", "Games", "Analysis"  
 
Lekshmi Murali Rani, Faezeh Mohammadi, Robert Feldt, Richard Berntsson Svensson, "An Empirical Study on Decision-Making Aspects in Responsible Software Engineering for AI"

Abstract: Incorporating responsible practices into Software Engineering (SE) for Artificial Intelligence (AI) is essential to ensure ethical principles, societal impact, and accountability remain at the forefront of AI system design and deployment. This study investigates the ethical challenges and complexities inherent in Responsible SE (RSE) for AI, underscoring the need for practical, scenario-driven operational guidelines. Given the complexity of AI and the relative inexperience of professionals in this rapidly evolving field, continuous learning and market adaptation are crucial. Through qualitative interviews with seven practitioners (conducted until saturation), quantitative surveys of 51 practitioners and static validation of results with four industry experts in AI, this study explores how personal values, emerging roles, and awareness of AI’s societal impact influence responsible decision-making in RSE for AI. A key finding is the gap between the current state of the art and actual practice in RSE for AI, particularly in the failure to operationalize ethical and responsible decision-making within the SE lifecycle for AI. While ethical issues in RSE for AI largely mirror those found in broader SE process, the study highlights a distinct lack of operational frameworks and resources to guide RSE practices for AI effectively. The results reveal that current ethical guidelines are insufficiently implemented at the operational level, reinforcing the complexity of embedding ethics throughout the software engineering life cycle. The study concludes that interdisciplinary collaboration, H-shaped competencies (Ethical-Technical dual competence), and a strong organizational culture of ethics are critical for fostering RSE practices for AI, with a particular focus on transparency and accountability.

 Tags: "SE for AI", "Human/Social", "Resp SE and Ethics"  
 
Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Nan Zhang, Ben Ferrari-Church, Satish Chandra, "How much does AI impact development speed? An enterprise-based randomized controlled trial"

Abstract: How much does AI assistance impact developer productivity? To date, the software engineering literature has provided a range of answers, targeting a diversity of outcomes: from perceived productivity to speed on task and developer throughput. Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on task. Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\%, although our confidence interval is large. We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI. Product and future research considerations are discussed. In particular, we invite further research that explores the impact of AI at the ecosystem level and across multiple suites of AI-enhanced tools, since we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time.

 Tags: "AI for SE", "Process"  
 
Behrooz Omidvar Tehrani, Ingo Gühring, Gauthier Guinet, Linbo Liu, Hoan Nguyen, Talha Oz, Qiang Zhou, Laurent Callot, Jun Huan, Omer Tripp, Anoop Deoras, "Migrating Java Applications with Amazon Q Developer Agent for Code Transformation"

Abstract: Software maintenance, particularly migrating Java applications from older JVM version to newer ones, presents significant challenges for developers due to the rapid evolution of programming languages and increasing reliance on third-party libraries. In this paper we introduce Amazon Q Developer Agent, a novel system for fully automated Java migrations. The system combines rule-based approaches, web-mined knowledge, and large language models (LLMs) to offer a comprehensive solution for Java upgrades. Amazon Q Developer Agent addresses key challenges in the migration process, including dependency updates and deprecated API replacements, while seamlessly integrating with popular integrated development environments (IDEs). The system significantly reduces the time and effort required for Java migrations, with case studies showing upgrades from Java 8 and 11 to 17 being completed in hours rather than the typical 50 developer-days. In a large-scale application, Amazon engineers successfully upgraded 30,000 internal Java 8 applications to Java 17 in less than a year using an internal version of this tooling. This paper details the system's architecture, methodology, and presents experimental results demonstrating its effectiveness in revolutionizing the software migration process, saving thousands of developer-years in maintenance efforts.

 Tags: "Prog Comprehension/Reeng/Maint", "AI for SE", "Design/Architecture"  
 
Haonan Chen, Daihang Chen, Yizhuo Yang, Lingyun Xu, Liang Gao, Mingyi Zhou, Chunming Hu, Li Li, "ArkAnalyzer: The Static Analysis Framework for OpenHarmony"

Abstract: ArkTS is a new programming language dedicated to developing apps for the emerging OpenHarmony mobile operating system. Like other programming languages (e.g., Typescripts) constantly suffering from performance-related code smells or vulnerabilities, the ArkTS programming language will likely encounter the same problems. The solution given by our research community is to invent static analyzers, which are often implemented on top of a common static analysis framework, to detect and subsequently repair those issues automatically. Unfortunately, such an essential framework is not available for the OpenHarmony community yet. Existing program analysis methods have several problems when handling the ArtTS code. To bridge the gap, we design and implement a framework named ArkAnalyzer and make it publicly available as an open-source project. Our ArkAnalyzer addresses the aforementioned problems and has already integrated a number of fundamental static analysis functions (e.g., control-flow graph constructions, call graph constructions, etc.) that are ready to be reused by developers to implement OpenHarmony app analyzers focusing on statically resolving dedicated issues such as performance bug detection, privacy leaks detection, compatibility issues detection, etc. Experiment results show that our ArkAnalyzer achieves both high analyzing efficiency and high effectiveness. In addition, we open-sourced the dataset that has numerous real-world ArkTS Apps.

 Tags: "Analysis"  
 
Haoran Xu, Chen Zhi, Tianyu Xiang, Zixuan Wu, Gaorong Zhang, Xinkui Zhao, Jianwei Yin, Shuiguang Deng, "Prioritizing Large-scale Natural Language Test Cases at OPPO"

Abstract: Regression testing is a crucial process for ensuring system stability following software updates. As a global leader in smart device manufacturing, OPPO releases a new version of its customized Android firmware, ColorOS, on a weekly basis. Testers must select test cases from a vast repository of manual test cases for regression testing. The tight schedule makes it difficult for testers to select the correct test cases from this extensive pool. Since these test cases are described in natural language, testers must manually execute them according to the operational steps, making the process labor-intensive and error-prone. Therefore, an effective test case recommendation system is needed to suggest appropriate test cases, reducing unnecessary human effort during weekly regression tests. To address these challenges, we propose a two-phase manual test case recommendation system. Our system first uses the BERT model to classify commit message, determining the most relevant test labels. Then, it employs the BGE embedding model to compute the semantic similarity between the commit message and the test cases, recommending the most suitable test cases. This approach has been practically deployed within OPPO, and feedback from several months of use shows that our test case recommendation accuracy reaches 91\%. The time testers spend selecting test cases has decreased by 61\%, the number of test cases executed per code change has dropped by 87\%, and the defect detection rate of the recommended test cases has increased by 182.35\%. Our method achieves high accuracy, low human effort, and a high defect detection rate. This paper introduces the integration of the BERT classification model and the BGE semantic similarity model in the context of manual test case recommendation, significantly improving the accuracy and efficiency of test case recommendations and providing valuable insights for regression testing in complex software systems.

 Tags: "Testing and Quality", "AI for SE"  
 
Mohammad Saiful Islam, Mohamed Sami Rakha, William Pourmajidi, Janakan Sivaloganathan, John Steinbacher, Andriy Miranskyy, "Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset"

Abstract: As Large-Scale Cloud Systems (LCS) become increasingly complex, effective anomaly detection is critical for ensuring system reliability and performance. However, there is a shortage of large-scale, real-world datasets available for benchmarking anomaly detection methods. To address this gap, we introduce a new high-dimensional dataset from IBM Cloud, collected over 4.5 months from the IBM Cloud Console. This dataset comprises 39,365 rows and 117,448 columns of telemetry data. Additionally, we demonstrate the application of machine learning models for anomaly detection and discuss the key challenges faced in this process. This study and the accompanying dataset provide a resource for researchers and practitioners in cloud system monitoring. It facilitates more efficient testing of anomaly detection methods in real-world data, helping to advance the development of robust solutions to maintain the health and performance of large-scale cloud infrastructures.

 Tags: "Analysis", "AI for SE", "Testing and Quality"  
 
Tri Minh-Triet Pham, Karthikeyan Premkumar, Mohamed Naili, Jinqiu Yang, "Time to Retrain? Detecting Concept Drifts in Machine Learning Systems"

Abstract: With the boom of machine learning (ML) tech- niques, software practitioners build ML systems to process the massive volume of streaming data for diverse software engineering tasks such as failure prediction in AIOps. Trained using historical data, such ML models encounter performance degradation caused by concept drift, i.e., data and inter-relationship (concept) changes between training and production. It is essential to use concept rift detection to monitor the deployed ML models and re-train the ML models when needed. In this work, we explore applying state-of-the-art (SOTA) concept drift detection techniques on synthetic and real-world datasets in an industrial setting. Such an industrial setting requires minimal manual effort in labeling and maximal generality in ML model architecture. We find that current SOTA semi-supervised methods not only require significant labeling effort but also only work for certain types of ML models. To overcome such limitations, we propose a novel model-agnostic technique (CDSeer) for detecting concept drift. Our evaluation shows that CDSeer has better precision and recall compared to the state-of-the-art while requiring significantly less manual labeling. We demonstrate the effectiveness of CDSeer at concept drift detection by evaluating it on eight datasets from different domains and use cases. Results from internal deployment of CDSeer on an industrial proprietary dataset show a 57.1% improvement in precision while using 99% fewer labels compared to the SOTA concept drift detection method. The performance is also comparable to the supervised concept drift detection method, which requires 100% of the data to be labeled. The improved performance and ease of adoption of CDSeer are valuable in making ML systems more reliable.

 Tags: "AI for SE", "Analysis"  
 
Delaram Ghobari, Mohammad Hossein Amini, Dai Quoc Tran, Seunghee Park, Shiva Nejati, Mehrdad Sabetzadeh, "Test Input Validation for Vision-based DL Systems: An Active Learning Approach"

Abstract: Testing deep learning (DL) systems requires extensive and diverse, yet valid, test inputs. While synthetic test input generation methods, such as metamorphic testing, are widely used for DL testing, they risk introducing invalid inputs that do not accurately reflect real-world scenarios. Invalid test inputs can lead to misleading results. Hence, there is a need for automated validation of test inputs to ensure effective assessment of DL systems. In this paper, we propose a test input validation approach for vision-based DL systems. Our approach uses active learning to balance the trade-off between accuracy and the manual effort required for test input validation. Further, by employing multiple image-comparison metrics, it achieves better results in classifying valid and invalid test inputs compared to methods that rely on single metrics. We evaluate our approach using an industrial and a public-domain dataset. Our evaluation shows that our multi-metric, active learning-based approach produces several optimal accuracy-effort trade-offs, including those deemed practical and desirable by our industry partner. Furthermore, provided with the same level of manual effort, our approach is significantly more accurate than two state-of-the-art test input validation methods, achieving an average accuracy of 97%. Specifically, the use of multiple metrics, rather than a single metric, results in an average improvement of at least 5.4% in overall accuracy compared to the state-of-the-art baselines. Incorporating an active learning loop for test input validation yields an additional 7.5% improvement in average accuracy, bringing the overall average improvement of our approach to at least 12.9% compared to the baselines.

 Tags: "Testing and Quality", "SE for AI"  
 
Nihal Jain, Robert Kwiatkowski, Baishakhi Ray, Murali Krishna Ramanathan, Varun Kumar, "On Mitigating Code LLM Hallucinations with API Documentation"

Abstract: In this study, we address the issue of API hallucinations in various software engineering contexts. We introduce CloudAPIBench, a new benchmark designed to measure API hallucination occurrences. CloudAPIBench also provides annotations for frequencies of API occurrences in the public domain, allowing us to study API hallucinations at various frequency levels. Our findings reveal that Code LLMs struggle with low frequency APIs: for e.g., GPT-4o achieves only $38.58$\% valid low frequency API invocations. We demonstrate that Documentation Augmented Generation (DAG) significantly improves performance for low frequency APIs (increase to $47.94$\% with DAG) but negatively impacts high frequency APIs when using sub-optimal retrievers (a $39.02$\% absolute drop). To mitigate this, we propose to intelligently trigger DAG where we check against an API index or leverage Code LLMs' confidence scores to retrieve only when needed. We demonstrate that our proposed methods enhance the balance between low and high frequency API performance, resulting in more reliable API invocations ($8.20$\% absolute improvement on CloudAPIBench for GPT-4o).

 Tags: "AI for SE", "Design/Architecture", "Analysis"  
 
Xiaoyun Liang, Jingyi Ren, Jiayi Qi, Chao Peng, Bo Jiang, Siyuan Jiang, Jia Li, He Zong, Huanyu Liu, Hao Zhu, Shukai Hu, Erlu Li, Jiazheng Ding, Ge Li, "aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing"

Abstract: Large Language Models (LLMs) have been widely used in code completion, and researchers are focusing on scaling up LLMs to improve their accuracy. However, larger LLMs will increase the response time of code completion and decrease the developers' productivity. In this paper, we propose a lightweight and effective LLM for code completion named aiXcoder-7B. Compared to existing LLMs, aiXcoder-7B achieves higher code completion accuracy while having smaller scales (i.e., 7 billion parameters). We attribute the superiority of aiXcoder-7B to three key factors: (1) Multi-objective training. We employ three training objectives, one of which is our proposed Structured Fill-In-the-Middle (SFIM). SFIM considers the syntax structures in code and effectively improves the performance of LLMs for code. (2) Diverse data sampling strategies. They consider inter-file relationships and enhance the capability of LLMs in understanding cross-file contexts. (3) Extensive high-quality data. We establish a rigorous data collection pipeline and consume a total of 1.2 trillion unique tokens for training aiXcoder-7B. This vast volume of data enables aiXcoder-7B to learn a broad distribution of code. We evaluate aiXcoder-7B in five popular code completion benchmarks and a new benchmark collected by this paper. The results show that aiXcoder-7B outperforms the latest six LLMs with similar sizes and even surpasses four larger LLMs (e.g., StarCoder2-15B and CodeLLaMa-34B), positioning aiXcoder-7B as a lightweight and effective LLM for academia and industry. Finally, we summarize three valuable insights for helping practitioners train the next generations of LLMs for code. aiXcoder-7B has been open-souced and gained significant attention. As of the submission date, aiXcoder-7B has received 2,193 GitHub Stars.

 Tags: "AI for SE", "Design/Architecture"  
 
Petr Tsvetkov, Aleksandra Eliseeva, Danny Dig, Alexander Bezzubov, Yaroslav Golubev, Timofey Bryksin, Yaroslav Zharov, "Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings"

Abstract: Commit message generation (CMG) is a crucial task in software engineering that is challenging to evaluate correctly. When a CMG system is integrated into the IDEs and other products at JetBrains, we perform online evaluation based on user acceptance of the generated messages. However, performing online experiments with every change to a CMG system is troublesome, as each iteration affects users and requires time to collect enough statistics. On the other hand, offline evaluation, a prevalent approach in the research literature, facilitates fast experiments but employs automatic metrics that are not guaranteed to represent the preferences of real users. In this work, we describe a novel way we employed to deal with this problem at JetBrains, by leveraging an online metric---the number of edits users introduce before committing the generated messages to the VCS---to select metrics for offline experiments. To support this new type of evaluation, we develop a novel markup collection tool mimicking the real workflow with a CMG system, collect a dataset with 57 pairs consisting of commit messages generated by GPT-4 and their counterparts edited by human experts, and design and verify a way to synthetically extend such a dataset. Then, we use the final dataset of 656 pairs to study how the widely used similarity metrics correlate with the online metric reflecting the real users' experience. Our results indicate that edit distance exhibits the highest correlation, whereas commonly used similarity metrics such as BLEU and METEOR demonstrate low correlation. This contradicts the previous studies on similarity metrics for CMG, suggesting that user interactions with a CMG system in real-world settings differ significantly from the responses by human labelers operating within controlled research environments. While our findings are tied to the CMG model we used, and the results may vary for models with cardinally different outputs, our proposed framework is relatively lightweight in terms of required human effort, and we release all the code and the dataset to support future research in the field of CMG: https://jb.gg/cmg-evaluation.

 Tags: "Analysis", "Process", "AI for SE"  
 
Spandan Garg, Roshanak Moghaddam, Neel Sundaresan, "Improving Code Performance Using LLMs in Zero-Shot: RAPGen"

Abstract: Performance bugs are non-functional bugs that can even manifest in well-tested commercial products. Fixing these performance bugs is an important yet challenging problem. In this work, we address this challenge and present a new approach called Retrieval-Augmented Prompt Generation (RAPGen). Given a code snippet with a performance issue, RAPGen first retrieves a prompt instruction from a pre-constructed knowledge-base of previous performance bug fixes and then generates a prompt using the retrieved instruction. It then uses this prompt on a Large Language Model (such as Codex) in zero-shot to generate a fix. We compare our approach with the various prompt variations and state of the art methods in the task of performance bug fixing. Our empirical evaluation shows that RAPGen can generate performance improvement suggestions equivalent or better than a developer in ~60% of the cases, getting ~42% of them verbatim, in an expert-verified dataset of past performance changes made by C# developers. Furthermore, we conduct an in-the-wild evaluation to verify the model's effectiveness in practice by suggesting fixes to developers in a large software company. So far, we have shared fixes on 10 codebases that represent production services running in the cloud and 7 of them have been accepted by the developers.

 Tags: "AI for SE", "Testing and Quality"  
 
Guilherme Vaz Pereira, Victoria Jackson, Rafael Prikladnicki, André van der Hoek, Luciane Fortes, Carolina Araújo, André Coelho, Ligia Chelli, Diego Ramos, "Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company"

Abstract: Recent progress in Generative AI (GenAI) impacts different software engineering (ES) tasks in software development cycle, e.g., from code generation to program repair, and presents a promising avenue for enhancing the productivity of development teams. GenAI based tools have the potential to change the way we develop software and have received attention from industry and academia. However, although some studies have been addressing the adoption of these tools in the software industry, little is known about what are developers' real experiences in a professional software development context, aside the hype. In this paper, we explore the use of GenAI tools by a large Brazilian media company that has teams developing software in-house. We observed practitioners for six weeks and used online surveys at different time points to understand their expectations, perceptions, and concerns about these tools in their daily work. In addition, we automatically collected quantitative data from the company's development systems, aiming at getting insights about how GenAI impacts the development process during the period. Our results provide insights into how practitioners perceive and utilize GenAI in their daily work in software development.

 Tags: "Human/Social", "AI for SE", "Analysis"  
 
Charlotte Brandebusemeyer, Tobias Schimmer, Bert Arnrich, "Wearables to measure developer experience at work"

Abstract: Software development presents various challenges for developers: Programming, problem-solving and communication skills are required in multiple deadline-oriented projects running in parallel. These challenges can adversely affect developers’ mental well-being and productivity. So far, software developers’ well-being has mainly been examined via questionnaires or extensive and costly setups in laboratory settings. Objective, continuous and minimally invasive measurements with wearables in real-life settings are scarce and remain underutilized. This exploratory study uses a mixed-method approach to investigate the cognitive load component of developer experience and how it impacts professional SAP software developers in their everyday working context. Previous studies have demonstrated that physiological activity recorded by wearables can effectively measure cognitive load. In this study, thirty professional SAP software developers evaluated their developer experience in a questionnaire. Twenty of them additionally recorded physiological data during one working day using the Empatica E4 wristband. During the recording, they documented their working tasks and rated their experienced cognitive load during those tasks. The questionnaire data served as support for the physiological data recording. Key findings are that subjective cognitive load ratings positively correlate with electrodermal activity and skin temperature, and negatively with wrist acceleration. Three task categories (development-heavy, collaboration-heavy and other) could be discriminated based on subjective load ratings. Skin temperature and wrist acceleration differed significantly between development-heavy and other tasks. Subjective evaluations of the software developers in this study indicate that work environment factors which the individual software developer cannot influence negatively impact the developer experience. This study motivates the use of wearables in a firm context to enable a holistic evaluation of developer experience by combining subjective and objective measures in a real-world setting. Efficient task scheduling and detecting first signs of burnout could be enabled, which increases the software developer’s well-being and productivity and long-term reduces sick days and turnover rates in firms.

 Tags: "Human/Social", "Process"  
 
Ajay Krishna Vajjala, Arun Krishna Vajjala, Carmen Badea, Christian Bird, Rob DeLine, Jason Entenmann, Nicole Forsgren, Aliaksandr Hramadski, Sandeepan Sanyal, Oleg Surmachev, Thomas Zimmermann, Haris Mohammad, Jade D'Souza, Mikhail Demyanyuk, "Enhancing Differential Testing: LLM-Powered Automation in Release Engineering"

Abstract: In modern software engineering, efficient release engineering workflows are essential for quickly delivering new features to production. This not only improves company productivity but also provides customers with frequent updates, which can lead to increased profits. At Microsoft, we collaborated with the Identity and Network Access (IDNA) team to automate their release engineering workflows. They use differential testing to classify differences between test and production environments, which helps them assess how new changes perform with real-world traffic before pushing updates to production. This process enhances resiliency and ensures robust changes to the system. However, on-call engineers (OCEs) must manually label hundreds or thousands of behavior differences, which is time-consuming. In this work, we present a method leveraging Large Language Models (LLMs) to automate the classification of these differences, which saves OCEs a significant amount time. Our experiments demonstrate that LLMs are effective classifiers for automating the task of behavior difference classification, which can lead to speeding up release workflows, and improved OCE productivity.

 Tags: "Process", "AI for SE", "Testing and Quality"  
 
Pat Rondon, Renyao Wei, Jose Cambronero, Jürgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano, Satish Chandra, "Evaluating Agent-based Program Repair at Google"

Abstract: Agent-based program repair offers the promise of automatically resolving complex bugs end-to-end by combining the planning, tool usage, and code-generating abilities of modern LLMs. Recent work has explored the use of agent-based repair approaches on the popular open-source SWE-Bench, a collection of bugs (and patches) from popular GitHub-hosted Python projects. In addition, various agentic approaches such as SWE-Agent have been proposed to solve bugs in this benchmark. This paper explores the opportunity of using a similar agentic approach to address bugs in an enterprise-scale context. We perform a systematic comparison of bugs in SWE-Bench and those found in Google's issue tracking system and show that they have different distributions in terms of language diversity, size and spread of changes, and ease of localization. Next, we implement Passerine, an agent similar in spirit to SWE-Agent that can work within Google's environment and produce patches for bugs in Google's code repository. To evaluate Passerine, we curate an evaluation set of 182 bugs, spanning human-reported (82) and machine-reported bugs (100) from Google's issue tracking system. We show that with 20 trajectory samples Passerine can produce a plausible patch for 70% of machine-reported and 14.6% of human-reported bugs in our evaluation set. After manual examination, we found that 42% of machine-reported bugs and 13.4% of human-reported bugs have at least one patch that is semantically equivalent to the ground-truth patch. This establishes a lower bound on performance that suggests agent-based APR holds promise for large-scale enterprise bug repair.

 Tags: "Analysis/Repair", "AI for SE", "Testing and Quality"  
 
Henri Aïdasso, Francis Bordeleau, Ali Tizghadam, "On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories"

Abstract: The continuous delivery of modern software requires the execution of many automated pipeline jobs. These jobs ensure the frequent release of new software versions while detecting code problems at an early stage. For TELUS, our industrial partner in the telecommunications field, reliable job execution is crucial to minimize wasted time and streamline Continuous Deployment (CD). In this context, flaky job failures are one of the main issues hindering CD. Prior studies proposed techniques based on machine learning to automate the detection of flaky jobs. While valuable, these solutions are insufficient to address the waste associated with the diagnosis of flaky failures, which remain largely unexplored due to the wide range of underlying causes. This study examines 4,511 flaky job failures at TELUS to identify the different categories of flaky failures that we prioritize based on Recency, Frequency, and Monetary (RFM) measures. We identified 46 flaky failure categories that we analyzed using clustering and RFM measures to determine 14 priority categories for future automated diagnosis and repair research. Our findings also provide valuable insights into the evolution and impact of these categories. The identification and prioritization of flaky failure categories using RFM analysis introduce a novel approach that can be used in other contexts.

 Tags: "Analysis", "Process", "Testing and Quality"  
 
Sukrit Kumar, Drishti Goel, Thomas Zimmermann, Brian Houck, B. Ashok, Chetan Bansal, "Time Warp: The Gap Between Developers’ Ideal vs Actual Workweeks in an AI-Driven Era"

Abstract: Software developers balance a variety of different tasks in a workweek, yet the allocation of time often differs from what they consider ideal. Identifying and addressing these deviations is crucial for organizations aiming to enhance the productivity and well-being of the developers. In this paper, we present the findings from a survey of 484 software developers at Microsoft, which aims to identify the key differences between how developers would like to allocate their time during an ideal workweek versus their actual workweek. Our analysis reveals significant deviations between a developer's ideal workweek and their actual workweek, with a clear correlation: as the gap between these two workweeks widens, we observe a decline in both productivity and satisfaction. By examining these deviations in specific activities, we assess their direct impact on the developers' satisfaction and productivity. Additionally, given the growing adoption of AI tools in software engineering, both in the industry and academia, we identify specific tasks and areas that could be strong candidates for automation. In this paper, we make three key contributions: 1) We quantify the impact of workweek deviations on developer productivity and satisfaction 2) We identify individual tasks that disproportionately affect satisfaction and productivity. 3) We provide actual data-driven insights to guide future AI automation efforts in software engineering, aligning them with the developers' requirements and ideal workflows for maximizing their productivity and satisfaction.

 Tags: "Human/Social", "Process"  
 
Stoyan Nikolov, Daniele Codecasa, Anna Sjovall, Maxim Tabachnyk, Siddharth Taneja, Celal Ziftci, Satish Chandra, "How is Google using AI for internal code migrations?"

Abstract: In recent years, there has been a tremendous interest in using generative AI, and particularly large language models (LLMs) in software engineering; indeed several companies now offer commercially available tools, and many large companies also have created their own ML-based tools for their software engineers. While the use of ML for common tasks such as code completion is available in commodity tools, there is a growing interest in application of LLMs for more bespoke purposes. One such purpose is code migration. This article is an experience report on using LLMs for code migrations at Google. It is not a research study, in the sense that we do not carry out comparisons against other approaches or evaluate research questions/hypotheses. Rather, we share our experiences in applying LLM-based code migration in an enterprise context across a range of migration cases, in the hope that other industry practitioners will find our insights useful. Many of these learnings apply to any bespoke application of ML in software engineering. We see evidence that the use of LLMs can reduce the time needed for migrations significantly, and can reduce barriers to get started and complete migration programs.

 Tags: "AI for SE", "Prog Comprehension/Reeng/Maint"  
 
Bobby R. Bruce, Aidan Dakhama, Karine Even-Mendoza, W.B Langdon, Héctor D. Menéndez, Justyna Petke, "Search+LLM-based Testing for ARM Simulators"

Abstract: In order to aid quality assurance of large complex hardware architectures, system simulators have been developed. However, such system simulators do not always accurately mirror what would have happened on a real device. A significant challenge in testing these simulators comes from the complexity of having to model both the simulation and the infinite number of software that could be run on such a device. Our previous work introduced SearchSYS, a testing framework for software simulators. SearchSYS leverages a large language model for initial seed C code generation which is then compiled, and the resultant binary is fed to a fuzzer. We then use differential testing by running the outputs of fuzzing on real hardware and a system simulator to identify mismatches. In this paper, we present and discuss our solution to the problem of testing software simulators, using SearchSYS to test the gem5 VLSI digital circuit simulator, employed by ARM to test their systems. In particular, we focus on the simulation of the ARM silicon chip Instruction Set Architecture (ISA). SearchSYS can create test cases that activate bugs by combining LLMs, fuzzing, and differential testing. Using only LLM, SearchSYS identified 74 test cases that activated bugs. By incorporating fuzzing, this number increased by 93 additional bug-activating cases within 24 hours. Through differential testing, we identified 624 bugs with LLM-generated test cases and 126 with fuzzed test inputs. Out of the total number of bug-activating test cases, 4 unique bugs have been reported and acknowledged by developers. Additionally, we provided developers with a test case suite and fuzzing statistics, and open-sourced SearchSYS.

 Tags: "Testing and Quality", "AI for SE"  
 
Wei Liu, Feng Lin, Linqiang Guo, Tse-Hsun (Peter) Chen, Ahmed E. Hassan, "GUIWatcher: Automatically Detecting GUI Lags by Analyzing Mobile Application Screencasts"

Abstract: The Graphical User Interface (GUI) plays a central role in mobile applications, directly affecting usability and user satisfaction. Poor GUI performance, such as lag or unresponsiveness, can lead to negative user experience and decreased mobile application (app) ratings. In this paper, we present GUIWatcher, a framework designed to detect GUI lags by analyzing screencasts recorded during mobile app testing. GUIWatcher uses computer vision techniques to identify three types of lag-inducing frames (i.e., janky frames, long loading frames, and frozen frames) and prioritizes the most severe ones that significantly impact user experience. Our approach was evaluated using real-world mobile application tests, achieving high results in detecting GUI lags in screencasts, with an average precision of 0.91 and recall of 0.96. The comprehensive bug reports generated from the lags detected by GUIWatcher help developers focus on the more critical issues and debug them efficiently. Additionally, GUIWatcher has been deployed in a real-world production environment, continuously monitoring app performance and successfully identifying critical GUI performance issues. By offering a practical solution for identifying and addressing GUI lags, GUIWatcher contributes to enhancing user satisfaction and the overall quality of mobile apps.

 Tags: "User experience", "AI for SE"  
 
Nadia Nahar, Christian Kaestner, Jenna L. Butler, Chris Parnin, Thomas Zimmermann, Christian Bird, "Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products"

Abstract: Large Language Models (LLMs) are increasingly embedded into software products across diverse industries, enhancing user experiences, but at the same time introducing numerous challenges for developers. Unique characteristics of LLMs force developers, who are accustomed to traditional software development and evaluation, out of their comfort zones as the LLM components shatter standard assumptions about software systems. This study explores the emerging solutions that software developers are adopting to navigate the encountered challenges. Leveraging a mixed-method research, including 26 interviews and a survey with 332 responses, the study identifies 19 emerging solutions regarding quality assurance that practitioners across several product teams at Microsoft are exploring. The findings provide valuable insights that can guide the development and evaluation of LLM-based products more broadly in the face of these challenges.

 Tags: "Human/Social", "AI for SE", "Testing and Quality", "User experience"  
 
Kaiyuan Wang, Yang Li, Junyang Shen, Kaikai Sheng, Yiwei You, Jiaqi Zhang, Srikar Ayyalasomayajula, Julian Grady, Martin Wicke, "Automating ML Model Development at Scale"

Abstract: Google has a large team of machine learning (ML) developers working on a large number of ML models. ML model development suffers from long edit/validate cycles compared to traditional software development. This makes it tedious and time consuming for modeling teams to propagate ML innovations to their models. We present HEINZELMAENNCHEN, an ML modeling automation system, which allows users to apply semantically specified modeling changes to models and evaluate them at scale. Three insights are key to creating this system: Automatic code modification allows us to mechanically apply modeling changes to a wide variety of models. Workflow automation systems are well suited to operate complex ML training machinery as if they were humans, saving significant manual effort. And finally, given a large enough model population, even imperfect automatic modeling with a lower-than-human success rate will generate significant aggregate gains. In this paper, We describe the design and a implementation of our system. We also evaluate the system performance and include an empirical study to demonstrate the utility and critical impact of the system. Our system is widely used by hundreds of ML developers and it significantly accelerates model development on hundreds of production models.

 Tags: "SE for AI", "Design/Architecture", "Analysis"  
 
Nimmi Weeraddana, Sarra Habchi, Shane McIntosh, Ike Obi, Jenna L. Butler, Sankeerti Haniyur, Brian Hassan, Margaret-Anne Storey, Brendan Murphy, "Identifying Factors Contributing to ``Bad Days'' for Software Developers: A Mixed-Methods Study"

Abstract: Software development is a dynamic activity that requires engineers to work effectively with tools, processes, and collaborative teams. As a result, the presence of friction can significantly hinder productivity, increase frustration, and contribute to low morale among developers. By contrast, higher satisfaction levels are positively correlated with higher levels of perceived productivity. Hence, understanding the factors that cause bad experiences for developers is critical for fostering a positive and productive engineering environment. In this research, we employed a mixed-method approach, including interviews, surveys, diary studies, and analysis of developer telemetry data to uncover and triangulate common factors that cause ``bad days'' for developers. The interviews involved 22 developers across different levels and roles. The survey captured the perception of 214 developers about factors that cause them to have ``bad days'', their frequency, and their impact on job satisfaction. The daily diary study engaged 79 developers for 30 days to document factors that caused ``bad days'' in the moment. We examined the telemetry signals of 131 consenting participants to validate the impact of bad developer experience using system data. Findings from our research revealed factors that cause ``bad days'' for developers and significantly impact their work and well-being. We discuss the implications of these findings and suggest future work.

 Tags: "Human/Social", "Process"  
 
Kirill Vasilevski, Dayi Lin, Ahmed E. Hassan, "Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models"

Abstract: To balance the quality and inference cost of a Foundation Model (FM, such as large language models (LLMs)) powered software, people often opt to train a routing model that routes requests to FMs with different sizes and capabilities. Existing routing models rely on learning the optimal routing decision from carefully curated data, require complex computations to be updated, and do not consider the potential evolution of weaker FMs. In this paper, we propose Real-time Adaptive Routing (RAR), an approach to continuously adapt FM routing decisions while using guided in-context learning to enhance the capabilities of weaker FM. The goal is to reduce reliance on stronger, more expensive FMs. We evaluate our approach on different subsets of the popular MMLU benchmark. Our approach routes 50.2\% fewer requests to computationally expensive models while maintaining around 90.5\% of the general response quality. In addition, the generated guidance from stronger models has shown intra-domain generalization and led to a better quality of responses compared to an equivalent approach with a standalone weaker FM.

 Tags: "SE for AI", "Analysis"  
 
Jenna L. Butler, Jina Suh, Sankeerti Haniyur, Constance Hadley, "Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplace"

Abstract: Generative AI coding tools are relatively new, and their impact on developers extends beyond traditional coding metrics, influencing beliefs about work and developers’ roles in the workplace. This study aims to illuminate developers' pre-existing beliefs about generative AI tools, their self-perceptions, and how regular use of these tools may alter these beliefs. Using a mixed-methods approach, including surveys, a randomized controlled trial, and a three-week diary study, we explored the real-world application of generative AI tools within a large multinational software company. Our findings reveal that the introduction and sustained use of generative AI coding tools significantly increases developers' perceptions of these tools as both useful and enjoyable. However, developers' views on the trustworthiness of AI-generated code remained unchanged. We also discovered unexpected uses of these tools, such as replacing web searches and fostering creative ideation. Additionally, 84\% of participants reported positive changes in their daily work practices, and 66\% noted shifts in their feelings about their work, ranging from increased enthusiasm to heightened awareness of the need to stay current with technological advances. This research provides both qualitative and quantitative insights into the evolving role of generative AI in software development and offers practical recommendations for maximizing the benefits of this emerging technology—particularly in balancing the productivity gains from AI-generated code with the need for increased scrutiny and critical evaluation of its outputs.

 Tags: "Human/Social", "AI for SE"  
 
Wannita Takerngsaksiri, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, Ming Wu, "Human-In-the-Loop Software Development Agents"

Abstract: Recently, Large Language Models (LLMs)-based multi-agent paradigms for software engineering are introduced to automatically resolve software development tasks (e.g., from a given issue to source code). However, existing work is evaluated based on historical benchmark datasets, does not consider human feedback at each stage of the automated software development process, and has not been deployed in practice. In this paper, we introduce a Human-in-the-loop LLM-based Agents framework (HULA) for software development that allows software engineers to refine and guide LLMs when generating coding plans and source code for a given task. We design, implement, and deploy the HULA framework into Atlassian JIRA for internal uses. Through a multi-stage evaluation of the HULA framework, Atlassian software engineers perceive that HULA can minimize the overall development time and effort, especially in initiating a coding plan and writing code for straightforward tasks. On the other hand, challenges around code quality are raised to be solved in some cases. We draw lessons learned and discuss opportunities for future work, which will pave the way for the advancement of LLM-based agents in software development.

 Tags: "AI for SE", "Human/Social", "Process"  
 
Hao Li, Cor-Paul Bezemer, Ahmed E. Hassan, "Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models"

Abstract: Foundation models (FMs) such as large language models (LLMs) have significantly impacted many fields, including software engineering (SE). The interaction between SE and FMs has led to the integration of FMs into SE practices (FM4SE) and the application of SE methodologies to FMs (SE4FM). While several literature surveys exist on academic contributions to these trends, we are the first to provide a practitioner's view. We analyze 155 FM4SE and 997 SE4FM blog posts from leading technology companies, leveraging an FM-powered surveying approach to systematically label and summarize the discussed activities and tasks. We observed that while code generation is the most prominent FM4SE task, FMs are leveraged for many other SE activities such as code understanding, summarization, and API recommendation. The majority of blog posts on SE4FM are about model deployment \& operation, and system architecture \& orchestration. Although the emphasis is on cloud deployments, there is a growing interest in compressing FMs and deploying them on smaller devices such as edge or mobile devices. We outline eight future research directions inspired by our gained insights, aiming to bridge the gap between academic findings and real-world applications. Our study not only enriches the body of knowledge on practical applications of FM4SE and SE4FM but also demonstrates the utility of FMs as a powerful and efficient approach in conducting literature surveys within technical and grey literature domains. Our dataset, results, code and used prompts can be found in our online replication package at https://github.com/SAILResearch/fmse-blogs.

 Tags: "AI for SE", "Human/Social", "SE for AI"  
 
Sanna Malinen, Matthias Galster, Antonija Mitrovic, Sreedevi Sankara Iyer, Pasan Peiris, April Clarke, "Soft Skills in Software Engineering: Insights from the Trenches"

Abstract: Context: Soft skills (e.g., communication skills, empathy, team skills) impact software project success. Objective: We aim to understand (1) how soft skills are defined and perceived in the software industry, (2) what soft skills are required in software development roles, (3) how soft skills are trained, and (4) how soft skills are assessed in the software industry. Method: We conducted 18 semi-structured interviews with software professionals in different roles. We manually analyzed transcripts following a general inductive approach. Results: There is ambiguity in soft skills definition, but agreement on their importance. Most critical soft skills were communication, leadership, and teamwork skills, but we also identified less frequently discussed skills: resilience and self-awareness. Further, we find that soft skills are not systematically assessed, likely due to difficulties in their assessment. A theme emerged on the importance of ongoing soft skills training as well as tailoring training to software professionals' needs. Conclusions: Our research supports past research on the importance of soft skills in software engineering and suggests that further emphasis is needed on soft skills assessment and training. We discuss implications for software professionals, those in leadership roles, and companies.

 Tags: "Human/Social"  
 
Sidong Feng, Changhao Du, Huaxiao Liu, Qingnan Wang, Zhengwei Lv, Gang Huo, Xu Yang, Chunyang Chen, "Agent for User: Testing Multi-User Interactive Features in TikTok"

Abstract: TikTok, a widely-used social media app boasting over a billion monthly active users, requires effective app quality assurance for its intricate features. Feature testing is crucial in achieving this goal. However, the multi-user interactive features within the app, such as voice calls, live streaming, and video conferencing, pose significant challenges for developers, who must handle simultaneous device management and user interaction coordination. Inspired by the concept of agents designed to autonomously and collaboratively tackle problems, we introduce a novel multi-agent approach, powered by the Large Language Models (LLMs), to automate the testing of multi-user interactive app features. In detail, we build a virtual device farm that allocates the necessary number of devices for a given multi-user interactive task. For each device, we deploy an LLM-based agent that simulates a user, thereby mimicking user interactions to collaboratively automate the testing process. The evaluations on 24 multi-user interactive tasks within the TikTok app, showcase its capability to cover 75% of tasks with 85.9% action similarity and offer 87% time savings for developers. Additionally, we have also integrated our approach into the real-world TikTok testing platform, aiding in the detection of 26 multi-user interactive bugs.

 Tags: "Testing and Quality", "AI for SE", "User experience"  
 
Tao Duan, Runqing Chen, Pinghui Wang, Junzhou Zhao, Jiongzhou Liu, Shujie Han, Yi Liu, Fan Xu, "BSODiag: A Global Diagnosis Framework for Batch Servers Outage in Large-scale Cloud Infrastructure Systems"

Abstract: Cloud infrastructure is the collective term for all physical devices within cloud systems. Failures within the cloud infrastructure system can severely compromise the stability and availability of cloud services. Particularly, batch servers outage, which is the most fatal failure, could result in the complete unavailability of all upstream services. In this work, we focus on the batch servers outage diagnosis problem, aiming to accurately and promptly analyze the root cause of outages to facilitate troubleshooting. However, our empirical study conducted in a real industrial system indicates that it is a challenging task. Firstly, the collected single-modal coarse-grained failure monitoring data (i.e., alert, incident, or change) in the cloud infrastructure system is insufficient for a comprehensive failure profiling. Secondly, due to the intricate dependencies among devices, outages are often the cumulative result of multiple failures, but correlations between failures are difficult to ascertain. To address these problems, we propose BSODiag, an unsupervised and lightweight framework designed for diagnosing batch servers outage. BSODiag provides a global analytical perspective, thoroughly explores failures information from multi-source monitoring data, models the spatio-temporal correlations among failures, and delivers accurate and interpretable diagnostic results. Experiments conducted on the Alibaba Cloud infrastructure system show that BSODiag achieves 87.5% PR@3 and 46.3% PCR, outperforming baseline methods by 10.2% and 3.7%, respectively.

 Tags: "Analysis", "Real-Time", "Testing and Quality"  
 
Martin Obaidi, Nicolas Voß, Hannah Deters, Jakob Droste, Marc Herrmann, Jannik Fischbach, Kurt Schneider, "Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry"

Abstract: Providing explanations in response to user reviews is a time-consuming and repetitive task for companies, as many reviews present similar issues requiring nearly identical responses. To improve efficiency, this paper proposes a semi-automated approach to managing explanation needs in user reviews. The approach leverages taxonomy categories to classify reviews and assign them to relevant internal teams or sources for responses. 2,366 app reviews from the Google Play Store and Apple App Store were scraped and analyzed using a word and phrase filtering system to detect explanation needs. The detected needs were categorized and assigned to specific internal teams at Graphmasters GmbH, using a hierarchical assignment strategy that prioritizes the most relevant teams. Additionally, external sources, such as existing support articles and past review responses, were integrated to provide comprehensive explanations. The system was evaluated through interviews and surveys with the Graphmasters support team. The results showed that the hierarchical assignment method improved the accuracy of team assignments, with correct teams being identified in 79.2% of cases. However, challenges in interrater agreement and the need for new responses in certain cases, particularly for Apple App Store reviews, were noted. Future work will focus on refining the taxonomy and enhancing the automation process to reduce manual intervention further.

 Tags: "Testing and Quality", "Human/Social"  
 
Arihant Bedagkar, Sayandeep Mitra, Raveendra Kumar M, Ravindra Naik, Samiran Pal, "LLM Driven Smart Assistant for Data Mapping"

Abstract: Data mapping is a crucial step during application migration and for application integration. Data (model) mapping comprises "schema matching" to identify semantically equivalent fields between two schema, and "transformation logic generation" to write rules for converting data from one schema to the other. In industry practice today, data mapping is largely manual in nature, done by domain experts. We present a data mapping assistant powered by Large Language Models (LLMs), providing disruptive precision improvement over SOTA methods and multiple automation workflows that let users provide different available input triggers (context) for inferring the mappings. We illustrate the contribution using various representative industrial datasets.

 Tags: "Analysis", "Databases", "AI for SE"  
 
:
: