SANER 2025
Tue 4 - Fri 7 March 2025 Montréal, Québec, Canada

This program is tentative and subject to change.

Wed 5 Mar 2025 11:15 - 11:30 at L-1720 - Software Maintenance and Evolution

Large language models (LLMs) like ChatGPT have shown the potential to assist developers with coding and debugging tasks. However, their role in collaborative issue resolution is underexplored. In this study, we analyzed 1,152 Developer-ChatGPT conversations across 1,012 issues in GitHub to examine the diverse usage of ChatGPT and reliance on its generated code. Our contributions are fourfold. First, we manually analyzed 289 conversations to understand ChatGPT’s usage in the GitHub Issues. Our analysis revealed that ChatGPT is primarily utilized for ideation, whereas its usage for validation (e.g., code documentation accuracy) is minimal. Second, we applied BERTopic modeling to identify key areas of engagement on the entire dataset. We found that backend issues (e.g., API management) dominate conversations, while testing is surprisingly less covered. Third, we utilized the CPD clone detection tool to check if the code generated by ChatGPT was used to address issues. Our findings revealed that ChatGPT-generated code was used as-is to resolve only 5.83% of the issues. Fourth, we estimated sentiment using a RoBERTa-based sentiment analysis model to determine developers’ satisfaction with different usages and engagement areas. We found positive sentiment (i.e., high satisfaction) about using ChatGPT for refactoring and addressing data analytics (e.g., categorizing table data) issues. On the contrary, we observed negative sentiment when using ChatGPT to debug issues and address automation tasks (e.g., GUI interactions). Our findings show the unmet needs and growing dissatisfaction among developers. Researchers and ChatGPT developers should focus on developing task-specific solutions that help resolve diverse issues, improving user satisfaction and problem-solving efficiency in software development.

This program is tentative and subject to change.

Wed 5 Mar

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
Software Maintenance and EvolutionJournal First Track / Industrial Track / Research Papers at L-1720
11:00
15m
Talk
How Effective are Large Language Models in Generating Software Specifications?
Research Papers
Danning Xie Purdue University, Byoungwoo Yoo UNIST, Nan Jiang Purdue University, Mijung Kim UNIST, Lin Tan Purdue University, Xiangyu Zhang Purdue University, Judy Lee ADP
11:15
15m
Talk
Why Do Developers Engage with ChatGPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code
Research Papers
Joy Krishan Das University of Saskatchewan, Saikat Mondal University of Saskatchewan, Chanchal K. Roy University of Saskatchewan, Canada
Pre-print
11:30
15m
Talk
FSECAM: A Contextual Thematic Approach for Linking Feature to Multi-level Software Architectural Components
Journal First Track
Amit Mondal Associate Professor, Khulna University, Muhammad Mainul Hossain University of Saskatchewan, Chanchal K. Roy University of Saskatchewan, Canada, Banani Roy University of Saskatchewan, Kevin Schneider University of Saskatchewan
11:45
15m
Talk
Evaluating ReLink for Traceability Link Recovery in Practice
Industrial Track
12:00
15m
Talk
Development of Automated Software Design Document Review Methods Using Large Language Models
Industrial Track
12:15
15m
Talk
Experiences on Using Large Language Models to Re-engineer a Legacy System at Volvo Group
Industrial Track
Vanshika Singh North Carolina State University, Caglar Korlu , Onur Orcun , Wesley Assunção North Carolina State University