TCSE logo 
 Sigsoft logo
Sustainability badge
Sat 3 May 2025 15:10 - 15:20 at 214 - Paper Session 3 Chair(s): Chao Peng

Code documentation can, if written precisely, help developers better understand the code they accompany. However, unlike code, code documentation cannot be automatically verified via execution, potentially leading to inconsistencies between documentation and the actual behavior. While such inconsistencies can harmful for developer’s understanding of the code, checking and finding them remains a costly task due to the involvement of human engineers. This paper proposes MetaMon, which uses an existing search-based test generation technique to capture the current program behavior in the form of test cases, and subsequently uses LLM-based code reasoning to identify the generated regression test oracles that are not consistent with the program specifications in the documentation. MetaMon is supported in this task by metamorphic testing and self-consistency. An empirical evaluation against 9,482 pairs of code documentation and code snippets, generated using five open-source projects from Defects4J v2.0.1, shows that MetaMon can classify the code-and-documentation inconsistencies with the precision of 0.72 and the recall of 0.48.

Sat 3 May

Displayed time zone: Eastern Time (US & Canada) change

14:00 - 15:30
Paper Session 3LLM4Code at 214
Chair(s): Chao Peng ByteDance
14:00
10m
Talk
Mix-of-Language-Experts Architecture for Multilingual Programming
LLM4Code
Yifan Zong University of Waterloo, Yuntian Deng University of Waterloo, Pengyu Nie University of Waterloo
14:10
10m
Talk
Proving the Coding Interview: A Benchmark for Formally Verified Code Generation
LLM4Code
Quinn Dougherty Unaffiliated, Ronak Mehta Unaffiliated
14:20
10m
Talk
LLM-ProS: Analyzing Large Language Models’ Performance in Competitive Problem Solving
LLM4Code
Md Sifat Hossain University of Dhaka, Anika Tabassum University of Dhaka, Md. Fahim Arefin University of Dhaka, Tarannum Shaila Zaman University of Maryland Baltimore County
Media Attached
14:30
10m
Talk
Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysis
LLM4Code
Manish Shetty University of California, Berkeley, Naman Jain University of California, Berkeley, Adwait Godbole University of California, Berkeley, Sanjit A. Seshia University of California, Berkeley, Koushik Sen University of California at Berkeley
14:40
10m
Talk
Evaluating Language Models for Computer Graphics Code Completion
LLM4Code
Jan Kels Heinrich-Heine-Universität Düsseldorf, Abdelhalim Dahou GESIS – Leibniz-Institute for the Social Sciences, Brigitte Mathiak GESIS – Leibniz-Institute for the Social Sciences
Link to publication Media Attached File Attached
14:50
10m
Talk
From Zero to Sixty at the Speed of RAG: Improving YAML Recipe Generation via Retrieval
LLM4Code
Farima Farmahinifarahani J.P. Morgan AI Research, Petr Babkin J.P. Morgan AI Research, Salwa Alamir J.P. Morgan AI Research, Xiaomo Liu J.P. Morgan AI Research
15:00
10m
Talk
SC-Bench: A Large-Scale Dataset for Smart Contract Auditing
LLM4Code
Shihao Xia The Pennsylvania State University, Mengting He The Pennsylvania State University, Linhai Song The Pennsylvania State University, Yiying Zhang University of California San Diego
15:10
10m
Talk
METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries
LLM4Code
Hyunseok Lee KAIST, Gabin An KAIST, Shin Yoo KAIST
Pre-print
15:20
10m
Talk
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
LLM4Code
Jinjun Peng Columbia University, Leyi Cui Columbia University, Kele Huang Columbia University, Junfeng Yang Columbia University, Baishakhi Ray Columbia University
:
:
:
: