TCSE logo 
 Sigsoft logo
Sustainability badge
Sat 3 May 2025 14:40 - 14:50 at 214 - Paper Session 3 Chair(s): Chao Peng

Evaluation benchmarks are essential for developing and training language models, providing both comparison and optimization targets. Existing code completion benchmarks, often based on standalone Python functions and unit tests, are overly simplistic, contaminated, and fail to reflect real-world scenarios. In this paper we present ShaderMatch, a novel benchmark for code completion in computer graphics programming. The benchmark is derived from real-world fragment shaders in OpenGL Shading Language (GLSL) from the Shadertoy platform, forming a zero-shot function completion task with 467 function headers and user-written comments as input. Besides, we propose a two-step evaluation metric: static code comparison followed by frame rendering comparison. Additionally, ShaderMatch introduces eight fine-grained labels for deeper insights. We evaluate over 20 open-source code-specific models and highlight notable performance outliers. Results show that even top models fail to generate working code in 31% of cases, highlighting the challenge posed by GLSL, a low-resource language rarely found in pretraining datasets. ShaderMatch provides a well-annotated, extendable dataset for future research. Data, code, leaderboard, and discussions are available at: https://hf.co/spaces/Vipitis/shadermatch.

Sat 3 May

Displayed time zone: Eastern Time (US & Canada) change

14:00 - 15:30
Paper Session 3LLM4Code at 214
Chair(s): Chao Peng ByteDance
14:00
10m
Talk
Mix-of-Language-Experts Architecture for Multilingual Programming
LLM4Code
Yifan Zong University of Waterloo, Yuntian Deng University of Waterloo, Pengyu Nie University of Waterloo
14:10
10m
Talk
Proving the Coding Interview: A Benchmark for Formally Verified Code Generation
LLM4Code
Quinn Dougherty Unaffiliated, Ronak Mehta Unaffiliated
14:20
10m
Talk
LLM-ProS: Analyzing Large Language Models’ Performance in Competitive Problem Solving
LLM4Code
Md Sifat Hossain University of Dhaka, Anika Tabassum University of Dhaka, Md. Fahim Arefin University of Dhaka, Tarannum Shaila Zaman University of Maryland Baltimore County
Media Attached
14:30
10m
Talk
Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysis
LLM4Code
Manish Shetty University of California, Berkeley, Naman Jain University of California, Berkeley, Adwait Godbole University of California, Berkeley, Sanjit A. Seshia University of California, Berkeley, Koushik Sen University of California at Berkeley
14:40
10m
Talk
Evaluating Language Models for Computer Graphics Code Completion
LLM4Code
Jan Kels Heinrich-Heine-Universität Düsseldorf, Abdelhalim Dahou GESIS – Leibniz-Institute for the Social Sciences, Brigitte Mathiak GESIS – Leibniz-Institute for the Social Sciences
Media Attached File Attached
14:50
10m
Talk
From Zero to Sixty at the Speed of RAG: Improving YAML Recipe Generation via Retrieval
LLM4Code
Farima Farmahinifarahani J.P. Morgan AI Research, Petr Babkin J.P. Morgan AI Research, Salwa Alamir J.P. Morgan AI Research, Xiaomo Liu J.P. Morgan AI Research
15:00
10m
Talk
SC-Bench: A Large-Scale Dataset for Smart Contract Auditing
LLM4Code
Shihao Xia The Pennsylvania State University, Mengting He The Pennsylvania State University, Linhai Song The Pennsylvania State University, Yiying Zhang University of California San Diego
15:10
10m
Talk
METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries
LLM4Code
Hyunseok Lee KAIST, Gabin An KAIST, Shin Yoo KAIST
Pre-print
15:20
10m
Talk
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
LLM4Code
Jinjun Peng Columbia University, Leyi Cui Columbia University, Kele Huang Columbia University, Junfeng Yang Columbia University, Baishakhi Ray Columbia University
Hide past events
:
:
:
: