ICSE 2026
Sun 12 - Sat 18 April 2026 Rio de Janeiro, Brazil
Thu 16 Apr 2026 16:15 - 16:30 at Europa II - AI for Software Engineering 18 Chair(s): Moritz Beller

Large Language Models (LLMs) have shown promise in various tasks, yet few benchmarks assess their capabilities in embedded system development. In this paper, we introduce EmbedAgent, a paradigm designed to simulate real-world roles in embedded system development, such as Embedded System Programmer, Architect, and Integrator. This paradigm enables LLMs to be tested in tasks that bridge the gap between digital and physical systems, allowing for a more comprehensive assessment of their capabilities. To evaluate LLMs on these tasks, we propose EmbedBench, the first comprehensive benchmark for embedded system programming, circuit design, and cross-platform migration. EmbedBench consists of 126 cases, covering 9 electronic components across 3 hardware platforms. Through extensive experiments on 10 mainstream LLMs, we uncover several key findings. Surprisingly, despite the simplicity of the cases, DeepSeek-R1 achieves only a 55.6% pass@1 rate when provided with schematic information, and 50.0% when tasked with generating the schematics itself. In the cross-platform migration tasks, LLMs show relatively strong performance with MicroPython on the Raspberry Pi Pico (with the top model achieving 73.8% pass@1), but perform poorly on ESP-IDF, where the best model reaches only 29.4% pass@1. Interestingly, we observe that general-purpose chat LLMs like DeepSeek-V3 often fail to utilize relevant pre-trained knowledge in this domain, while reasoning LLMs tend to overthink and overlook efficient knowledge during pretraining. Based on these insights, we propose two strategies—retrieval augmented generation and compiler feedback-to enhance LLM performance. These strategies result in significant improvements, with Deepseek-R1 reaching a 65.1% pass@1 with correct schematics, and 53.1% without. Additionally, the accuracy of the Arduino to ESP32 migration task improves from 21.4% to 27.8%.

Thu 16 Apr

Displayed time zone: Brasilia, Distrito Federal, Brazil change

16:00 - 17:30
AI for Software Engineering 18Research Track at Europa II
Chair(s): Moritz Beller Meta Platforms, Inc., USA
16:00
15m
Talk
Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study
Research Track
You Wang Zhejiang University, Michael Pradel CISPA Helmholtz Center for Information Security, Zhongxin Liu Zhejiang University
16:15
15m
Talk
EmbedAgent: Benchmarking Large Language Models in Embedded System DevelopmentVirtual Attendance
Research Track
Ruiyang Xu University of Chinese Academy of Sciences, Jialun Cao Hong Kong University of Science and Technology, Mingyuan Wu Southern University of Science and Technology, Wenliang Zhong Institute of Software, Chinese Academy of Sciences, Yaojie Lu Institute of Software, Chinese Academy of Sciences, Ben He University of Chinese Academy of Sciences, Xianpei Han Institute of Software, Chinese Academy of Sciences, Shing-Chi Cheung Hong Kong University of Science and Technology, Le Sun Institute of Software, Chinese Academy of Sciences
Media Attached
16:30
15m
Talk
When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions
Research Track
Maya LARBI University of Luxembourg, Amal Akli University of Luxembourg, Mike Papadakis University of Luxembourg, Rihab BOUYOUSFI Ecole nationale Supérieure d’Informatique (ESI), Maxime Cordy University of Luxembourg, Luxembourg, Federica Sarro University College London, Yves Le Traon University of Luxembourg, Luxembourg
Pre-print
16:45
15m
Talk
Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies
Research Track
Florian Angermeir fortiss, Maximilian Amougou fortiss GmbH, Mark Kreitz University of the Bundeswehr Munich, Andreas Bauer Technische Hochschule Nürnberg Georg Simon Ohm, Matthias Linhuber Technical University Munich, Davide Fucci Blekinge Institute of Technology, Fabiola Moyón Siemens Technology and Technical University of Munich, Daniel Mendez Blekinge Institute of Technology and fortiss, Tony Gorschek Blekinge Institute of Technology / DocEngineering
DOI Pre-print
17:00
15m
Talk
FreshBrew: A Benchmark for Evaluating AI Agents on Java Code MigrationVirtual Attendance
Research Track
Victor May Google, Diganta Misra Max Planck Institut für Intelligente Systeme (MPI-IS) and ELLIS Institute, Tübingen, Yanqi Luo Salesforce, Anjali Sridhar Google, Justine Gehring Gologic, Silvio Soares Ribeiro Junior Google
17:15
15m
Talk
ProxyWar: Dynamic Assessment of LLM Code Generation in Game ArenasDistinguished Paper Award
Research Track
Xinyu Wang The University of Adelaide, Wenjun Peng The University of Adelaide, Qi Wu University of Adelaide