EmbedAgent: Benchmarking Large Language Models in Embedded System Development
Large Language Models (LLMs) have shown promise in various tasks, yet few benchmarks assess their capabilities in embedded system development. In this paper, we introduce EmbedAgent, a paradigm designed to simulate real-world roles in embedded system development, such as Embedded System Programmer, Architect, and Integrator. This paradigm enables LLMs to be tested in tasks that bridge the gap between digital and physical systems, allowing for a more comprehensive assessment of their capabilities. To evaluate LLMs on these tasks, we propose EmbedBench, the first comprehensive benchmark for embedded system programming, circuit design, and cross-platform migration. EmbedBench consists of 126 cases, covering 9 electronic components across 3 hardware platforms. Through extensive experiments on 10 mainstream LLMs, we uncover several key findings. Surprisingly, despite the simplicity of the cases, DeepSeek-R1 achieves only a 55.6% pass@1 rate when provided with schematic information, and 50.0% when tasked with generating the schematics itself. In the cross-platform migration tasks, LLMs show relatively strong performance with MicroPython on the Raspberry Pi Pico (with the top model achieving 73.8% pass@1), but perform poorly on ESP-IDF, where the best model reaches only 29.4% pass@1. Interestingly, we observe that general-purpose chat LLMs like DeepSeek-V3 often fail to utilize relevant pre-trained knowledge in this domain, while reasoning LLMs tend to overthink and overlook efficient knowledge during pretraining. Based on these insights, we propose two strategies—retrieval augmented generation and compiler feedback-to enhance LLM performance. These strategies result in significant improvements, with Deepseek-R1 reaching a 65.1% pass@1 with correct schematics, and 53.1% without. Additionally, the accuracy of the Arduino to ESP32 migration task improves from 21.4% to 27.8%.
Thu 16 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 17:30 | AI for Software Engineering 18Research Track at Europa II Chair(s): Moritz Beller Meta Platforms, Inc., USA | ||
16:00 15mTalk | Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study Research Track You Wang Zhejiang University, Michael Pradel CISPA Helmholtz Center for Information Security, Zhongxin Liu Zhejiang University | ||
16:15 15mTalk | EmbedAgent: Benchmarking Large Language Models in Embedded System Development Research Track Ruiyang Xu University of Chinese Academy of Sciences, Jialun Cao Hong Kong University of Science and Technology, Mingyuan Wu Southern University of Science and Technology, Wenliang Zhong Institute of Software, Chinese Academy of Sciences, Yaojie Lu Institute of Software, Chinese Academy of Sciences, Ben He University of Chinese Academy of Sciences, Xianpei Han Institute of Software, Chinese Academy of Sciences, Shing-Chi Cheung Hong Kong University of Science and Technology, Le Sun Institute of Software, Chinese Academy of Sciences Media Attached | ||
16:30 15mTalk | When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions Research Track Maya LARBI University of Luxembourg, Amal Akli University of Luxembourg, Mike Papadakis University of Luxembourg, Rihab BOUYOUSFI Ecole nationale Supérieure d’Informatique (ESI), Maxime Cordy University of Luxembourg, Luxembourg, Federica Sarro University College London, Yves Le Traon University of Luxembourg, Luxembourg Pre-print | ||
16:45 15mTalk | Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies Research Track Florian Angermeir fortiss, Maximilian Amougou fortiss GmbH, Mark Kreitz University of the Bundeswehr Munich, Andreas Bauer Technische Hochschule Nürnberg Georg Simon Ohm, Matthias Linhuber Technical University Munich, Davide Fucci Blekinge Institute of Technology, Fabiola Moyón Siemens Technology and Technical University of Munich, Daniel Mendez Blekinge Institute of Technology and fortiss, Tony Gorschek Blekinge Institute of Technology / DocEngineering DOI Pre-print | ||
17:00 15mTalk | FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration Research Track Victor May Google, Diganta Misra Max Planck Institut für Intelligente Systeme (MPI-IS) and ELLIS Institute, Tübingen, Yanqi Luo Salesforce, Anjali Sridhar Google, Justine Gehring Gologic, Silvio Soares Ribeiro Junior Google | ||
17:15 15mTalk | ProxyWar: Dynamic Assessment of LLM Code Generation in Game ArenasDistinguished Paper Award Research Track Xinyu Wang The University of Adelaide, Wenjun Peng The University of Adelaide, Qi Wu University of Adelaide | ||