Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML
Large language models have shown impressive performance in various domains, including code generation across diverse open-source domains. However, their applicability in proprietary industrial settings, where domain-specific constraints and code interdependencies are prevalent, remains largely unexplored. We present a case study conducted in collaboration with the leveling department at ASML to investigate the performance of LLMs in generating functional, maintainable code within a closed, highly specialized software environment. We developed an evaluation framework tailored to ASML’s proprietary codebase and introduced a new benchmark. Additionally, we proposed a new evaluation metric, build@k, to assess whether LLM-generated code successfully compiles and integrates within real industrial repositories. We investigate various prompting techniques, compare the performance of generic and code-specific LLMs, and examine the impact of model size on code generation capabilities, using both match-based and execution-based metrics. The findings reveal that prompting techniques and model size have a significant impact on output quality, with few-shot and chain-of-thought prompting yielding the highest build success rates. The difference in performance between the code-specific LLMs and generic LLMs was less pronounced and varied substantially across different model families.
Tue 18 NovDisplayed time zone: Seoul change
16:00 - 17:00 | |||
16:00 10mTalk | Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat Industry Showcase Kexing Ji , Shiyun Fu The Chinese University of Hong Kong, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Yujia Chen The Chinese University of Hong Kong, Zezhou Yang Tencent Inc., Chaozheng Wang The Chinese University of Hong Kong, Yuetang Deng Tencent | ||
16:10 10mTalk | Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML Industry Showcase Yash Mundhra Delft University of Technology, Max Valk ASML, Maliheh Izadi Delft University of Technology | ||
16:20 10mTalk | IntelliTopo: An IaC Generation Service for Industrial Network Topology Construction Industry Showcase Mingyu Shao Harbin Institute of Technology, Shenzhen; PengCheng Laboratory, Zhao Liu PengCheng Laboratory, Weihong Han Peng Cheng Laboratory, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Jiachen Liu Harbin Institute of Technology, Shenzhen, Qing Liao Harbin Institute of Technology | ||
16:30 10mTalk | RepoMasterEval: Evaluating Code Completion via Real-World Repositories Industry Showcase Qinyun Wu Bytedance Ltd., Chao Peng ByteDance, Pengfei Gao ByteDance, Ruida Hu Harbin Institute of Technology, Shenzhen, Haoyu Gan ByteDance, Bo Jiang Bytedance Network Technology, Jinhe Tang ByteDance, Zhiwen Deng ByteDance, Zhanming Guan ByteDance, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Xia Liu ByteDance, Ping Yang Bytedance Network Technology | ||
16:40 10mTalk | Multiple Schema-Conformant Declarative Code Generation NIER | ||
16:50 10mTalk | Tuning LLM-based Code Optimization via Meta-Prompting: An Industrial Perspective Industry Showcase Jingzhi Gong University of Leeds, Rafail Giavrimis Turing Intelligence Technology, Paul Brookes TurinTech AI, Vardan Voskanyan TurinTech AI, Fan Wu TurinTech AI, Mari Ashiga University of West London/TurinTech AI, Matthew Truscott TurinTech AI, Michail Basios Turing Intelligence Technology, Leslie Kanthan TurinTech AI, Jie Xu University of Leeds, Zheng Wang University of Leeds | ||