Quantization Is Not a Dealbreaker: Empirical Insights from Large Code Models
This program is tentative and subject to change.
Large Language Models (LLMs) have showcased exceptional capabilities across a wide range of domains, including Software Engineering (SE). Within this field, Large Code Models (LCMs)—a specialized subset of LLMs tailored to assist with coding tasks—have made significant strides in automating SE-related practices such as bug-fixing, code generation, and code summarization, elevating their effectiveness to unprecedented levels. These models, often feature billions of parameters, deliver outstanding performance but at the expense of substantial memory and computational requirements. The growing scale of LLMs not only demands extensive computational resources but also raises environmental concerns due to their increasing carbon footprint. Model quantization emerges as an effective approach that can reduce the resource demands of LLMs and particularly LCMs by decreasing parameter precision without substantially affecting performance (eg, 16 bit -> 4 bit). While recent studies confirm quantization guarantees code correctness, they provide limited insights into practical considerations, particularly regarding the impact on software quality attributes such as reliability, maintainability, security, and static properties (eg, cyclomatic complexity). Building upon this line of research, our study investigates the impact of quantization on the qualitative aspects of the automatically generated code. To this extent, we apply Activation-aware Weight Quantization (AWQ) to two popular code models–CodeLlama and DeepSeekCoder–to generate Java and Python code. Using advanced static analysis tools, we measure software quality metrics and static features, including cyclomatic complexity, cognitive complexity, and the LoC(Line of Code). Our findings reveal mixed outcomes: quantized models generally produce code that is more complex, longer, and less reliable, yet more maintainable than their full-precision counterparts, with notable variations across different model sizes. These results emphasize that quantization is not a ‘one-size fits all’ technique, highlighting the necessity of taking model-specific factors into account in real-world applications.
This program is tentative and subject to change.
Wed 10 SepDisplayed time zone: Auckland, Wellington change
10:30 - 12:00 | Session 2 - Quality Assurance 1Tool Demonstration Track / Research Papers Track / Industry Track / NIER Track / Journal First Track at Case Room 260-057 Chair(s): Coen De Roover Vrije Universiteit Brussel | ||
10:30 15m | A Jump-Table-Agnostic Switch Recovery on ASTs Research Papers Track | ||
10:45 15m | Quantization Is Not a Dealbreaker: Empirical Insights from Large Code Models Research Papers Track Saima Afrin William & Mary, Antonio Mastropaolo William and Mary, USA, Bowen Xu North Carolina State University Pre-print | ||
11:00 10m | AI-Powered Commit Explorer (APCE) Tool Demonstration Track Yousab Grees Belmont University, Polina Iaremchuk Belmont University, Ramtin Ehsani Drexel University, Esteban Parra Belmont University, Preetha Chatterjee Drexel University, USA, Sonia Haiduc Florida State University | ||
11:10 10m | JDala - A Simple Capability System for Java Tool Demonstration Track Quinten Smit Victoria University of Wellington, Jens Dietrich Victoria University of Wellington, Michael Homer Victoria University of Wellington, Andrew Fawcet Victoria University of Wellington, James Noble Independent. Wellington, NZ | ||
11:20 10m | ExpertCache: GPU-Efficient MoE Inference through Reinforcement Learning-Guided Expert Selection NIER Track Xunzhu Tang University of Luxembourg, Tiezhu Sun University of Luxembourg, Yewei Song University of Luxembourg, SiYuanMa , Jacques Klein University of Luxembourg, Tegawendé F. Bissyandé University of Luxembourg | ||
11:30 15m | Efficient Detection of Intermittent Job Failures Using Few-Shot Learning Industry Track Henri Aïdasso École de technologie supérieure (ÉTS), Francis Bordeleau École de Technologie Supérieure (ETS), Ali Tizghadam TELUS | ||
11:45 15m | LogOW: A Semi-Supervised Log Anomaly Detection Model in Open-World Setting Journal First Track Jingwei Ye Nankai University, Chunbo Liu Civil Aviation University of China, Zhaojun Gu Civil Aviation University of China, Zhikai Zhang Civil Aviation University of China, Xuying Meng The Institute of Computing Technology, Chinese Academy of Sciences, Weiyao Zhang The Institute of Computing Technology, Chinese Academy of Sciences, Yujun Zhang The Institute of Computing Technology, Chinese Academy of Sciences |