ICSE 2026
Sun 12 - Sat 18 April 2026 Rio de Janeiro, Brazil
Sun 12 Apr 2026 15:12 - 15:30 at Bora Bora I - Session 3 AI-SQE

Contemporary code-generation benchmarks often report high-accuracy metrics that substantially diverge from empirical performance in enterprise environments. This study documents a systematic 51.8% performance differential between public benchmark assessments (mean 74.1% across HumanEval and MBPP) and private repository task evaluations (22.3% aggregate performance). This disparity stems from a fundamental architectural misalignment: public benchmarks predominantly assess isolated, algorithmically-oriented problems, whereas enterprise software development requires repository-level understanding, cross-file dependency resolution, and domain-specific pattern recognition. We present a comprehensive triadic evaluation framework that integrates public benchmark performance, private repository task assessment, and complexity-aware scoring mechanisms to characterize enterprise deployment readiness accurately. Baseline evaluation across two distinct model architectures—code-specialized Qwen2.5-Coder-7B-Instruct and general-purpose Qwen3-8B—reveals that general-purpose architectures exhibit substantially degraded functionality on private repository tasks (2.8% in the evaluated dataset) despite demonstrating respectable public benchmark performance (63.4% on HumanEval). To address this performance gap, we introduce a joint optimization framework that combines Cross-Entropy and Margin Loss objectives to optimize generation quality and semantic alignment simultaneously. Empirical results demonstrate architecture-dependent optimal configurations: code-specialized models achieve peak performance at 50:50 CE:ML weight allocation (+32.5% aggregate improvement, +18% functionality enhancement). In contrast, general-purpose models require near-exclusive margin emphasis at 1:99 weight distribution (+41% aggregate improvement, +320.28% functionality enhancement). This triadic evaluation methodology and architectural sensitivity analysis provide practitioners with a systematic framework for developing enterprise-grade code generation systems.