Beyond Public Benchmarks: An LLM-as-Judge Framework for Enterprise Code Evaluation with Margin-Driven Optimization
Contemporary code-generation benchmarks often report high-accuracy metrics that substantially diverge from empirical performance in enterprise environments. This study documents a systematic 51.8% performance differential between public benchmark assessments (mean 74.1% across HumanEval and MBPP) and private repository task evaluations (22.3% aggregate performance). This disparity stems from a fundamental architectural misalignment: public benchmarks predominantly assess isolated, algorithmically-oriented problems, whereas enterprise software development requires repository-level understanding, cross-file dependency resolution, and domain-specific pattern recognition. We present a comprehensive triadic evaluation framework that integrates public benchmark performance, private repository task assessment, and complexity-aware scoring mechanisms to characterize enterprise deployment readiness accurately. Baseline evaluation across two distinct model architectures—code-specialized Qwen2.5-Coder-7B-Instruct and general-purpose Qwen3-8B—reveals that general-purpose architectures exhibit substantially degraded functionality on private repository tasks (2.8% in the evaluated dataset) despite demonstrating respectable public benchmark performance (63.4% on HumanEval). To address this performance gap, we introduce a joint optimization framework that combines Cross-Entropy and Margin Loss objectives to optimize generation quality and semantic alignment simultaneously. Empirical results demonstrate architecture-dependent optimal configurations: code-specialized models achieve peak performance at 50:50 CE:ML weight allocation (+32.5% aggregate improvement, +18% functionality enhancement). In contrast, general-purpose models require near-exclusive margin emphasis at 1:99 weight distribution (+41% aggregate improvement, +320.28% functionality enhancement). This triadic evaluation methodology and architectural sensitivity analysis provide practitioners with a systematic framework for developing enterprise-grade code generation systems.
Sun 12 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
14:00 - 15:30 | |||
14:00 18mOther | Workshop opening AI-SQE | ||
14:18 18mPaper | Evaluating perturbation robustness of generative systems that use COBOL code inputs AI-SQE | ||
14:36 18mPaper | How Good is ChatGPT in Assessing Architecture Diagrams? An Exploratory Study with Four Software Engineering Tools AI-SQE | ||
14:54 18mPaper | Multicalibration for LLM-based Code Generation AI-SQE | ||
15:12 18mPaper | Beyond Public Benchmarks: An LLM-as-Judge Framework for Enterprise Code Evaluation with Margin-Driven Optimization AI-SQE | ||