Multilingual Code Evaluation with LLM-as-a-Judge: AI-Assisted Feedback for Human-Centric Understanding
The rapid deployment of large language models (LLMs) in software development has created an urgent demand for reliable evaluation of code quality beyond functional correctness. Current human assessments of LLM-generated code across non-functional dimensions, such as Readability, Modularity, Style, Understandability, and Effort, often suffer from low reproducibility because of ambiguous guidelines, which lead to conflicting judgments and undermine benchmark reliability, thereby slowing progress in aligning LLM outputs with human expectations. To address this problem, we introduce a controlled multi-language study with rubric-guided, AI-assisted feedback designed to clarify guideline interpretation while preserving rater autonomy. Across Python, Java, JavaScript, Go, and Swift, we collected over 2,400 ratings under two conditions: No-AI (baseline) and AI-assisted (rubric-tuned feedback). Results show that AI-assisted feedback improves inter-rater reliability (Cronbach’s α α increases by up to +0.19 in Python and Swift), strengthens dimensional coherence (e.g., Readability–Understandability ρ = 0.85 ρ=0.85), and produces more structured latent rating distributions via PCA and clustering, thereby reducing annotation variance. Crucially, these gains stem from the resolution of ambiguity-driven inconsistencies, while preserving the valuable diversity of judgment, demonstrating that rubric-aligned AI-assisted feedback can transform fragmented assessments into a reproducible and scalable framework for the trustworthy study of LLM-generated code quality.