Data Annotation Errors in AI-Enabled Perception System Development: A Multi-Organisation Case Study in the Automotive DomainFull Paper
Data annotation is a critical yet error-prone activity in developing AI-enabled perception systems (AIePS) for automated driving. Annotation quality directly affects AI model development performance, safety and reliability. Yet, there is little empirically grounded understanding of how annotation errors arise and propagate across the multi-organisational automotive supply chain. This study investigates the types, causes, and effects of data annotation errors through a multi-organisation case study involving six companies (e.g., technology and AI companies that develop software for ad- advanced driver assistance systems and self-driving cars) and four research institutes in Europe and the UK. We conducted 19 semi-structured interviews with 20 experts (≈ 50 hours of transcripts) and applied a six-phase thematic analysis. The resulting data annotation errors taxonomy identifies 18 recurring error types across three major data-quality dimensions: completeness ( attribute omission, missing feedback loop, privacy/compliance omission, edge-case omission, selection bias, sensor synchronisation issues), accuracy (wrong class label, bounding-box errors, granularity mismatch, insufficient guidance, bias-driven errors), and consistency (inter-annotator disagreement, ambiguous instructions, lack of purpose knowledge, misaligned hand-offs, limited review and logging, lack of frameworks and standards, cross-modality misalignment). For re- search rigour and triangulation, we further validated the proposed taxonomy of data annotation errors with industry practitioners, confirming its usefulness for data annotation errors root-cause analysis, supplier quality reviews, new projects onboarding, and data annotation guideline optimisation. Practitioners described it as a “failure-mode catalogue” comparable to FMEA (Failure Mode and Effects Analysis). By framing annotation quality as an AI-enabled system development lifecycle and supply-chain concern, this work advances SE4AI by providing a shared vocabulary, diagnostic check-list, and actionable guidance for trustworthy AIePS development.