Annotation Requirements to Annotation Errors: A Causal Modeling Analysis for AI-enabled Perception Systems Development
Context: Data annotation is a foundational software engineering activity in the development of AI-enabled perception systems (AIePS) for safety-critical automotive domains. Despite quality assurance, recurring data annotation errors (DAEs) still degrade robustness, safety, and trustworthiness. Prior work often treats DAEs as execution-level defects, leaving their upstream, requirements-level origins in data annotation requirements (DARs) underexplained.
Objective: This study conducts a causal analysis of how DAR challenges propagate into DAEs across completeness, accuracy, and consistency.
Method: We apply an integrative empirical synthesis grounded in a qualitative interview study ($>$50 hours transcripts; 19 interviews, 20 participants) spanning the automotive AI supply chain. We evaluate five DAR challenges (edge-case coverage gaps, ambiguity, evolution, misalignment, and resource limitations) and an empirically grounded taxonomy of 18 recurring DAE types. We then construct theory-informed Directed Acyclic Graphs (DAGs) that trace propagation paths from DAR challenges through intermediate mechanisms to downstream DAEs. We release an interactive visualisation tool and an open dataset to facilitate the replication and practical use of DAGs.
Results: The DAGs reveal systematic multi-step propagation. Edge-case coverage gaps split into complexity-driven accuracy/consistency degradation (semantic–spatial confusion, disagreement) and definition-driven completeness degradation (missing triage, biased sampling). Ambiguous DARs distort labeling decisions, increasing wrong labels, bounding-box errors, granularity mismatch, bias-driven errors, and inter-annotator disagreement. Evolving DARs and misalignment sustain cross-batch drift via taxonomy/detail inconsistencies, hand-off rule drift, weak traceability, and cross-modality misalignment. Resource limitations amplify all pathways by weakening QA, feedback, and auditability.
Conclusion: DAEs are structural consequences of DAR weaknesses and governance constraints, motivating proactive DAR governance to stabilise specifications, improve hand-offs, and strengthen traceability and verification.
Fri 21 AugDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:30 | Requirements for AI/ML SystemsResearch Papers at A-1600 Chair(s): Fabiano Dalpiaz Utrecht University | ||
11:00 30mTalk | Annotation Requirements to Annotation Errors: A Causal Modeling Analysis for AI-enabled Perception Systems Development Research Papers Hina Saeeda Chalmers University Sweden, Jinlu Yu Chalmers University of Technology, Tommy Johansson Kognic AB Sweden, Eric Knauss Chalmers | University of Gothenburg, Fredrik Warg RISE Research Institutes of Sweden | ||
11:30 30mTalk | From Failure to Alignment: A Requirements Engineering Framework for Machine Learning Systems Research Papers Amel Bennaceur The Open University, UK, Gopi Rajbahadur Queen’s University, Prince Mercy University of Limerick, Bashar Nuseibeh City St George’s, University of London, Faeq Alrimawi Lero - the Science Foundation Ireland Research Centre for Software Pre-print | ||
12:00 30mTalk | From Business Problems to AI Solutions: Where Does Transformation Support Fail? Research Papers Abir Trabelsi École de Technologie Supérieure, Imen Benzarti ETS Montreal, University of Quebec, Hafedh Mili Université du Québec à Montréal, Darine Ameyed Université du Québec à Chicotimi Pre-print | ||
Main Auditorium