Testing AI Systems

The 17th edition of the Workshop on Automating Testing (A-TEST 2026) focuses on the challenges of testing AI systems and AI-enabled software.

Testing AI systems is difficult because their behaviour is often probabilistic, context-dependent, and continuously evolving. Existing challenges, such as reproducibility, test coverage, and the definition of suitable test oracles, become significantly harder in AI systems, especially for systems based on generative AI and large language models.

A-TEST 2026 aims to bring together researchers and practitioners from academia and industry to discuss how software testing and quality engineering practices must evolve to address these challenges.

The workshop provides a forum for presenting and discussing novel testing techniques, empirical evaluations, industrial experiences, benchmarks, metrics, tools, and quality assurance approaches for AI-based and AI-enabled systems.

Topics include testing AI systems, test oracles, human-in-the-loop test processes, empirical evaluation, trustworthy AI, AI quality metrics, and education and training for responsible AI testing.

Previous Editions

A-TEST is a well-established workshop series on automated testing and software quality engineering. Since its origins in 2009, the workshop has been successfully co-located with leading software engineering conferences including ESEC/FSE, ASE, ISSTA/ECOOP, ICST, and PROFES.

Recent editions include:

• A-TEST 2025, co-located with ICST 2025 (Naples, Italy)
• A-TEST 2024, co-located with ISSTA/ECOOP 2024 (Vienna, Austria)
• A-TEST 2023, co-located with ASE 2023 (Luxembourg)
• A-TEST 2022, co-located with ESEC/FSE 2022 (Singapore)

More information about previous editions, programs, proceedings, and accepted papers can be found at:

https://a-test.org/

Call for papers

Testing AI Systems

The rapid adoption of AI-based and AI-enabled software systems raises new challenges for software testing and quality assurance. Unlike traditional software, AI systems often exhibit probabilistic behaviour, context-dependent responses, non-determinism, and continuous evolution. These characteristics make it difficult to define suitable test oracles, establish coverage criteria, assess reliability, and ensure trustworthiness.

A-TEST 2026 aims to bring together researchers and practitioners from academia and industry to discuss advances, challenges, and experiences related to testing AI systems and assuring their quality.

Topics of Interest

Topics include, but are not limited to:

• Testing of AI-based and AI-enabled systems
• Testing generative AI applications and large language model based systems
• Testing AI agents and autonomous systems
• Test oracles for AI systems
• Automated evaluation of AI outputs
• Test coverage and adequacy criteria for AI systems
• Robustness, reliability, safety, and trustworthiness testing
• Human-in-the-loop testing processes
• Continuous testing and monitoring of AI systems
• Measurement, metrics, and benchmarks for AI quality
• Empirical studies and industrial experiences
• AI-assisted software testing
• Education and training for AI testing
• Critical thinking and responsible use of AI in testing practice

Submission Categories

We invite submissions in the following categories:

• Full papers: up to 8 pages, describing original and completed research, either empirical or theoretical.
• Work-in-progress papers: up to 4 pages, describing novel, interesting, and promising work in progress.
• Tool papers: up to 4 pages, presenting academic or industrial testing tools.
• Technology transfer papers: up to 4 pages, describing industry-academia cooperation.
• Position papers: up to 2 pages, analysing trends and raising issues of importance intended to stimulate discussion and debate.

All submissions can include 1 page with references and should present original, unpublished work related to the workshop topics.

Please submit your paper in PDF format through EasyChair. Make sure to select the track “A-TEST 2026” during the submission process.

Reviewing and Selection

All submissions will undergo peer review by at least three members of the Program Committee.

Submissions will be evaluated based on:

• Relevance to the workshop topics
• Originality
• Technical quality
• Clarity of presentation
• Potential to stimulate discussion

The final selection will be based on the reviews and discussion among the organizers and the Program Committee.

Contact

For questions regarding the workshop, please contact the organizers.