IEEE AITest 2025 is the seventh edition of the IEEE series conference, focusing on the synergy of artificial intelligence (AI) and software testing. This conference provides an international forum for researchers and practitioners to exchange novel research results, articulate the problems and challenges from practices, deepen our understanding of the subject area with new theories, methodologies, techniques, process models, impacts, etc., and improve the practices with new tools and resources. This year’s conference is scheduled in Tucson, Arizona, USA, from 21-24 July 2025. The conference is part of the IEEE CISOSE 2025 congress.

Topics of Interest

Topics for IEEE AITest 2025 encompass key methodologies for verifying and validating AI systems, along with the innovative use of AI for software testing. They also address the challenges and emerging areas in large language models, data quality, policy, domain-specific AI testing, and the ethical implications of responsible AI development.

For specific topics, please refer to the Call for Paper Page.

News

The second round of paper submission begins! The deadline is April 30th! –04/09/2025
The deadline for the first round of paper submission has been extended to April 8th! –03/30/2025
New workshop – Challenges and Innovations in Large Language Models: Reasoning, Architecture, and Beyond – 03/26/2025
Call for Workshops and Tutorials! – 03/01/2025
Call for Panels! – 03/01/2025
Call for Papers! – 01/01/2025

Contact

For more information and any questions, please contact the AITest 2025 PC chairs:

Monowar Bhuyan, monowar@cs.umu.se, Umea University
Haihua Chen, Haihua.Chen@unt.edu, University of North Texas

Challenges and Innovations in Large Language Models: Reasoning, Architecture, and Beyond

Workshop Description

Duration: One day (tentative)
Date: July 23-24, 2025 (tentative)

1. Overview and Motivation

Large Language Models (LLMs), particularly those leveraging Transformer-based architectures, have achieved significant milestones across a variety of NLP tasks such as text generation, summarization, and machine translation. However, these models are not without notable limitations:

Mathematical Reasoning Challenges: LLMs often display inconsistencies in arithmetic and algebraic reasoning, especially when applying chain-of-thought techniques.
Architectural Bottlenecks: The Transformer’s self-attention mechanism introduces computational inefficiencies, hindering the effective processing of long sequences.
Long-Form Generation Difficulties: Generating contextually coherent and factually consistent long-form content remains a key issue due to limitations in memory and context management.

This workshop seeks to unite researchers and industry practitioners to address these challenges. It will focus on sharing innovative research, proposing novel methodologies, and fostering interdisciplinary discussions across NLP, cognitive science, and machine learning.

2. Scope and Objectives

The primary objectives of this workshop are to:

Identify Challenges: Explore key limitations in reasoning, generation, and architecture within current LLMs.
Investigate Novel Architectures: Highlight emerging models that go beyond traditional Transformers to improve scalability and context handling.
Improve Reasoning Methods: Examine advanced reasoning techniques, such as refined chain-of-thought models, to enhance logical and mathematical consistency.
Propose Innovative Solutions: Encourage discussions on hybrid systems, retrieval-augmented models, memory-augmented architectures, and efficient computation techniques.
Advance Evaluation Benchmarks: Develop or improve metrics and datasets to better assess reasoning, factual accuracy, and long-form coherence.

Workshop Structure

Format: One-day workshop
Planned Agenda:
- One keynote session
- One technical paper session
- Open Q&A / Panel discussion

Topical Outline

We invite contributions on (but not limited to) the following topics:

Mathematical Reasoning and Computation

Enhancing numerical and logical accuracy in LLM outputs
Techniques for step-by-step reasoning and error mitigation
Data augmentation and specialized pre-training for mathematical reasoning

Transformer Limitations and Architectural Innovations

Novel models addressing long-sequence processing (e.g., efficient Transformers, recurrent-transformer hybrids, state-space models)
Reducing memory and computational overhead in large-scale models
Enhancing attention mechanisms for improved context retention

Long-Form Generation and Coherence

Approaches to maintain narrative coherence and factual accuracy in long texts
Metrics and datasets for evaluating document-level consistency
Hierarchical and content-planning approaches for structured generation

Interpretability and Debugging of LLMs

Visualizing model internals: attention, embeddings, and reasoning chains
Detecting and mitigating hallucinations, biases, and logical fallacies
Developing user-friendly tools for debugging LLMs

Hybrid Systems and Multimodal Reasoning

Combining symbolic reasoning, external memory, and neural networks
Cross-modal models integrating text with visual, audio, or other data modalities
Applications requiring reasoning over structured and unstructured information

Ethical and Societal Considerations

Addressing bias, misinformation, and risks in deploying advanced LLMs
Policy and regulatory frameworks for responsible LLM deployment
Developing community guidelines and best practices for safe AI usage

Submission

Submission Link

Important Dates

April 30th, 2025 Submission deadline
May 10th, 2025 Author’s notification
June 1st, 2025 Camera-ready and author’s registration

Anticipated Audience

We anticipate a hybrid event with approximately:

20 in-person participants
20 online attendees

Expected Outcomes

All accepted papers will be published in the official conference proceedings.
Selected papers will be invited to a special issue of Electronics on “Advances in Generative AI and Computational Linguistics.”

Short Biographies and Contact Information of the Organizers

Dr. Yang Zhang

Yang Zhang received his Ph.D. in Computer Science from the School of Computing at Macquarie University, Sydney, New South Wales, Australia. Dr. Yang Zhang serves as a Clinical Assistant Professor in the Anuradha and Vikas Sinha Department of Data Science at the University of North Texas, Denton, Texas, USA. Yang’s research primarily focuses on the subdomains of the emerging AI paradigm, with a special emphasis on text mining. Currently, Yang’s research focuses on deploying Large Language Models (LLMs) for sophisticated text processing, with particular attention to Reasoning and Retrieval-Augmented Generation (RAG), which are essential for innovation and efficiency in NLP. His research interests also include AIoT, Web service, Recommender Systems, and Big data privacy and cybersecurity. His research has been published in several recognized venues (e.g., CIKM2024, ICDM2024, UBICOMP2024, and INFOCOM2025).
Email: Yang.zhang@unt.edu

Dr. Taotao Cai

Taotao Cai is a Senior Lecturer in Computing at the University of Southern Queensland (UniSQ). His primary focus is on research in the field of Data Science, including graph data processing, social network analytics, recommendation systems, and complexity science. He completed his Ph.D. degree from Deakin University in 2020. Prior to joining the faculty at UniSQ, Taotao held positions as a Postdoctoral Research Fellow at Macquarie University (Mar 2021 - Jan 2023) and an Associate Research Fellow at Deakin University (July 2020 - Feb 2021). During this time, he made significant research contributions, which have been published in leading international conferences and journals such as IEEE ICDE, Information Systems, and IEEE TKDE.
Email: Taotao.Cai@unisq.edu.au

Dr. Yipeng Zhou

Yipeng Zhou is a senior lecturer with the School of Computing and secretary of the Centre of Frontier AI Research (FAIR), Faculty of Science and Engineering, Macquarie University. Before joining Macquarie University, he was a research fellow with the University of South Australia, and a lecturer with Shenzhen University, respectively. He got his Ph.D. and M.Phil degrees from The Chinese University of Hong Kong, and a B.S. degree from the University of Science and Technology of China. He received the 2023 Macquarie University Vice-Chancellor’s Research Excellence Award and the 2023 IEEE Open Journal of the Communications Society Best Editor Award. He was the recipient of the 2018 Australia Research Council Discover Early Career Researcher Award (DECRA). His research interests lie in federated learning, data privacy-preservation, networking, etc. He has published 130+ papers in top venues, including ICML, INFOCOM, IJCAI, AAAI, TheWebConf, ICNP, ToN, TDSC, JSAC, TPDS, TMC, etc.
Email: yipeng.zhou@mq.edu.au

Dr. Xuan Lu

Xuan Lu is a tenure-track assistant professor at the College of Information Science, at the University of Arizona. Her research focuses on creating novel methodologies of human-centered data science and using them to understand and optimize the activities and outcomes of our future human society, especially those triggered by technological innovations, with a recent emphasis on the domain of the Future of Work. Her work has been published in multiple leading conferences and journals in the field of data science, and she is a recipient of the WWW Best Paper Award (2019) and the Microsoft Research Asia Fellowship (2017).
Email: luxuan@arizona.edu

Dr. Wei Ni

Wei Ni received the B.E. and Ph.D. degrees in Electronic Engineering from Fudan University, Shanghai, China, in 2000 and 2005, respectively. He is a Principal Research Scientist at CSIRO, Sydney, Australia. He is also a Conjoint Professor at the University of New South Wales, an Adjunct Professor at the University of Technology Sydney, and an Honorary Professor at Macquarie University. He serves as a Technical Expert at Standards Australia in support of the ISO standardization of AI and Big Data. Previously, he was a Postdoctoral Research Fellow at Shanghai Jiaotong University (2005–2008); Deputy Project Manager at Bell Labs, Alcatel/Alcatel-Lucent (2005–2008); and Senior Researcher at Devices R&D, Nokia (2008–2009). He has co-authored one book, ten book chapters, more than 300 journal papers, 100+ conference papers, 26 patents, ten standard proposals accepted by IEEE, and three technical contributions accepted by ISO. His research interests include 6G security and privacy, machine learning, stochastic optimization, and their applications to system efficiency and integrity. Dr. Ni has been an Editor of IEEE Transactions on Wireless Communications since 2018, IEEE Transactions on Vehicular Technology since 2022, IEEE Transactions on Information Forensics and Security since 2024, IEEE Communications Surveys and Tutorials since 2024, and Cambridge Press New Research Directions: Cyber-Physical Systems since 2022.
Email: Wei.Ni@data61.csiro.au

Distinguished Professor Michael Sheng

Quan Z. Sheng is a Distinguished Professor and Head of School of Computing at Macquarie University, Sydney, Australia. His research interests include Service-oriented Computing, Distributed Computing, Internet Computing, and the Internet of Things (IoT). Michael holds a Ph.D. degree in computer science from the University of New South Wales (UNSW) and performed his postdoc as a research scientist at CSIRO ICT Centre. He has more than 400 publications. Prof. Sheng is the recipient of the AMiner Most Influential Scholar Award in IoT in 2019, an ARC (Australian Research Council) Future Fellowship in 2014, the Chris Wallace Award for Outstanding Research Contribution in 2012, and the Microsoft Research Fellowship in 2003. He is ranked by Microsoft Academic as one of the Most Impactful Authors in Services Computing (ranked 4th all time) and in Web of Things (Top 20 all time) in 2021. Prof. Sheng is the Vice Chair of the Executive Committee of the IEEE Technical Community on Services Computing (IEEE TCSVC), the Associate Director (Smart Technologies) of Macquarie University Smart Green Cities Research Centre, and a member of the ACS (Australian Computer Society) Technical Advisory Board on IoT.
Email: michael.sheng@mq.edu.au

Topics of Interest

Topics of interest include, but are not limited to:

1. Testing of AI

Methodologies, theories, techniques, and tools for testing, verification, and validation of AI
Test Oracle for testing AI
Tools and resources for automated testing of AI
Techniques for testing deep neural network learning, reinforcement learning, and graph learning

2. AI for Software Testing

AI techniques to software testing
AI applications to software testing
Human testers and AI-based testing
Crowdsourcing and swarm intelligence in software testing
Genetic algorithms, search-based techniques, and heuristics to optimize testing
Constraint programming for test case generation and test suite reduction
Constraint scheduling and optimization for test case prioritization and test execution scheduling

3. Large Language Models (LLMs)

Testing of Large Language Models (LLMs)
Quality evaluation and assurance for LLMs
LLMs for software engineering and testing
Fairness, ethics, bias, and trustworthiness for LLM applications

4. Data Quality and Policy

Data quality and validation for AI
Quality assurance for unstructured training data
Large-scale unstructured data quality certification
AI and data management policies

5. Domain-Specific Testing

Specific concerns of testing with domain-specific AI
Computer Vision Testing
Intelligent Chatbot Testing
Smart Machine (Robot/AV/UAV) Testing
Impact of GAI on education
Responsible AI testing

Important Dates

Main paper

~~April 1st, 2025 Submission deadline~~
April 8th, 2025 first round submission deadline
April 30th, 2025 second round submission deadline
May 10th, 2025 Author’s notification
June 1st, 2025 Camera-ready and author’s registration

Workshop paper

May 15th, 2025 Submission deadline
May 22nd, 2025 Author’s notification
June 1st, 2025 Camera-ready and author’s registration

Submission

Page Limits

Submit original manuscripts (not published or submitted elsewhere) with the following page limits:

regular papers (8 pages)
short papers (4 pages)
AI testing in practice (8 pages)
tool demo track (6 pages).

Content

We welcome submissions of both regular research papers that describe original and significant work or reports on case studies and empirical research and short papers that describe late-breaking research results or work in progress with timely and innovative ideas. The AI Testing in Practice Track provides a forum for networking, exchanging ideas, and innovative or experimental practices to address SE research that directly impacts the practice of software testing for AI. The tool track provides a forum to present and demonstrate innovative tools and/or new benchmarking datasets in the context of software testing for AI.

Formats and Submission Instructions

All papers must be written in English. Papers must include a title, an abstract, and a list of 4-6 keywords.
All types of papers can have 2 extra pages subject to page charges.
All papers must be prepared in the IEEE double-column proceedings format
Authors must submit their manuscripts via easychair IEEE AI Test 2025 by April 8, 2025, 23:59 AoE. at the latest.

For more information, please visit the conference website. The use of content generated by AI in an article (including but not limited to text, figures, images, and code) shall be disclosed in the acknowledgments section of the submitted article.

Conference Proceedings & Special Section of SCI journals

All accepted papers will be published by IEEE Computer Society Press (EI-Index) and included in the IEEE Digital Library.
The best papers will be invited to submit an extended version (with at least 30% novel content) to the selected special issues (TBA).

Aims and Scope

The 2025 IEEE International Conference on Artificial Intelligence Testing (AITest) is pleased to invite proposals for workshops and tutorials. Workshops are intended to bring together communities of interest, both in established communities and in communities interested in the discussion and exploration of a new or emerging issue. They can range in format from formal, perhaps centering on the presentation of refereed papers, to informal, perhaps centering on an extended roundtable discussion among the selected participants.

Tutorials provide an opportunity to offer in-depth education on a topic or solution relevant to research or practice in AI Testing. They should address a single topic in detail. They are not intended to be venues for commercial product training. Workshops and tutorials can span a half-day or a full day.

Proposal Format

Successful proposals should include:

The title of the workshop or tutorial
A description of the workshop/tutorial, including whether it is planned as a half-day or full-day event
A topical outline of the workshop/tutorial
Anticipated audience for the workshop/tutorial
Expected outcomes of the workshop/tutorial
Short biographies and contact information of the proposers

Submission Guidelines

All submissions must be in English, in PDF format, in the current IEEE double-column proceedings format, and up to 2 pages. Suitable LaTeX, Word, and Overleaf templates are available from the IEEE Website.

All proposals should be submitted by email to:

(Subject: “IEEE AITest 2025 Workshop/Tutorial proposal”)

Organization and Logistics

All workshops and tutorials are planned as in-person events. However, hybrid events (with at least one organizer present in person) can also be accommodated. Please indicate your plan and preference in your proposal.

Important Dates

All dates are Anywhere on Earth (AoE)

April 1, 2025 – Deadline for submission
April 7, 2025 – Notification of acceptance
May 15, 2025 – Workshop paper submission deadline
May 22, 2025 – Workshop paper author’s notification
July 23-24, 2025 – Workshop or Tutorial date (tentative)

Contact

For more information and any questions, please contact the AITest 2025 PC chairs:

Monowar Bhuyan, monowar@cs.umu.se, Umeå University
Haihua Chen, Haihua.Chen@unt.edu, University of North Texas

Aims and Scope

Panels at the 2025 IEEE International Conference on Artificial Intelligence Testing (AITest) are intended to draw together communities of interest, including testing AI systems, using AI techniques for software testing, as well as those involving emerging issues or techniques of interest to members of the community at large.

The panels typically last about 60–90 minutes and include an extended round-table discussion among the selected participants and the audience members. All proposals are welcome to suggest panel formats that will engage and inform the audience and, if accepted, AITest 2025 will work to provide appropriate facilities and setups to enable the panel techniques. Panels can be comprised of short position statements followed by discussion or can be structured as conversations that engage audience members from the outset. While topics are open, preference will be given to panels that align with the topics of interest of AITest 2025.

Proposal Format

Submissions should include:

A statement of goals or learning objectives for the panel
An outline for the panel topics
The expected audience and expected number of attendees
A tentative list of panelists and their bios. Please indicate whether the panelists have already been contacted about the panel.
A discussion of any engagement techniques that will require specific physical or technical requirements for the local hosts (e.g., part of the speakers being online). Please note that at least one panelist and/or organizer must be physically on site.
Contact and biographical information about the organizers. (It is possible for organizers to serve as panelists as well, but this is not a requirement.) Note organizers’ prior experience with organizing any similarly themed panel or workshop.