Accepted Papers

	Title
	A Combinatorial Testing Approach to Hyperparameter OptimizationDistinguished paper Award Candidate Research and Experience Papers Krishna Khadka, Jaganmohan Chandrasekaran, Jeff Yu Lei, Raghu Kacker, D. Richard Kuhn
	An Exploratory Study of Dataset and Model Management in Open Source Machine Learning ApplicationsDistinguished paper Award Candidate Research and Experience Papers Tajkia Rahman Toma, Cor-Paul Bezemer
	An Exploratory Study of V-Model in Building ML-Enabled Software: A Systems Engineering PerspectiveDistinguished paper Award Candidate Research and Experience Papers Jie JW Wu Pre-print
	Approach for Argumenting Safety on Basis of an Operational Design Domain Research and Experience Papers Gereon Weiss, Marc Zeller, Hannes Schoenhaar, Christian Drabek, Andreas Kreutz
	A Taxonomy of Foundation Model based Systems through the Lens of Software Architecture Research and Experience Papers Qinghua Lu, Liming Zhu, Xiwei (Sherry) Xu, Yue Liu, Zhenchang Xing, Jon Whittle
	Data : Q&A Session Research and Experience Papers
	Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes Research and Experience Papers Gustavo Pinto, Cleidson de Souza, Thayssa Rocha, Igor Steinmacher, Alberto de Souza, Edward Monteiro
	DVC in Open Source AI-development: The Action and the Reaction Research and Experience Papers Lorena Barreto Simedo Pacheco, Musfiqur Rahman, Fazle Rabbi, Pouya Fathollahzadeh, Ahmad Abdellatif, Emad Shihab, Tse-Hsun (Peter) Chen, Jinqiu Yang, Ying Zou
	Energy: Q&A Session Research and Experience Papers
	Engineering Carbon Emission-aware Machine Learning Pipelines Research and Experience Papers Erik Johannes Husom, Sagar Sen, Arda Goknil
	GenAI : Q&A Research and Experience Papers
	Green AI: a Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructures Research and Experience Papers Negar Alizadeh, Fernando Castor
	Green Runner: A tool for efficient deep learning component selection Research and Experience Papers Jai Kannan, Scott Barnett, Anj Simmons, Taylan Selvi, Luís Cruz
	Identifying architectural design decisions for achieving green ML serving Research and Experience Papers Francisco Durán, Silverio Martínez-Fernández, Matias Martinez, Patricia Lago Pre-print
	Investigating the Impact of Solid Design Principles on Machine Learning Code UnderstandingDistinguished paper Award Candidate Research and Experience Papers Raphael Cabral, Marcos Kalinowski, Maria Teresa Baldassarre, Hugo Villamizar, Tatiana Escovedo, Helio Côrtes Vieira Lopes Pre-print
	Is Your Anomaly Detector Ready for Change? Adapting AIOps Solutions to the Real World Research and Experience Papers Lorena Poenaru-Olaru, Natalia Karpova, Luís Cruz, Jan S. Rellermeyer, Arie van Deursen
	Keynote by Pedro Bizarro - To have great machine learning models in production in harsh environments, first focus on the harsh environments Research and Experience Papers
	LLMs for Test Input Generation for Semantic Applications Research and Experience Papers Zafaryab Rasool, Scott Barnett, David Willie, Stefanus Kurniawan, Sherwin Balugo, Srikanth Thudumu, Mohamed Abdelrazek
	ML-On-Rails: Safeguarding Machine Learning Models in Software Systems – A Case Study Research and Experience Papers Hala Abdelkader, Mohamed Abdelrazek, Scott Barnett, Jean-Guy Schneider, Priya Rani, Rajesh Vasa
	Modeling Resilience of Collaborative AI Systems Research and Experience Papers Diaeddin Rimawi, Antonio Liotta, Marco Todescato, Barbara Russo
	Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs Research and Experience Papers Ziyu Li, Donghwan Shin
	Novel Contract-based Runtime Explainability Framework for End-to-End Ensemble Machine Learning Serving Research and Experience Papers Minh-Tri Nguyen, Hong-Linh Truong, Tram Truong-Huu
	Opening Research and Experience Papers
	POLARIS: A framework to guide the development of Trustworthy AI systems Research and Experience Papers Maria Teresa Baldassarre, Domenico Gigante, Marcos Kalinowski, Azzurra Ragone
	Privacy and Copyright Protection in Generative AI: A Lifecycle Perspective Research and Experience Papers Dawen (David) Zhang, Boming Xia, Yue Liu, Xiwei (Sherry) Xu, Thong Hoang, Zhenchang Xing, Mark Staples, Qinghua Lu, Liming Zhu
	Seven Failure Points When Engineering a Retrieval Augmented Generation System Research and Experience Papers Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, Mohamed Abdelrazek
	System Qualities: Q&A Session Research and Experience Papers
	Test - Q&A Session Research and Experience Papers
	The Impact of Knowledge Distillation on the Performance and Energy Consumption of NLP Models Research and Experience Papers Ye Yuan, Jiacheng Shi, Zongyao Zhang, Kaiwei Chen, Eloise Zhang, Vincenzo Stoico, Ivano Malavolta
	Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI AccountabilityDistinguished paper Award Candidate Research and Experience Papers Boming Xia, Qinghua Lu, Liming Zhu, Sung Une (Sunny) Lee, Yue Liu, Zhenchang Xing Pre-print
	Unmasking Data Secrets: An Empirical Investigation into Data Smells and Their Impact on Data Quality Research and Experience Papers Gilberto Recupito, Raimondo Rapacciuolo, Dario Di Nucci, Fabio Palomba
	Welcome Your New AI Teammate: On Safety Analysis by Leashing Large Language Models Research and Experience Papers Ali Nouri, Beatriz Cabrero-Daniel, Fredrik Torner, Hakan Sivencrona, Christian Berger
	What About the Data? A Mapping Study on Data Engineering for AI Systems Research and Experience Papers Petra Heck Pre-print
	(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs Research and Experience Papers MA Wanqin, Chenyang Yang, Christian Kästner
	Worst-Case Convergence Time of ML Algorithms via Extreme Value Theory Research and Experience Papers A: Saeid Tizpaz-Niari, A: Sriram Sankaranarayanan

Call for Papers

We invite submissions of research and experience papers in two categories:

• Long paper: Long papers are research or experience papers describing research results, case studies, or insights from industry experience. A research or experience full paper is up to 10 pages plus a maximum of 2 pages for references.

• Short paper: Papers describing new challenges, new research results, visionary ideas, or experiences from, or in cooperation with, practitioners are welcome as short papers. In progress research with interim results is also appropriate as a short paper. A short paper is up to 5 pages plus a maximum of 1 page for references. The paper submissions will undergo a review process with three independent reviews and a virtual PC discussion. Acceptance criteria include contribution to the field of software engineering for AI, novelty, research and industrial relevance, soundness, and results. The accepted full and short papers will be published in ACM Proceedings.

Scope and Topics of Interest

The area of interest for CAIN is Software Engineering for AI — improving the development of AI-based systems throughout the full life cycle. Topics include but are not limited to:

• System and software requirements and their relationship to AI/ML modeling.
• Data management ensuring relevance and efficiency related to business goals.
• System and software architecture for AI-enabled systems.
• Integration of AI and software development processes into the AI system development life cycle, including continuous integration and deployment, and system and software evolution.
• Ensuring and managing system and software nonfunctional properties and their relationship to AI/ML properties, including runtime properties such as performance, safety, security, and reliability; and life-cycle properties including reusability, maintainability and evolution.
• Collaboration, organizational, and management practices for a successful development of AI-enabled systems.
• Building effective infrastructures to support development of AI systems and components.

Note: Submissions that report strictly on data science or model development without any connection to software engineering and AI-enabled systems will be desk-rejected. As stated earlier, there are many venues for those papers where authors would get much more valuable and relevant feedback.

Submission Form

Research and experience papers should be submitted to HotCrp. The submission deadline is firm, no extensions.

All submissions must adhere to the following requirements:

• Page limit is 10 pages plus 2 additional pages of references for long papers and 5 pages plus 1 additional page for references for short papers.
• Submissions must be unpublished original work and should not be under review or submitted elsewhere while being under consideration.
• By submitting to CAIN, authors acknowledge that they are aware of and agree to be bound by the ACM Policy and Procedures on Plagiarism and IEEE Plagiarism FAQ. The authors also acknowledge that they conform to the authorship policy of the ACM and the authorship policy of the IEEE.
• Paper review will employ a double-anonymous review process. Thus, no submission may reveal its authors’ identities. The authors must make every effort to honor the double-anonymous review process. In particular:
  • Authors’ names must be omitted from the submitted paper.
  • All references to the author’s prior work should be in the third person.
  • Authors are encouraged to title their submission differently than preprints of the authors on ArXiV or similar sites. During review, authors should not publicly use the submission title.

All authors should use the official “ACM Primary Article Template” that can be obtained from the ACM Proceedings Template page. LaTeX users should use the sigconf option, as well as the review (to produce line numbers for easy reference by the reviewers) and anonymous (omitting author names) options. To that end, the following LaTeX code can be placed at the start of the LaTeX document:

\documentclass[sigconf,review,anonymous]{acmart}
\acmConference[CAIN 2024]{3rd International Conference on AI Engineering — Software Engineering for AI}{April 2024}{Lisbon, Portugal}

Accepted papers will be published in the ICSE 2024 Co-located Event Proceedings and included in the IEEE and ACM Digital Libraries. Authors of accepted papers are required to register and present their accepted paper at the conference in order for the paper to be included in the proceedings and the Digital Libraries.

The official publication date is the date the proceedings are made available in the ACM or IEEE Digital Libraries. This date may be up to two weeks prior to the first day of ICSE 2024. The official publication date affects the deadline for any patent filings related to published work.

Authors of papers receiving a Conditional Accept decision are expected to submit the revised papers with changes marked in a different color, such as using LaTeXdiff. The authors also need to submit an “Author Response” document capturing the authors’ response to each reviewer comment and how those comments were addressed in the revision. This is similar to the “Summary of Changes and Response” document that is typically submitted by authors for a journal paper major revision. The reviewers will check the revised paper against the original paper and the suggested changes. Conditional Accepts will be checked by only one member of the Program Committee, and this will be done in one pass.

Authors of rejected long papers may receive an acceptance as a short paper if the PC chairs and reviewers agree that it better meets the criteria for short papers. In this case, authors may decide to accept or reject the invitation if they would rather submit as a long paper to a different venue.

Similarly, authors of rejected long and short papers relevant to the field of AI Engineering may have their papers sent to a different CAIN track. Also in this case, authors may decide to accept or reject the invitation.

Distinguished Paper Award

Investigating the Impact of SOLID Design Principles on Machine Learning Code Understanding

Authors: Raphael Cabral, Marcos Kalinowski, Maria Teresa Baldassarre, Hugo Villamizar, Tatiana Escovedo , Helio Lopes

Long papers

Engineering Carbon Emission-aware Machine Learning Pipelines

Authors: E. Husom, S. Sen, A. Göknil

POLARIS: A Framework to Guide the Development of Trustworthy AI Systems

Authors: M. Baldassarre, D. Gigante, M. Kalinowski, A. Ragone

Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes

Authors: G. Pinto, C. de Souza, T. Rocha, I. Steinmacher, A. Souza, E. Monteiro

An Exploratory Study of V-Model in Building ML-Enabled Software: A Systems Engineering Perspective

Authors: J. Wu

Is Your Anomaly Detector Ready for Change? Adapting AIOps Solutions to the Real World

Authors: L. Poenaru-Olaru, N. Karpova, L. Cruz, J. Rellermeyer, A. van Deursen

A Combinatorial Approach to Hyperparameter Optimization

Authors: K. Khadka, J. Chandrasekaran, Y. Lei, R. Kacker, D. Kuhn

Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs

Authors: Z. Li, D. Shin

What About the Data? A Mapping Study on Data Engineering for AI Systems

Authors: P. Heck

Investigating the Impact of Solid Design Principles on Machine Learning Code Understanding

Authors: R. Cabral, M. Kalinowski, M. Baldassarre, H. Villamizar, T. Escovedo, H. Lopes

Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI Accountability

Authors: B. Xia, Q. Lu, L. Zhu, S. Lee, Y. Liu, Z. Xing

Unmasking Data Secrets: An Empirical Investigation into Data Smells and Their Impact on Data Quality

Authors: G. Recupito, R. Rapacciuolo, D. Di Nucci, F. Palomba

An Exploratory Study of Dataset and Model Management in Open Source Machine Learning Applications

Authors: T. Toma, C. Bezemer

Novel Contract-based Runtime Explainability Framework for End-to-End Ensemble Machine Learning Serving

Authors: M. Nguyen, H. Truong, T. Truong-Huu

Approach for Argumenting Safety on Basis of an Operational Design Domain

Authors: G. Weiss, M. Zeller, H. Schoenhaar, C. Drabek, A. Kreutz

Worst-Case Convergence Time of ML Algorithms via Extreme Value Theory

Authors: S. Tizpaz-Niari, S. Sankaranarayanan

Short Papers

A Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructures

Authors: N. Alizadeh, F. Castor

Seven Failure Points When Engineering a Retrieval Augmented Generation System

Authors: S. Barnett, S. Kurniawan, S. Thudumu, Z. Brannelly, M. Abdelrazek

LLMs for Test Input Generation for Semantic Applications

Authors: Z. Rasool, S. Barnett, D. Willie, S. Kurniawan, S. Balugo, S. Thudumu, M. Abdelrazek

(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs

Authors: W. Ma, C. Yang, C. Kästner

Modeling Resilience of Collaborative AI Systems

Authors: D. Rimawi, A. Liotta, M. Todescato, B. Russo

ML-On-Rails: Safeguarding Machine Learning Models in Software Systems – A Case Study

Authors: H. Abdelkader, M. Abdelrazek, S. Barnett, J. Schneider, P. Rani, R. Vasa

DVC in Open Source ML-development: The Action and the Reaction

Authors: L. Pacheco, M. Rahman, F. Rabbi, P. Fathollahzadeh, A. Abdellatif, E. Shihab, T. Chen, J. Yang, Y. Zou

Welcome Your New AI Teammate: On Safety Analysis by Leashing Large Language Models

Authors: A. Nouri, B. Cabrero-Daniel, F. Torner, H. Sivencrona, C. Berger

A Taxonomy of Foundation Model based Systems through the Lens of Software Architecture

Authors: Q. Lu, L. Zhu, X. Xu, Y. Liu, Z. Xing, J. Whittle

Green Runner: A Tool for Efficient Deep Learning Component Selection

Authors: J. Kannan, S. Barnett, A. Simmons, T. Selvi, L. Cruz

Privacy and Copyright Protection in Generative AI: A Lifecycle Perspective

Authors: D. Zhang, B. Zia, Y. Liu, X. Xu, T. Hoang, Z. Xing, M. Staples, Q. Lu, L. Zhu

Identifying Architectural Design Decisions for Achieving Green ML Serving

Authors: F. Durán, S. Martínez-Fernández, M. Martinez, P. Lago

The Impact of Knowledge Distillation on the Energy Consumption and Runtime Efficiency of NLP Models

Authors: Y. Yuan, J. Zhang, Z. Zhang, K. Chen, J. Shi, V. Stoico, I. Malavolta