An LLM-based Approach for Automatic ML Prototype Review
When developing machine learning (ML) solutions, it is crucial to build prototypes that demonstrate the solution’s technical feasibility and potential value. These ML prototypes are typically Jupyter notebooks. However, manually reviewing ML prototypes is time-consuming and can lead to relevant qualities being overlooked from diverse stakeholders’ perspectives.
This paper introduces an innovative approach that uses LLMs to automate the ML prototype review process, thereby improving quality and stakeholder awareness. Through a systematic literature review, we identified key quality characteristics and information needs. The result is an ML prototype review catalog containing a quality model, a list of information needs, and stakeholder personas.
We present Proto-Check, a JupyterLab extension that implements our LLM-based review process. Evaluation results demonstrate high usefulness and usability, as well as heightened developer awareness of stakeholder qualities and needs.
Tue 17 MarDisplayed time zone: Athens change
14:00 - 15:30 | |||
14:15 15mTalk | CVE-Poisoning: Towards Human-Guided and Cost-Effective Detection of a Novel AI Data Poisoning Attack Workshops & Tutorials Norbert Szolnoki Sándor Department of Software Engineering, University of Szeged, Gergő Balogh Department of Software Engineering, University of Szeged, Szabina Herman University of Szeged, Gabor Antal Department of Software Engineering, University of Szeged | ||
14:30 15mTalk | An LLM-based Approach for Automatic ML Prototype Review Workshops & Tutorials Selin Coban Research Group Software Construction RWTH Aachen University, Miguel Perez Research Group Software Construction RWTH Aachen University, Cagatay Akpinar Research Group Software Construction RWTH Aachen University, Baran Tanriverdi Research Group Software Construction RWTH Aachen University, Horst Lichter RWTH Aachen University | ||
14:45 15mTalk | From Threat Reports to Security Knowledge: Building an LLM-based Pipeline for AI Systems Workshops & Tutorials Takuma Tsuchida Waseda University, Yuya Fujiwara Waseda University, Hironori Washizaki Waseda University, Naoyasu Ubayashi Waseda University | ||
15:00 15mTalk | Enhancing Security Requirements Coverage via RAG and Automated Feedback Loops Workshops & Tutorials Giuseppe Sabetta University of Salerno, Alfonso Cannavale University of Salerno, Fabio Palomba University of Salerno, Andrea De Lucia University of Salerno | ||
15:15 15mTalk | Empirical Evaluation of Open Source Large Language Models for Paper Selection: Are LLMs Trustworthy Tools for Scoping Reviews? Workshops & Tutorials Homayoun Safarpour University of Szeged, Gergő Balogh Department of Software Engineering, University of Szeged, Aondowase James Orban | ||