Themes of Building LLM-based Applications for Production: A Practitioner's View
This program is tentative and subject to change.
Background: Large language models (LLMs) have become a paramount interest of researchers and practitioners alike, yet a comprehensive overview of key considerations for those developing LLM-based systems is lacking. This study addresses this gap by collecting and mapping the topics practitioners discuss online, offering practical insights into where priorities lie in developing LLM-based applications. Method: We collected 189 videos from 2022 to 2024 from practitioners actively developing such systems and discussing various aspects they encounter during development and deployment of LLMs in production. We analyzed the transcripts using BERTopic, then manually sorted and merged the generated topics into themes, leading to a total of 20 topics in 8 themes. Results: The most prevalent topics fall within the theme Design & Architecture, with a strong focus on retrieval-augmented generation (RAG) systems. Other frequently discussed topics include model capabilities and enhancement techniques (e.g., fine-tuning, prompt engineering), infrastructure and tooling, and risks and ethical challenges. Implications: Our results highlight current discussions and challenges in deploying LLMs in production. This way, we provide a systematic overview of key aspects practitioners should be aware of when developing LLM-based applications. We further pale off topics of interest for academics where further research is needed.
This program is tentative and subject to change.
Sun 27 AprDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | |||
14:00 15mTalk | Themes of Building LLM-based Applications for Production: A Practitioner's View Research and Experience Papers Alina Mailach Leipzig University, Sebastian Simon Leipzig University, Johannes Dorn Leipzig University, Norbert Siegmund Leipzig University | ||
14:15 15mTalk | LLM-Based Safety Case Generation for Baidu Apollo: Are We There Yet? Research and Experience Papers | ||
14:30 15mTalk | An AI-driven Requirements Engineering Framework Tailored for Evaluating AI-Based Software Research and Experience Papers Hamed Barzamini , Fatemeh Nazaritiji Northern Illinois University, Annalise Brockmann Northern Illinois University, Hasan Ferdowsi Northern Illinois university, Mona Rahimi Northern Illinois University | ||
14:46 14mTalk | Engineering LLM Powered Multi-agent Framework for Autonomous CloudOps Research and Experience Papers Kannan Parthasarathy MontyCloud, Karthik Vaidhyanathan IIIT Hyderabad, Rudra Dhar SERC, IIIT Hyderabad, India, Venkat Krishnamachari MontyCloud, Adyansh Kakran International Institute of Information Technology, Hyderabad, Sreemaee Akshathala IIIT Hyderabad, Shrikara Arun IIIT Hyderabad, Amey Karan IIIT Hyderabad, Basil Muhammed MontyCloud, Sumant Dubey MontyCloud, Mohan Veerubhotla MontyCloud | ||
15:00 15mTalk | Generating and Verifying Synthetic Datasets with Requirements Engineering Research and Experience Papers Lynn Vonderhaar Embry-Riddle Aeronautical University, Timothy Elvira Embry-Riddle Aeronautical University, Omar Ochoa Embry-Riddle Aeronautical University | ||
15:15 15mTalk | InsightAI: Root Cause Analysis in Large Hierarchical Log Files with Private Data Using Large Language Models Research and Experience Papers Maryam Ekhlasi Polytechnique Montreal, Anurag Prakash Ciena, Michel Dagenais Polytechnique Montréal, Maxime Lamothe Polytechnique Montreal |