Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes
In the rapidly advancing field of artificial intelligence, software development has emerged as a key area of innovation. Despite the plethora of \emph{general-purpose} AI assistants available, their effectiveness diminishes in complex, domain-specific scenarios. Noting this limitation, both the academic community and industry players are relying on \emph{contextualized} coding AI assistants. These assistants surpass general AI tools by integrating proprietary, domain-specific knowledge, offering precise and relevant solutions. Our study focuses on the initial experiences of 62 participants that used a contextualized coding AI assistant — named StackSpot AI— in a controlled setting. According to the participants, the assistants’ use resulted in significant time savings, easier access to documentation, and the generation of accurate codes for internal APIs. However, challenges like variable responses and limitations in handling complex codes were observed. The study’s findings, detailing both the benefits and challenges of contextualized AI assistants, underscore their potential in revolutionizing software development practices, while also highlighting areas for further refinement.
Sun 14 AprDisplayed time zone: Lisbon change
16:00 - 18:00 | Generative AI EngineeringIndustry Talks / Research and Experience Papers at Pequeno Auditório Chair(s): Ipek Ozkaya Carnegie Mellon University | ||
16:00 15mTalk | Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes Research and Experience Papers Gustavo Pinto Federal University of Pará (UFPA) and Zup Innovation, Cleidson de Souza Federal University of Pará, Brazil, Thayssa Rocha Zup Innovation & UFPA, Igor Steinmacher Northern Arizona University, Alberto de Souza Zup Innovation, Edward Monteiro StackSpot | ||
16:15 10mTalk | Privacy and Copyright Protection in Generative AI: A Lifecycle Perspective Research and Experience Papers Dawen (David) Zhang CSIRO's Data61, Boming Xia CSIRO's Data61 & University of New South Wales, Yue Liu CSIRO's Data61 & University of New South Wales, Xiwei (Sherry) Xu Data61, CSIRO, Thong Hoang CSIRO's Data61, Zhenchang Xing CSIRO's Data61, Mark Staples CSIRO, Australia, Qinghua Lu Data61, CSIRO, Liming Zhu CSIRO’s Data61 | ||
16:25 10mIndustry talk | Innovating Translation: Lessons Learned from BWX Generative Language Engine Industry Talks | ||
16:35 15mTalk | Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI AccountabilityDistinguished paper Award Candidate Research and Experience Papers Boming Xia CSIRO's Data61 & University of New South Wales, Qinghua Lu Data61, CSIRO, Liming Zhu CSIRO’s Data61, Sung Une (Sunny) Lee CSIRO's Data61, Yue Liu CSIRO's Data61 & University of New South Wales, Zhenchang Xing CSIRO's Data61 Pre-print | ||
16:50 10mLive Q&A | GenAI : Q&A Research and Experience Papers | ||
17:00 60mPanel | Industry Panel Industry Talks |