LLM2FedLLM - A Tool for Simulating Federated LLMs for Software Engineering Tasks
The paper introduces LLM2FedLLM, a tool designed for Software Engineering (SE) researchers to simulate fine-tuning Large Language Models (LLMs) within a federated learning (FL) framework. Unlike existing FL frameworks that facilitate real client collaboration, our simulator provides a controlled environment for experimenting with FL scenarios on a single machine.
The LLM2FedLLM Simulator approaches SE code tasks, such as code summarization, code review, and code translation, within a federated learning framework by first partitioning the selected code dataset into heterogeneous subsets for multiple clients. It then fine-tunes the chosen LLM and evaluates its performance against vanilla, centralized, and individual client models using various metrics. The tool supports several federated aggregation methods and PEFT for supervised learning, with the flexibility to easily integrate additional techniques.
The evaluation of our tool on Python code summarization showed that FedLLM performs comparably to centralized models and outperforms individual clients, particularly in low-data scenarios. Our tool aims to facilitate research advances in secure collaborative training simulations within the SE community.
Mon 28 AprDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:30 | Empirical Findings, Future Visions, Recommendations Replications and Negative Results (RENE) / Early Research Achievements (ERA) / Tool Demonstration / Research Track at 205 Chair(s): Mark Hills Appalachian State University, Coen De Roover Vrije Universiteit Brussel, Gema Rodríguez-Pérez University of British Columbia (UBC) | ||
11:00 10mTalk | Terminal Lucidity: Envisioning the Future of the Terminal Research Track Pre-print | ||
11:10 6mTalk | Exploring Code Comprehension in Scientific Programming: Preliminary Insights from Research Scientists Early Research Achievements (ERA) Alyssia Chen University of Hawaii at Manoa, Carol Wong University of Hawaii at Manoa, Bonita Sharif University of Nebraska-Lincoln, USA, Anthony Peruma University of Hawai‘i at Mānoa | ||
11:16 10mTalk | Method Names in Jupyter Notebooks: An Exploratory Study Research Track Carol Wong University of Hawaii at Manoa, Gunnar Larsen University of Hawaii at Manoa, Rocky Huang University of Hawaii at Manoa, Bonita Sharif University of Nebraska-Lincoln, USA, Anthony Peruma University of Hawai‘i at Mānoa | ||
11:26 6mTalk | SCALAR: A Part-of-speech Tagger for Identifiers Tool Demonstration Christian Newman , Brandon Scholten Kent State University, Sophia Testa Kent State University, Joshua Behler Kent State University, Syreen Banabilah Kent State University, Michael L. Collard The University of Akron, Michael J. Decker Bowling Green State University, Mohamed Wiem Mkaouer University of Michigan - Flint, Marcos Zampieri George mason University, Eman Abdullah AlOmar Stevens Institute of Technology, USA, Reem Alsuhaibani Prince Sultan University, Anthony Peruma University of Hawai‘i at Mānoa, Jonathan I. Maletic Kent State University | ||
11:32 6mTalk | How do Papers Make into Machine Learning Frameworks: A Preliminary Study on TensorFlow Early Research Achievements (ERA) Federica Pepe University of Sannio, Claudia Farkas York University, Maleknaz Nayebi York University, Giulio Antoniol Ecole Polytechnique de Montreal, Massimiliano Di Penta University of Sannio, Italy | ||
11:38 4mTalk | Toward Neurosymbolic Program Comprehension Early Research Achievements (ERA) Alejandro Velasco William & Mary, Aya Garryyeva William and Mary, David Nader Palacio William & Mary, Antonio Mastropaolo William and Mary, USA, Denys Poshyvanyk William & Mary Pre-print | ||
11:42 10mTalk | Combining Static Analysis Techniques for Program Comprehension Using Slicito Tool Demonstration Pre-print | ||
11:52 6mTalk | Mining Code Change Patterns in Ada Projects Replications and Negative Results (RENE) | ||
11:58 6mTalk | Telling Software Evolution Stories With Sonification Early Research Achievements (ERA) | ||
12:04 10mTalk | Attributed Multiplex Learning for Analogical Third-Party Library Recommendation and Retrieval Research Track Baihui Sang State Key Laboratory for Novel Software Technology, Nanjing University, Liang Wang Nanjing University, Jierui Zhang Nanjing University, Xianping Tao Nanjing University | ||
12:14 6mTalk | LLM2FedLLM - A Tool for Simulating Federated LLMs for Software Engineering Tasks Tool Demonstration Jahnavi Kumar Indian Institute of Technology Tirupati, India, Siddhartha Gandu Indian Institute of Technology Tirupati, Sridhar Chimalakonda Indian Institute of Technology, Tirupati | ||
12:20 10mLive Q&A | Session's Discussion: "Empirical Findings, Future Visions, Recommendations" Research Track | ||