Medusa: A Framework for Collaborative Development of Foundation Models with Automated Parameter Ownership Assignment
Foundation models (FMs) become the backbone of intelligent systems. Collaborative development of FMs enables multiple teams to fine-tune different aspects of an FM simultaneously. However, conflicts in model updates across teams, particularly when modifying overlapping parameters, pose significant challenges to maintaining model performance. In this paper, we propose \toolname{}, a novel framework designed to support collaborative FM development by managing model branches and introducing a structured system of parameter ownership. Medusa tracks fine-tuning efforts as separate branches, similar to Git, allowing developers to work on different tasks without destabilizing the base model. Instead of passively merging parameters from already fine-tuned models, \toolname{} proactively controls the merging through our parameter ownership assignment algorithm to generate merging-aware masks to guide the fine-tuning process, ensuring that only specific branches can modify designated parameters. \toolname{} approximates the optimal assignment even as model complexity increases, ensuring scalability in large, fine-tuned models.
We conduct extensive evaluations on five datasets and three large models with state-of-the-art post-training model merging approaches to investigate the efficacy of \toolname{}. Evaluation results show that \toolname{} substantially and generally improves the effectiveness of collaborative model development, across different models, fine-tuning methods, and datasets. Specifically, with automated parameter ownership assignment and masked fine-tuning, \toolname{} outperforms state-of-the-art post-training model merging approaches by improving 3.19% absolute model performance after merging. Ablation studies further demonstrate the efficacy of algorithms in \toolname{}.
Wed 25 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:30 | Program Analysis 4Demonstrations / Journal First / Research Papers at Pirsenteret 150 Chair(s): Matthew B Dwyer University of Virginia | ||
14:00 10mTalk | HeaderSplit: An Automated Tool for Splitting Header Files in C Projects Demonstrations Yue Wang Peking University, Jiaxuan Sun Peking University, Tongwei Deng Peking University, Wenhui Chang Peking University, Yanzhen Zou Peking University, Bing Xie Peking University | ||
14:10 20mTalk | Recasting Type Hints from WebAssembly Contracts Research Papers Kunsong Zhao The Hong Kong Polytechnic University, Zihao Li Hong Kong Polytechnic University, Weimin Chen The Hong Kong Polytechnic University, Xiapu Luo Hong Kong Polytechnic University, Ting Chen University of Electronic Science and Technology of China, Guozhu Meng Institute of Information Engineering, Chinese Academy of Sciences, Yajin Zhou Zhejiang University; ZJU-Hangzhou Global Scientific and Technological Innovation Center DOI | ||
14:30 20mTalk | Medusa: A Framework for Collaborative Development of Foundation Models with Automated Parameter Ownership Assignment Research Papers Dezhi Ran Peking University, Yuan Cao Peking University, Yuzhe Guo Beijing Jiaotong University, Yuetong Li The University of Chicago, Mengzhou Wu Peking University, Simin Chen University of Texas at Dallas, Wei Yang UT Dallas, Tao Xie Peking University DOI | ||
14:50 10mTalk | jAST: Analyzing and Modifying Java ASTs with Python Demonstrations Marius Smytzek CISPA Helmholtz Center for Information Security, Martin Eberlein Humboldt-Universtität zu Berlin, Lars Grunske Humboldt-Universität zu Berlin, Andreas Zeller CISPA Helmholtz Center for Information Security | ||
15:00 20mTalk | Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis Journal First Fengjie Li Tianjin University, Jiajun Jiang Tianjin University, Jiajun Sun Tianjin University, Hongyu Zhang Chongqing University | ||
15:20 10mTalk | JSSpecVis: A JavaScript Language Specification Visualization Tool Demonstrations Minseok Choe Korea University, Kyungho Song Sogang University, Hyunjoon Kim Korea University, Jihyeok Park Korea University |
This room is located outside Clarion Hotel
This room is located in the Pirsenteret (The Pier Center) convention center. It is just outside the hotel, on the back, towards the fjord.
You should be able to go through the emergency exit at Clarion, just on the side of the Cosmos 3 wing, which will be bring you close to Pirsenteret.
The entrance to the center is from here:
https://maps.app.goo.gl/dU3qH6kAimXGBNHe7
Once inside, go all straight and you will find signage to reach the room. The room is known as room 150 inside the center.