ClauseBench: Enhancing Software License Analysis with Clause-Level Benchmarking
Open-source software (OSS) has revolutionized modern software development by fostering collaboration across diverse teams. However, as OSS projects grow in size and complexity, managing license compliance has become increasingly challenging. A critical issue lies in accurately recognizing and interpreting the varied clauses within OSS licenses, particularly when multiple licenses coexist, each with distinct permissions, obligations, and restrictions. Traditional license analysis tools, often rule-based, struggle to identify nuanced conflicts between license clauses, leading to potential compliance risks. In response to these challenges, this paper presents a fine-grained, high-quality dataset of 634 SPDX-certified licenses, annotated with 3,396 individual clauses across 14 categories. Each clause has been meticulously reviewed and validated using model-assisted checks to ensure accuracy, providing a solid foundation for detailed clause-level analysis. To improve clause recognition and conflict detection, we introduce ClauseBench, a benchmarking framework that leverages large language models (LLMs) to detect and interpret license clauses with high precision. ClauseBench improves detection accuracy by 50 % compared to traditional document-level methods and significantly reduces hallucination rates by focusing on individual clauses, where precise distinctions in legal language are crucial. Additionally, we implemented a contextual prompt engineering strategy to optimize model performance, achieving 90% accuracy in clause identification. Our framework sets a new standard for automated license conflict detection in OSS, demonstrating the potential of LLMs to manage the complexities of legal text interpretation. This work not only advances the field of license analysis but also opens the door to future research on integrating LLMs with OSS compliance tools.
Thu 1 MayDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | Industry Challenge PresentationsIndustry Challenge Track at 211 Chair(s): Federica Sarro University College London, Xin Xia Huawei | ||
14:00 15mTalk | CKGFuzzer: LLM-Based Fuzz Driver Generation Enhanced By Code Knowledge GraphAward Winner Industry Challenge Track Hanxiang Xu Huazhong University of Science and Technology, Wei Ma , Ting Zhou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Kai Chen Huazhong University of Science and Technology, Qiang Hu The University of Tokyo, Yang Liu Nanyang Technological University, Haoyu Wang Huazhong University of Science and Technology | ||
14:15 15mTalk | ClauseBench: Enhancing Software License Analysis with Clause-Level Benchmarking Industry Challenge Track Qiang Ke Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology | ||
14:30 15mTalk | CodeMorph: Mitigating Data Leakage in Large Language Model Assessment Industry Challenge Track Hongzhou Rao Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Wenjie Zhu Huazhong University of Science and Technology, Ling Xiao Huazhong University of Science and Technology, Meizhen Wang Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology | ||
14:45 15mTalk | CommitShield: Tracking Vulnerability Introduction and Fix in Version Control SystemsSecurity Industry Challenge Track Zhaonan Wu Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Chen Wei MYbank, Ant Group, Zirui Wan Huazhong University of Science and Technology, Yue Liu Monash University, Haoyu Wang Huazhong University of Science and Technology | ||
15:00 15mTalk | Exploring Large Language Models for Analyzing Open Source License Conflicts: How Far Are We? Industry Challenge Track Xing Cui Institute of Software, Chinese Academy of Sciences, Jingzheng Wu Institute of Software, The Chinese Academy of Sciences, Xiang Ling Institute of Software, Chinese Academy of Sciences, Tianyue Luo Institute of Software, Chinese Academy of Sciences, Mutian Yang Beijing ZhongKeWeiLan Technology Co.,Ltd., Wenxiang Ou Institute of Software, Chinese Academy of Sciences | ||
15:15 15mTalk | OSS-LCAF: Open-Source Software License Conflict Analysis Framework Industry Challenge Track |