Data and BenchmarkingFORGE 2026

Accepted Papers

	Title
	COMPASS: A Psychometrics-Guided Multi-Dimensional Benchmark for Code Generation Evaluation Data and Benchmarking James Meaden, Markus Borg Pre-print
	MiG.4: A Curated Dataset of Library Migrations in Java and Python Data and Benchmarking Matheus Barbosa, Pedro Baptista, João Eduardo Montandon, MATHEUS LIMA, Pedro Henrique Fernandes Baptista
	MIRROR: A Dataset of Structural Metrics for Repackaged Android Apps Data and Benchmarking Sebastian Siedler, Karim Elish
	OmniBench-RAG: A Multi-Domain Evaluation Platform for Retrieval-Augmented Generation Tools Data and Benchmarking Jiaxuan Liang, shide zhou, Kailong Wang, JIAXUAN LIANG
	PromiseAwait: A Dataset of JavaScript Migrations from Promises to Async/Await Data and Benchmarking Rafael Araujo Magesty, João Eduardo Montandon, Rafael Magesty
	SEMODS: A Validated Dataset of Open-Source Software Engineering Models Data and Benchmarking Alexandra González, Xavier Franch, Silverio Martínez-Fernández DOI Pre-print
	Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering Data and Benchmarking Daniel Rodriguez-Cardenas, Xiaochang Li, Marcos Macedo, Antonio Mastropaolo, Dipin Khati, Yuan Tian, Huajie Shao, Denys Poshyvanyk
	Tricky²: Towards a Benchmark for Evaluating Human and LLM Error Interactions Data and Benchmarking Cole Granger, Dipin Khati, Daniel Rodriguez-Cardenas, Denys Poshyvanyk
	VHDL-Instruct: Training Open Dataset for LLMs Benchmarking and HDL Code Generation Data and Benchmarking Patrik Drazic, Benaoumeur Senouci, Boualem Benatallah Media Attached

Call for Papers

High-quality datasets and robust evaluation frameworks are essential for the advancement of foundation models (FM). The Benchmarking Track at FORGE 2026 provides a dedicated forum for publishing rigorous research on machine learning datasets and benchmarking methodologies that go beyond conventional evaluation metrics. In particular, this track encourages work that advances the state of the art in data quality, reproducibility, and benchmarking standards, with a focus on enabling the development and assessment of FM for software engineering (SE).

Scope

This track welcomes two types of submissions in the context of software engineering: (1) Data papers and (2) Benchmarking papers.

1. Data Papers

   Contributions may include:

   • New datasets, or thoughtfully designed (collections of) datasets built from previously available data.

   • Data generators and reinforcement learning environments.

   • Data-centric AI methods and tools (e.g., methods to measure/improve data quality, or studies that bring new insights).

   • Advanced practices in data collection and curation, even if the data itself cannot be shared.

   • Frameworks for responsible dataset development, dataset audits, or identifying significant issues in existing datasets.

2. Benchmarking Papers

   Contributions may include:

   • New benchmarks, metrics, or benchmarking tools.

   • Systematic analyses of existing systems on novel datasets that yield new insights.

Evaluation Criteria

1. Data Papers

   • Value, usefulness, and reusability of the dataset or tools.

   • Quality and clarity of presentation.

   • Clear positioning within related work and relevance to SE.

   • Accessibility of datasets/tools (e.g., data and required code should be findable, usable, and open source where possible).

2. Benchmarking Papers

   • Relevance of the proposed benchmark to the FORGE audience.

   • Originality of the ideas.

   • Clarity and quality of presentation.

   • Usefulness and outreach of the results, tools, or datasets.

Submission Instructions

• Length: Maximum of 4 pages, plus 1 page of references.

• Artifacts: Authors are strongly encouraged to share (anonymized/curated) data and artifacts to foster reproducibility, though this is not mandatory. If artifacts are not shared, justification is expected.

• Format: All submissions must be in PDF format and conform, at time of submission, to the official “ACM Primary Article Template”, which can be obtained from the ACM Proceedings Template page. LaTeX users should use the sigconf option, as well as the review (to produce line numbers for easy reference by the reviewers) and anonymous (omitting author names) options. To that end, the following LaTeX code can be placed at the start of the LaTeX document: \documentclass[sigconf,review,anonymous]{acmart}

• Reviewing: Double-anonymous. Remove author names and affiliations; cite your prior work in the third person. For guidance, see the ICSE Research Track Q&A on double-anonymous reviewing.

• Language: All papers must be in English. Use the HotCRP format checker before submission. Minor warnings (e.g., small fonts in figures/footnotes) will not cause rejection if the main text complies.

• Accessibility: Authors should ensure papers are accessible to people with disabilities. See SIGACCESS guidelines.

Submission site: https://forge26-benchmarking.hotcrp.com/

Important dates (AOE)

• Full paper submission deadline: Friday, Nov 21, 2025

• Author notification: Friday, Dec 26, 2025

• Camera-ready deadline: Friday, Jan 23, 2026