ICSE 2027 Competitions

Call for Competition Proposals

ICSE 2027 invites proposals for the first-ever Competition Track: a home for hands-on, benchmark-driven competitions that bring researchers, practitioners, and open-source communities together around shared tasks, datasets, and leaderboards.
We’re looking for competitions that are engaging for participants, rewarding for organisers and the community, rigorously judged, and built around problems that are genuinely important for software engineering.

What is a competition?

For this track, a competition is a time-bounded challenge with:

• a clear task and public rules
• a well-defined evaluation corpus
• an explicit metric that determines ranking
• a reproducible evaluation setup, ideally automated
• an online submission workflow and a public leaderboard
• [optional] an opportunity for top teams to submit solution papers
• a plan to run the competition in 2026 and present outcomes at ICSE 2027

For reference on similar formats, think KDD Cup, SemEval, or NeurIPS Competitions, the hands-on benchmark-driven competition tracks that have long driven progress in the neighbouring communities of NLP and machine learning.

What a competition is not:

• a broad challenge problem evaluated by a committee without a leaderboard
• a paper track in disguise (where ranking is driven by reviews rather than measured performance)
• an on-site-only hackathon without a benchmark and scoring protocol

Scope

We welcome proposals around problems across the field of software engineering (SE), including (but not limited to):

• testing, verification, program analysis, repair, debugging
• mining software repositories, empirical SE, developer productivity
• code search / understanding / summarisation, documentation
• AI-assisted development and evaluation of developer tools
• security and reliability, performance, and maintainability
• new datasets and benchmarks for SE tasks (including IDE-centric or multimodal settings)

Format at ICSE 2027

Accepted competitions will typically have:

• an online phase run by the organisers before ICSE
• an in-person session at ICSE 2027 to present results, discuss solutions, and share lessons learned

From the conference side, this is workshop-like in logistics; from the organisers’ side, it’s all about the problem, the benchmark, and the participant experience.

What makes a strong proposal

We would love to see proposals that have:

• a problem with clear relevance to software engineering
• clean, fair evaluation: metric, baselines, test integrity, anti-leakage measures
• a great participant journey: starter kit, baseline, docs, support channel
• realistic feasibility: you can actually run it end-to-end
• responsible data handling: licensing, privacy, ethics

Platform and evaluation

We encourage using established infrastructure (e.g., Kaggle, EvalAI, AICrowd). Self-hosting is fine as well if the setup is reliable and accessible.

Your proposal should clearly state:

• what participants submit: predictions, containers, etc.
• what gets scored and how: metrics, tie-breaking
• how you ensure reproducibility and protect the private test set
• compute needs and engineering effort for evaluation, and how you keep them reasonable

Awards and winner support

Organisers are encouraged to offer prizes to the winners of the competitions.

If organisers expect to benefit materially from the outcomes of the competition (e.g., a company considers using competition results to inform development of commercial products), this should be disclosed in the proposal. In this case, we strongly encourage sponsoring at least one full ICSE registration per winning team as part of the prize package, or providing equivalent support, and articulating this as an incentive in the competition announcement.

Solution papers

We encourage the organisers to invite the top performing teams to submit papers describing their approach to the competition problem and the evolution of their solutions, to be published in the proceedings of the track. This can help promote participation from academia but requires extra effort from the organisers to (1) provide clear criteria for a team to be eligible to submit a paper, e.g. a non-trivial baseline to beat, and (2) review the content of the solution papers.

Solution papers should describe the authors’ approach to the problem, the key technical details of their solution, the evolution of the solution towards the final version, and include any relevant details on team composition, communication processes, and responsibilities.
Solution papers will be limited to 4 pages (including references) and conform to ICSE formatting guidelines.

The track chairs will provide a unified submission portal for solution papers from all accepted competitions that choose to offer submissions.
Competition organisers will review solution papers for technical correctness, along with the track chairs, who will ensure the correct structure of the papers and check the format of the camera-ready version. After the review period, the participants will be provided with an opportunity to address reviewers’ comments before definitive acceptance of the paper.

Solution papers will appear in the conference proceedings.

Organiser reports

We expect the organisers of accepted competitions to submit a report outlining the design and organisation of the competition to share their experience with the community. The report should be submitted upon completion of the competition.

Organiser reports will be reviewed by the track chairs and appear in the conference proceedings.

Submitting competition proposals

Submit your proposal via HotCRP https://icse2027-competitions.hotcrp.com/ by 26 June, 2026.

The proposals should be formatted similarly to submissions to the Research Track (\documentclass[10pt,conference]{IEEEtran}). We suggest a length of 4 to 6 pages with no strict limit.
Please strive for the competition to be launch-ready by early July: the contents of your proposal should be close to what you’d publish as the competition announcement on acceptance.
Proposals are not double-blind.

Content of proposals

• Title + short abstract
• Task description (why it matters, inputs/outputs)
• Organisers and roles (main contact, team composition, experience)
• Data/benchmark (origin, splits, licensing)
• Evaluation protocol (metrics, baselines, tie-breaking rules)
• Mechanics (platform, submission format, rules, integrity / anti-cheating measures)
• Timeline (high-level; exact dates can be TBD at the point of proposal submission)
• Participant experience and outreach plan
• Criteria for eligibility to submit solution papers (optional if solution papers are invited)
• Prizes and winner support
• ICSE session plan + post-competition report plan

Evaluation criteria

The proposals will be evaluated by the track committee based on the following criteria:

• Motivation, relevance, and novelty of the problem behind the competition.
• Adequacy of the competition setup (data, platform, metrics, team responsibilities) relative to the problem
• Participant experience: well-organised competitions should provide baseline solutions, a starter kit, and clear documentation to enable quick onboarding of participants.

Organiser commitments

Organisers of accepted competitions are expected to:

• publish a competition website (rules, timeline, FAQ)
• provide the dataset or benchmark + licence / usage policy
• provide evaluation protocol and a leaderboard / submission system
• provide a baseline solution and a starter kit to help participants get on board
• run participant support and maintain competition integrity
• host a session at ICSE 2027 focused on insights (not just awards)
• publish a short post-competition report summarising its setup and outcomes

Important dates

Fri 26 June 2026: Submission of competition proposals

Fri 10 July 2026: Notification of accepted proposals

Fri 4 December 2026: Submission of solution papers and organiser reports

Wed 16 December 2026: Reviewer response for solution papers and organiser reports

Wed 6 January 2027: Revisions due for solution papers and organiser reports

Wed 13 January 2027: Final notification for solution papers and organiser reports

Wed 20 January 2027: Camera-ready deadline for solution papers and organiser reports

Sun 25 April - Sat 1 May 2027: In-person sessions at ICSE 2027

The organisers are free to plan the competitions around the dates above. The key synchronisation points with the track timeline are the acceptance date and paper-related deadlines.

Contact

For questions, please email the track chairs at vladimir.kovalenko@jetbrains.com, mentioning “ICSE 2027” in the email subject. The chairs will keep the FAQ section below up to date.

FAQ

How does the Competition Track differ from Industry Challenge?

Industry Challenges typically rely on broader problem statements and committee judgment. In this track, we require a benchmark, a metric, an evaluation harness, and a leaderboard so participants can measure their progress and fairly compete with other solutions.