NLBSE 2027

Natural language artifacts, from requirements and design documents to issue reports, code comments, and user feedback, are created and (re)used throughout software development and evolution. Natural language processing (NLP) tools and techniques are commonly used to optimize many aspects of the software development life cycle relying on such artifacts. More recently, coding assistants and agentic workflows based on large language and large code models have introduced new opportunities and challenges, with natural language specifications increasingly serving as executable artifacts. Yet open questions remain about the suitability of specialized versus generalized NLP solutions, their computational efficiency and deployability on consumer hardware, and the scientific rigor and reproducibility of reported results.

The main objective of the Natural Language-Based Software Engineering Workshop (NLBSE) is to bring together researchers and industrial practitioners from the NLP and software engineering communities to share experiences across this diversity of methods, artifacts, and application contexts. To keep discussions grounded in practice, the workshop also hosts tool competitions in which participants develop and evaluate NLP-based solutions on shared tasks and datasets.

Workshop previous editions: NLBSE series

Tool competitions

Multitask Code Classification

Competition Overview

Automatically identifying low-quality and insecure code is an important software-maintenance challenge. Relevant evidence is often distributed across two complementary artefacts: source code and the natural-language comments written by developers.

The NLBSE’27 Code Classification Competition invites participants to build a model that uses both artefacts to address two related tasks jointly:

• vulnerability classification for source code; and
• Self-Admitted Technical Debt (SATD) classification for developer comments.

Models are assessed not only by their predictive performance, but also by their execution time and computational cost in a shared evaluation environment.

Participants must develop one multitask model that receives a composite code-and-comment artefact and jointly predicts two binary labels. Each input contains a normalised source-code function together with its associated developer comments.

The expected output is a two-element binary vector in the following order:

Output	Meaning
[0, 0]	No vulnerability and no SATD
[1, 0]	Vulnerability and no SATD
[0, 1]	No vulnerability and SATD
[1, 1]	Vulnerability and SATD

The primary predictive objective is to maximise the arithmetic mean of the vulnerability and SATD F1-scores. Participants are encouraged to investigate representations and integration strategies that exploit the complementary information in code and comments.

Dataset

The competition uses a multi-annotated dataset based on MADE-WIC (https://doi.org/10.1145/3691620.3695348), a collection of functions and comments mined from open-source projects. Each instance contains source code and its associated comments, together with the following annotations:

1. a vulnerability label associated with the source-code component; and
2. an SATD label associated with the comment component.

The competition dataset is divided into three partitions:

Partition	Availability	Intended use
Training set	Public	Train the submitted model
Validation set	Public	Measure the performance of the trained model
Test set	Hidden	Final evaluation by the competition organisers only

The hidden test set will be used to determine the final winner. It is not disclosed to participants and is withheld to measure how well submissions generalise beyond the public training and validation data.

Requirements

You must train, tune, and evaluate your model on the provided data. We look forward to solutions that outperform our baseline model.

Detailed instructions about the competition (data, rules, baseline, results, etc.) can be found in our GitHub Repository (https://github.com/nlbse2027/code-comment-classification) and a Google Colab Notebook (https://colab.research.google.com/drive/1b_EiXx5woyCrDntwbljfi-BwS9Dwvhoa?usp=sharing).

Competition Organizers

The competition is organized by: Moritz Mock (momock@unibz.it), Thomas Borsani (tborsani@unibz.it), and Barbara Russo (brusso@unibz.it)

Participation Requirements

To participate in the competition, you must train, tune, and evaluate your model using the provided training and validation sets.

Additionally, you must write a paper (2-4 pages) describing:

• The architecture and details of the classification model
• The procedure used to pre-process the data
• The procedure used to tune the classifier on the training set
• The results of your classifier on the validation set
• A link to the code/tool with proper documentation on how to run it and replicate the results

Submission instructions and the submission site will be announced soon. All submissions must conform to the NLBSE 2027 submission requirements and the ICSE 2027 formatting and submission instructions.

Submission Acceptance

Submissions will be evaluated and accepted based on correctness and reproducibility, defined by the following criteria:

• Clarity and detail of the paper content
• Availability of the code/tool, including the training/tuning/evaluation pipeline, released as open-source
• Correct training/tuning/evaluation of your code/tool on the provided data
• Correct report of the metrics and results
• Clarity of the code documentation

We will use a formula to rank the competition submissions and determine a winner. Details will be provided in the Google Colab notebook.

The accepted submissions will be published in the workshop proceedings.

Skill Classification Competition

Competition Overview

The 2027 competition consists of building and assessing multi-label classifiers that predict, for each issue, the set of domains and sub-domains representing the skills required to solve it.

Dataset

We release a dataset mined from 7,245 merged pull requests across 11 popular Java repositories (57,206 source files; 59,644 methods; 13,097 classes) annotated with 217 skill labels composed by domain/sub-domains. You may find the dataset in a SQLite database, named skillscope_data.db, in our GitHub repository (https://github.com/nlbse2027/skill-classification). This dataset is an updated version of the one submitted in the original SkillScope paper.

• Ready-made Table: Inside the database you will find a table named nlbse_tool_competition_data_by_issue that joins each pull request’s textual and code-context features with its canonical domain/sub-domain labels per issue. There is also a view vw_nlbse_tool_competition_data_by_file that labels each filename/function associated with each issue.
• The nlbse_tool_competition_data_by_issue table contains a column for each domain and subdomain with an integer count of the number of matching APIs found for that issue. A value greater than zero indicates that domain/subdomain is present in the issue.
• The vw_nlbse_tool_competition_data_by_file is present for convenience purposes only and will not be used in evaluation of the model.

Additional Data Usage

Your model is allowed to use input data that is outside what is given in the database. You may use additional GitHub APIs to fetch more metadata, or download relevant files in an issue for further analysis. However, you must not use any third-party classification engine or the outputs present in skillscope_data.db as direct inputs to the model.

Baselines

Your models will be compared against the SkillScope Random-Forest + TF-IDF baselines reported in the paper by evaluating the overall prediction metrics against the issue classifications as recorded in the nlbse_tool_competition_data_by_issue table. The models you create should return a multi-label classification encoded in a one-hot encoded vector. However, it is the metrics of your models which will be the subject to the evaluation instead of the specific model output. Random Forest Baseline is provided in the folder competition_baseline with README and requirements.txt.

Goal

Train, tune and evaluate your models on the provided splits and improve at least one of precision, recall, or micro-F1 while not decreasing the remaining metrics relative to the best baseline.

Competition Organizers

The skill classification competition is organized by: Fabio Santos (fabio.deabreusantos@colostate.edu), and Jacob Penney (jmp458@nau.edu)

Submission Acceptance & Competition

Submissions will first be filtered to ensure they do not lower any of the three core metrics (precision, recall, micro-F1) compared with the baseline. Among qualifying submissions, ranking is determined by the largest positive improvement in micro-F1. Ties will be broken by (1) precision, then (2) runtime.

Submissions will be judged on:

• Clarity and completeness of the paper.
• Availability and reproducibility of code/tool (open-source required).
• Correct reporting of metrics.
• Quality of documentation.

Accepted papers will appear in the NLBSE ’27 proceedings.

Ranking Details

Participants submit a single multi-label classifier. Rankings proceed in two stages:

1. Eligibility check: The submission must not reduce any of precision, recall, or micro-F1 compared with the baseline.
2. Ordering: Qualifying submissions are ordered by the greatest positive delta-micro-F1.

Citing Relevant Work

General Competition Citation

Please cite the following work if participating in either competition:

Mock, M., Borsani, T., Russo, B., Santos, F., Penney, J., Valenzuela-Toledo, P., Kehrer, T., and Panichella, S. (2027). The NLBSE’27 Tool Competition and Challenge. In Proceedings of the 6th International Workshop on Natural Language-based Software Engineering (NLBSE’27).

Multitask Code Classification Citations

Please also cite the work underlying the dataset and baseline:

Mock, M., Melegati, J., Kretschmann, M., Diaz Ferreyra, N. E., and Russo, B. (2024). MADE-WIC: Multiple Annotated Datasets for Exploring Weaknesses in Code. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24), pp. 2346–2349. Association for Computing Machinery. https://doi.org/10.1145/3691620.3695348

Russo, B., Melegati, J., and Mock, M. (2025). Leveraging Multi-Task Learning to Improve the Detection of SATD and Vulnerability. In 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC), pp. 1–12. https://doi.org/10.1109/ICPC66645.2025.00017

Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., and Zhou, M. (2020). CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536–1547. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.139

Skill Classification Citations

Please cite the following work if participating in the Skill Classification Competition:

Carter, B. C., Contreras, J. R., Llanes Villegas, C. A., Acharya, P., Utzerath, J., Farner, A. O., Jenkins, H., Johnson, D., Penney, J., Steinmacher, I., Gerosa, M. A., and Santos, F. (2025). SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub. In 2025 IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE), pp. 9–12. https://doi.org/10.1109/NLBSE66842.2025.00007

Santos, F., Vargovich, J., Trinkenreich, B., Santos, I., Penney, J., Britto, R., Pimentel, J. F., Wiese, I., Steinmacher, I., Sarma, A., et al. (2023). Tag that issue: Applying API-domain labels in issue tracking systems. Empirical Software Engineering, 28(5), Article 116. Springer.

Vargovich, J., Santos, F., Penney, J., Gerosa, M. A., and Steinmacher, I. (2023). Givemelabeledissues: An Open Source Issue Recommendation System. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pp. 402–406.

Call for papers

Researchers and practitioners are invited to submit:

• Full papers (maximum of 8 pages, including references). Original research in NLP for SE, either empirical, theoretical, or showing the practical experience of using NLP techniques and/or NLP tools for addressing software engineering-specific challenges
• Education tools and materials (maximum of 8 pages, including references). Original contributions covering all dimensions of learning and teaching NLP in software engineering. This also includes experience reports providing informal proof by outlining a particular experience connected to education and training, such as a course, an educational or training method. The submission should translate the experience into practical guidance and insights gained, without the requirement for thorough evaluation or the application of rigorous research techniques to back its claims.
• Replication Studies and Negative results (maximum of 8 pages, including references). Research papers and reviews focusing on negative results or the reproducibility of previously published work. We believe that publishing negative results, alongside positive ones, provides a more holistic view of the research landscape, fostering transparency, credibility, and the elimination of publication bias.
• Short and demonstration papers (maximum of 4 pages, including references). Work that describes novel techniques, tools, ideas, and positions that have yet to be fully developed; or are a discussion of the importance of a recently published NLP result by another author in setting a direction for the SE community, and/or the potential applicability (or not) of the result in an industrial context.
• Position papers (maximum of 2 pages, including references). Contributions that analyze trends in NLBSE and raise issues of importance. Position papers are intended to seed discussion and debate at the workshop, and thus will be reviewed with respect to relevance and their ability to spark discussions.
• Tool Competition entries (maximum of 4 pages, including references). We invite researchers, students, and tool developers to design innovative solutions to tackle the automated classification of code comments and automated skill classification. For submissions of this kind, please refer to the instructions detailed in the Tool competitions / Research challenge tabs.

In all cases, papers should address a problem in the software engineering domain or combine elements of NLP research with other concerns in the software engineering lifecycle. Examples of problems in the software engineering domain include (but are not limited to): key information identification and extraction from natural language software artifacts; elicitation, modeling, and verification of requirements; generation of source code documentation; software verification and validation support; classification, summarization, and prioritization of development tasks; changes, developers, and solutions recommendation; maintenance effort minimization; quality assessment of natural language software artifacts.

The solution should apply NLP-based approaches and/or models such as (but not limited to) textual analysis, text summarization, topics or aspects modeling and extraction, machine translation, natural language parsing, semantic parsing, natural language generation, sentiment analysis, discourse analysis.

All submissions must conform to the ICSE’27 formatting and submission instructions. All submissions must be anonymized, in PDF format and should be performed electronically through HotCRP: https://icse2027-nlbse.hotcrp.com/. For tool competition submissions, check the details reported on the Tool competitions / Research challenge tabs.

All dates are Anywhere on Earth (AoE).

Research challenge

The 2027 GitHub Agentic Workflows Challenge invites researchers to investigate how developers specify, configure, and evolve natural-language-based agentic workflows in real-world GitHub repositories.

GitHub Agentic Workflows (GH-AW) are an emerging form of repository automation in which developers define agentic workflows in Markdown and execute them through GitHub Actions. These workflows combine natural-language instructions with structured configuration, allowing AI agents to interact with repositories and support software engineering tasks. Learn more: https://github.github.com/gh-aw/

The challenge focuses on the empirical study of these artifacts and their evolution, with particular attention to the role of natural language in software engineering automation.

Dataset

We release GHAW-H, a dataset of GitHub Agentic Workflow Histories. GHAW-H covers 262 public GitHub repositories that used GH-AW between February 13 and June 6, 2026. The dataset connects agentic workflow specifications with their compiled representations and version histories.

GHAW-H includes 604 source histories, 2,820 source Markdown file snapshots, 2,820 source-version records, and 2,820 lock-file snapshots. The dataset contains natural-language instructions written in Markdown, structured frontmatter declarations, compiled GitHub Actions workflow definitions in YAML, and historical versions of these artifacts.

• Ready-made dataset: GHAW-H provides repository-level and file-level artifacts for studying GitHub Agentic Workflows and their histories.
• Source specifications: Markdown files contain natural-language instructions and structured frontmatter used to define agentic workflows.
• Compiled workflows: Lock-file snapshots provide the compiled GitHub Actions workflow definitions associated with the source specifications.
• Histories: Source-version records allow researchers to study how workflow specifications and compiled representations evolve over time.

The dataset and documentation can be found on the GHAW-H website (https://pavt.github.io/GHAW-H/), GitHub repository (https://github.com/pavt/GHAW-H), Hugging Face (https://huggingface.co/datasets/pavtch/GHAW-H), and Zenodo (https://zenodo.org/records/22084012).

Additional Data Usage

Participants are allowed to use additional public data beyond what is included in GHAW-H. For example, participants may use GitHub APIs to collect repository metadata, workflow execution information, issue or pull request data, source code context, or other public project information relevant to their study.

Any additional data sources must be clearly documented, including the collection or extraction procedure, the date of collection, and how the additional data was linked to GHAW-H. Participants should also clearly identify the version of GHAW-H used in their study.

Baselines

This challenge is an open research challenge and does not prescribe a single baseline model or ranking metric. Participants may define baselines appropriate to their research question, such as keyword-based analyses, repository mining heuristics, NLP pipelines, machine-learning models, LLM-based methods, or manual coding protocols.

Submissions should justify the selected baseline or comparison point and explain how it supports the research question being investigated.

Goal

Use GHAW-H to investigate a research question related to natural language and agentic workflows in software engineering. The goal is to produce reproducible research that improves our understanding of how developers specify, configure, analyze, or evolve agentic workflows in practice.

Example research directions include, but are not limited to:

• How do developers express software engineering tasks and goals in natural language when defining agentic workflows?
• How are GH-AW capabilities, permissions, triggers, tools, and execution options configured in practice?
• What characteristics distinguish different types of agentic workflow specifications?
• How are Markdown specifications reflected in their compiled GitHub Actions workflow definitions?
• How do natural-language instructions, frontmatter configurations, and compiled workflows change over time?
• How do changes to natural-language instructions relate to changes in structured configuration and compiled workflow definitions?
• How do GH-AW specifications differ from other forms of agent or software automation configuration?
• How can NLP, machine learning, LLM-based, or other analysis techniques support the study, understanding, or development of agentic workflows?

Participation Requirements

To participate in the challenge, authors must write a paper describing a study, method, tool, or empirical analysis based on GHAW-H.

The paper should describe:

• The software engineering problem or research question being investigated.
• The role of natural language in the investigated problem.
• The GHAW-H artifacts used in the study.
• The version of GHAW-H used.
• Any additional data sources and how they were collected.
• The study design, methods, tools, and analysis protocol.
• The main findings or resulting contribution.
• The implications of the results for Natural Language-based Software Engineering.
• A link to code, data, notebooks, tools, or other artifacts with documentation on how to reproduce the results, whenever possible.

Submission instructions and the submission site will be announced soon. All submissions must conform to the NLBSE 2027 submission requirements and the ICSE 2027 formatting and submission instructions.

Submission Acceptance & Challenge

Submissions will be evaluated and accepted based on relevance, correctness, clarity, and reproducibility.

Submissions will be judged on:

• Clarity and completeness of the paper.
• Relevance of the research question to Natural Language-based Software Engineering.
• Appropriate and well-documented use of GHAW-H.
• Soundness of the study design, method, or tool.
• Correct documentation of additional data sources, when used.
• Clear reporting of findings, limitations, and implications.
• Availability and reproducibility of code, data, notebooks, or tools whenever possible.
• Quality of artifact documentation.

Accepted papers will appear in the NLBSE ’27 proceedings.

Ranking Details

The GitHub Agentic Workflows Challenge is not ranked by a single quantitative metric. Submissions will be assessed according to the relevance of the research question, the soundness of the method, the appropriateness of the use of GHAW-H, and the reproducibility of the reported results.

Paper / Reference Publication

The reference paper, GHAW-H: A Dataset of GitHub Agentic Workflow Histories (https://pavt.github.io/GHAW-H/assets/GHAW-H.pdf?v=420906b), describes the construction, structure, and intended research uses of GHAW-H. Participants should consult the paper when designing studies and cite it together with the archived dataset release; see BibTeX entries below.

Citing Relevant Work

Please cite the archived dataset release if participating in the GitHub Agentic Workflows Challenge:

Valenzuela-Toledo, P., Kehrer, T., and Panichella, S. (2026). GHAW-H: A Dataset of GitHub Agentic Workflow Histories (version v0.1.2). Zenodo. https://doi.org/10.5281/zenodo.22084012

Please also cite the reference paper when available:

Valenzuela-Toledo, P., Kehrer, T., and Panichella, S. (2027). GHAW-H: A Dataset of GitHub Agentic Workflow Histories. IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE@ICSE 2027), Dublin, Ireland. Reference paper (publication metadata to be updated).

Organizers

Pablo Valenzuela-Toledo (University of Bern and Universidad de La Frontera), Timo Kehrer (University of Bern), Sebastiano Panichella (University of Bern and AI4I).