Recovering Trace Links Between Software Documentation And Code
Replication Package For Recovering Trace Links Between Software Documentation And Code
by Jan Keim, Sophie Corallo, Dominik Fuchß, Tobias Telge, and Anne Koziolek
Abstract
Introduction Software development involves creating various artifacts at different levels of abstraction and establishing relationships between them is essential. Traceability link recovery (TLR) automates this process, enhancing software quality by aiding tasks like maintenance and evolution. However, automating TLR is challenging due to semantic gaps resulting from different levels of abstraction. While automated TLR approaches exist for requirements and code, architecture documentation lacks tailored solutions, hindering the preservation of architecture knowledge and design decisions.
Methods In our paper, we present our approach TransArC for TLR between architecture documentation and code, using component-based architecture models as intermediate artifacts to bridge the semantic gap. We create transitive trace links by combining the existing approach ArDoCo for linking architecture documentation to models with our novel approach ArCoTL for linking architecture models to code.
Results We evaluate our approaches with five open-source projects, comparing our results to baseline approaches. The model-to-code TLR approach achieves an average F1-score of 0.98, while the documentation-to-code TLR approach achieves a promising average F1-score of 0.82, significantly outperforming baselines.
Replication This replication package includes all used datasets, benchmarks, baselines, and approaches. The package allows to reproduce the evaluation results, as well as to apply TransArC to other projects.
Content
This replication package belongs to Recovering Trace Links Between Software Documentation And Code by Keim et al. (Preprint at DOI 10.5445/IR/1000165692).
This replication package allows to 1. get additional details to the publication (pseudocode for the computation of trace links between SAM and Code), 2. reproduce the complete evaluation results, and 3. reuse the approach TransArC (ArDoCo+ArCoTL) on other projects.
Therefore, we apply for both badges, availability and reusability:
Availability: The replication package is placed on Zenodo with a DOI (10.5281/zenodo.10411853) and is publicly available on GitHub.
Reusability: The replication package includes a Docker image to reproduce the complete evaluation results, including all baselines. It provides an extensive documentation and data to rebuild the Docker image or execute it locally. The replication package also allows the reuse of TransArC on other projects, the required documentation for the CLI is also provided.
The replication package is structured as follows:
- ardoco+arcotl: contains the source code of ArCoTL and ArDoCo
- baselines: contains the baselines used in the paper
- data: contains the data used in the paper, including the textual software architecture documentation, the architecture models and the gold standards.
- evaluator: contains helper scripts to evaluate the generated results
- results: contains the results of the experiments
Please be aware that, due to the inclusion of all baseline approaches, the replication package is extensive and requires ~50GB storage for the full version. Due to space limitations, this abstract only covers some parts; all other information can be found in the README of the replication package.
Reproduction of Complete Evaluation Results
Please be aware that the reproduction of the complete evaluation results can take some time. On our system (Tesla V100S, 32GB VRAM, 256GB RAM, 112 CPUs a 2.7Ghz), the execution takes approximately 4h for CodeBERT, 20 min for FTLR, 10 min for TAROT, and 7 min for TransArC. Everything was tested on Linux.
Hardware Requirements: We recommend the execution on a system with at least 16 GB RAM. To execute the baseline CodeBERT, the system also needs a GPU with CUDA 12.2 and ~16GB vRAM.
Setup: The evaluation results can be created in three ways: 1. Use the provided Docker image, 2. Build the Docker image yourself, or 3. Generate the results locally by yourself. To use the provided Docker image, run: docker run -it --rm --gpus all ghcr.io/ardoco/icse24
Reusing TransArC
To reuse TransArC, you can use the provided CLI. For example, to execute SAD-Code, use java -jar ardoco-cli.jar -t SAD-Code -n NAME -d DOCUMENTATION -m MODEL -c CODE -o OUT. Here, NAME is the name of the project, DOCUMENTATION and MODEL are the paths to the documentation file and architecture model respectively, and OUT is the path where the output should be written to. The output files are CSV-files that contain the found trace links for each task. To evaluate the CSV-files against a goldstandard, you can use the provided evaluator in the evaluator folder, for example like this: java -jar evaluator.jar -t RESULT-CSV-FILE -g GOLDSTANDARD-CSV-FILE
