What Happened in This Pipeline? Diffing Build Logs With CiDiff
Continuous integration (CI) is widely used by developers to ensure the quality and reliability of their software projects. However, diagnosing a CI regression requires comparing and analyzing lengthy logs of passing and failing builds, which can be laborious. As off-the-shelf diff algorithms produce suboptimal results, in this work we introduce a new diff algorithm specifically tailored to build logs called CiDiff. We evaluate CiDiff against the baseline LCS-diff on a novel dataset of 17,906 CI regressions, using quantitative metrics and a user study. Our algorithm reduces the output size by about 20% and the number of new lines to inspect by about 60% in the median case, with reasonable overhead. Finally, our algorithm is favored by the majority of participants in 70% of the regression cases, whereas LCS-diff is preferred in only 5% of the cases.