Background: Nowadays, Continuous Integration (CI) has become a widely adopted software development practice that enables faster code change integration and better software maintenance. At the same time, Machine Learning (ML) is being used by software applications for real-world scenarios like autonomous driving, which they previously could not resolve. ML projects employ development processes different from those of traditional software projects, but they also require multiple iterations to integrate new functionality and improve their quality, and thus may benefit from CI practices.

Aims: While there are many works covering CI within traditional software, none of them have empirically explored the adoption of CI and its associated failures and errors in the context of ML projects’ development. To address this knowledge gap, we performed an empirical analysis to compare CI adoption between ML projects and Non-ML projects in GitHub.

Method: We developed TraVanalyzer, the first Travis CI configuration analyzer, to analyze the different CI adoption practices in ML projects, and also developed a CI log analyzer to identify different types of CI problems in ML projects.

Results: We found that Travis CI is the most popular CI tool for ML projects, and that their CI adoption in general lags behind that of Non-ML projects, but that ML projects which adopted CI, used it for building, testing, code analysis, and automatic deployment more than Non-ML projects. We also found that only 24.6% of Travis-using ML projects adopted automated deployment, and that the majority of them perform their testing in CI using traditional unit testing frameworks, even though ML testing differs from regular unit testing. Furthermore, while CI in ML projects is as likely to experience problems as CI in Non-ML projects, it has more varied reasons for build-breakage. Yet, the most frequent CI failures of ML projects are testing-related problems such as unit test failures due to exceptions and test misconfiguration, similar to CI failures of Non-ML and OSS projects.

Conclusion: To the best of our knowledge, this is the first work that has analyzed ML projects’ CI usage, practices, and issues, contextualized its results by comparing them with similar Non-ML projects, and which provided findings for researchers and ML developers to identify possible issues and improvement scopes for CI in ML projects.