Empirical Analysis on CI/CD Pipeline Evolution in Machine Learning Projects

Abstract

The growing popularity of Machine Learning (ML) and the integration of ML components with other software artifacts has led to the use of CI/CD tools, such as Travis CI, GitHub Actions, etc., that enable faster integration and testing for ML projects. Such CI/CD configurations and services require synchronization during the life cycle of the projects. Severalworks discussed how CI/CD configuration and services change during their usage in traditional software systems. However, there is minimal knowledge of how CI/CD configuration and services change in ML projects.To fill this knowledge gap, this work presents the first empirical analysis of how CI/CD configuration evolves for ML software systems. We manually analyzed 343 commits collected from 508 open-source ML projects to identify frequent CI/CD configuration changecategories in ML projects. We devised a taxonomy of 14 co-changes in CI/CD and MLcomponents. Moreover, we developed a CI/CD configuration change clustering tool that identified frequent CI/CD configuration change patterns in 15,634 commits. Furthermore, we measured the expertise of ML developers who modify CI/CD configurations. Based on this analysis, we found that 61.8% of commits include a change to the build policy and minimal changes related to performance and maintainability compared to general open-source projects. Additionally, the co-evolution analysis identified that CI/CD configurations, in many cases, changed unnecessarily due to bad practices, such as the direct inclusion of dependencies and a lack of usage of standardized testing frameworks. More practices were found through the change patterns analysis, which used deprecated settings and relied on a generic build language. Finally, our developer’s expertise analysis suggests that experienced developers are more inclined to modify CI/CD configurations.