Intelligent detection and diagnosis of anomalies in software delivery pipelines

Abstract

Modern organizations have widely adopted DevOps automation practices, enabling them to deliver software versions quickly and frequently while ensuring the quality of the code produced by developers. These practices are mainly enabled by continuous integration and continuous deployment (CI/CD) pipelines, whose execution results are used to certify software quality and confirm successful deployments in case of “success”, and to report code issues to developers in case of “failure”. As such, the value of CI/CD pipelines lies essentially in reliable and deterministic execution results. In practice, however, anomalies related to complex and distributed environments lead to unexpected and misleading pipeline results (i.e., false alarms known as “intermittent failures” or, worse, deceptive successes referred to as “silent failures”). These anomalies cause significant waste due to sporadic reruns and associated delays. Conducted in collaboration with TELUS, this research argues that pipeline anomalies can be mitigated through intelligent automation based on pipeline execution data, including execution logs and metrics. The first study proposes an efficient detection approach for intermittent failures based on fine-tuning pre-trained text embedding models. This approach achieves high performance (70-88% F1) using only twelve pairs of manually labeled logs and outperforms state-of-the-art approaches that rely on large, often inaccurately labeled datasets. The second study presents a first-of-its-kind diagnostic catalog of 46 categories of intermittent failures and demonstrates that effective prioritization of these categories requires looking beyond their raw frequency. Hence, we introduce FlakeRanker, a tool that determines the categories to be prioritized (14 at TELUS) using a clustering algorithm that integrates their frequency, recency and organizational cost. The third study introduces FlaXifyer, a classifier trained with only twelve labeled logs per category to automatically predict which of the 14 priority categories an intermittent job failure belongs to. This study also presents LogSift, an interpretability technique that highlights the most influential log segments, accelerating triage and diagnosis. The fourth study examines the phenomenon of silent failures and outlines the key factors associated with reruns of pipelines that incorrectly report successes, thereby facilitating their detection. It also presents a taxonomy of 11 themes observed in industrial contexts and summarizes the available workarounds and corrective solutions. Finally, the fifth study introduces the CBDT Framework, a digital twin-inspired architecture for orchestrating detection, diagnosis, and repair services in production CI/CD systems. Together, these efforts pave the way for intelligent automation capable of orchestrating the autonomous detection, diagnosis, and repair of pipeline anomalies. This approach not only improves pipeline reliability and the quality of delivered software, but also reduces developers’ cognitive load and minimizes costs associated with delivery delays and software defects.

Date	20 Apr 2026
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Francis Bordeleau (Supervisor)