Project – continuous validation of the data

Now that we have learned how to define the SLOs of our projects and how to transform these SLOs into rules, it is time to learn how to integrate these rules into a CI/CD process and how to implement an end-to-end data validation pipeline.

Concepts of CI/CD

For several years now, software development has adopted a set of best practices called CI/CD, which is aimed at eliminating the distance that exists between development and operations activities. This objective is mainly realized by forcing teams to automate the building, testing, and deployment phases of applications.

The acronym CI/CD stands for continuous integration (CI) and continuous delivery (CD), or in some cases, continuous deployment (CD). Before introducing the concept of continuous data validation, it is important to understand these concepts in detail.

In Figure 5.2, we can see a graphical representation of the main stages of these processes:

...