Rationalizing the costs

At this point, most companies have been building data pipelines for decades, and what initially started as a simple process of transforming and uploading dashboards has now evolved into real data departments with tens, hundreds, and thousands of people working with data. We started by having and maintaining a few pipelines, but today, we have companies with thousands of pipelines that read and write from thousands of different data sources. Therefore, a critical aspect is governing this ecosystem of data pipelines and data stakeholders as well as governing the associated costs. This is especially true when we speak about cloud data architectures based on Software-as-a-Service (SaaS) being available on demand, a kind of provisioning well known for being difficult to measure, control, and predict costs.

Due to this, rationalizing data pipeline costs has become not only important but crucial to guaranteeing the right return on investment and making data analysis...