Managing the substantial amount of data that’s generated by every company daily is a complex endeavor. It calls for dedicated resources and technological support in the form of a specific data platform.

Nowadays, data platforms often fall short in delivering the expected value compared to the investments made, primarily due to organizations’ inability to sustainably manage the complexity they generate over time.

System complexity

The complexity of a system is determined by the number of its components multiplied by the number of correlations between them. A database with 10,000 tables is not much more complex than a database with 100 tables if the tables themselves are not correlated. Each table tells its own story. It can be manipulated without concern for the meaning of other tables and the potential impacts that the executed action may have on them. However, the complexity between the two databases is very...

If we look at the evolution of data management over the last 40 years, we’ll see a story of incredible technological revolutions and just as many project failures. At the beginning of this chapter, we mentioned that the main reason for these failures is the complexity generated by data management platforms, and this complexity grows approximately quadratically with the size of the platform. Therefore, these are not typical project failures as we are accustomed to understanding them. Data platforms rarely fail before their launch, never making it into production. Instead, they often experience failures related to their ability to evolve and survive over time. Platforms don’t fail immediately but over time, as they struggle to deliver the expected value in proportion to the constantly increasing maintenance costs they generate.

Like a Jenga tower becoming increasingly unstable as more pieces are added until it collapses...

To escape the quagmire we find ourselves in, it is necessary to radically change the mental model we use to approach data management and, consequently, the organizational structures and associated operational practices. It’s a systemic change – a paradigm shift in data management practice.

As we’ve seen in the previous sections, attempts to address these problems have been predominantly cosmetic, not radical. We’ve tried to modify the system tactically, reacting to surface-level problems as they arise.

In system thinking, a system can be changed from the outside by acting on parts of it where small changes can lead to significant and lasting changes over time; these parts are called leverage points. Donella Meadows, a renowned researcher in this field, has classified possible leverage points into 12 categories, ranking them by effectiveness. It’s not necessary to delve into the details of each...

There is no doubt about the importance that data holds for organizations today to compete in the market. However, how we traditionally manage data has led to the construction of monolithic platforms unable to survive the complexity they generate.

Technological innovations have helped us solve many problems related to data management over time, but not the root causes of these problems. With each new generation of data platforms, the same issues have always resurfaced.

In this chapter, we saw how a paradigm shift in data management is necessary to make our platforms sustainable over time – a paradigm shift centered around the idea of treating data as a product and building modular platforms capable of governing the intrinsic complexity they generate without collapsing. It is a systemic transformation that touches on different levels of the organization. Throughout, we have shown the desired transformation required for each level, from mental models to operating...

For more information on the topics that were covered in this chapter, please take a look at the following resources:

• An Architecture for a Business and Information System, by B. Devlin and Paul T. (1988): https://www.semanticscholar.org/paper/An-Architecture-for-a-Business-and-Information-Devlin-Murphy/c22ce1eeafb01f0682e194a2a22349aa141b78f6
• Building the Data Warehouse, by W. H. Inmon (1992): https://www.amazon.com/Building-Data-Warehouse-W-Inmon/dp/0764599445
• The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, by R. Kimball and Margy Ross (1996): https://www.amazon.com/Data-Warehouse-Toolkit-Definitive-Dimensional/dp/1118530802/
• Data Lake Architecture: Designing the Data Lake and Avoiding the Garbage Dump, by W. H. Inmon (2016): https://www.amazon.com/Data-Lake-Architecture-Designing-Avoiding/dp/B01HN4JOPC/
• The Modern Data Stack: Past, Present, and Future, by Tristan Handy (2020): https://www.getdbt.com/blog/future-of...