Preface

When I started writing this book, I looked back at my experience in architecting and developing data engineering solutions, delivering and running those solutions effectively in production, and helping many companies to build and manage scalable and robust data pipelines and asked myself – What are the most useful things that I can share to help an aspiring or beginner data architect, a data engineer, or a Java developer to become an expert data architect? This book reflects the work I do on a daily basis, to design, develop, and maintain scalable, robust, and cost-effective solutions for different data-engineering problems.

Java architectural patterns and tools enable architects to develop reliable, scalable, and secure data engineering solutions to collect, manipulate, manage, and publish data. There are many books and online materials that discuss data architectures in general. There are other sets of books and online materials that focus on and dive deep into the technology stack. While such materials provide architects with essential knowledge, they often lack details on how an architect should approach a data engineering problem practically and create the best-suited architecture by using logical inference. In this book, I have tried to formalize a few techniques by which a data architect can approach a problem to create effective solutions.

In this book, I will take you on a journey in which you learn the basics of data engineering and how to use the basics to analyze and propose solutions for a data engineering problem. I also discuss how a beginner architect can choose the correct technology stack to implement a solution. I also touch upon data security and governance for those solutions.

One of the challenges that architects face is there is always more than one way to do things. We also discuss how to measure different architectural alternatives and how you can correctly choose the best-suited alternative using data-driven techniques.