This book is different from usual IT books.
The Idea, instead of being a reference handbook, this book is a story from start to finish for the most part. Convergent evolution (CE) and patterns can be explored on their own. Still, each chapter builds up to the actual data engineering design patterns (DEDP). If you skip, you might not understand the underlying history and concepts of certain concepts lived ahead of them.
My 20-year journey in the field—transitioning from business intelligence to big data, and eventually, data engineering. Along the way, I've explored the convergent evolution of design patterns, detailing my experiences as a data engineer and technical author.
This book provides a bottom-up approach from standard data engineering terms established over the last two decades, analyzing beyond the hype. Analyze different convergent evolution and their definitions, history, and core concepts, finding foundational patterns they follow despite the various evolutions they were invented. Based on the several CEs and found data engineering patterns, I tried to convert them into best practices and applicable data engineering design patterns.
After a short intro to the book and the relevant terms, you can choose to explore the convergent evolution and the design patterns interesting to you and read through how I discovered them through relevant CE and arrived at a recurring pattern and ultimately, the design patterns.
I will add personal anecdotes from my 20+ years of experience where I find it appropriate and to be adding value.
Good to know:
- All definitions of data engineering that are outdated quickly are referenced to the Data Engineering Vault where they get updated on a more regular basis. Sometimes, I skip over a deeper explanation, as you'll find more details in the referenced Vault article. As this is the online version of the book, it's easy to look up each term you do not know and skip the one you do know.
- The definitions of terms in the field are often outdated faster than this book can cope with, that's why the focus of this book is on the patterns that are long-term and updated less frequently.
And lastly, the book is designed to be written short and succinctly. This hopefully leads to a more interesting book by cutting out anything non-relevant.
Navigation on the website is trivial, but here are some hints:
sanywhere for a search within the whole book
- Hit the burger icon for toggle the left panel with chapter
- hit the brush for changing the theme (I recommend
Rose Pink). Light theme is the default, as Diagrams will be shown best with a white background.
To align on terminologies used in this book, here are the common abbreviations for this book in the glossary and explainer for callouts that highlight certain parts of this book.
The call-outs (also known as
admonitions) are hints to highlight certain things. It's a way to include less relevant side content without significantly interrupting the reading flow.
Below are the most used callouts shortly, explaining what you can expect from them.
This icon represents info that can be left out of the context of this book, but might still be interesting to know and relevant.
Hopefully, most terms are explained with the first use in this book, but in any case here are some abbreviations and common terms I use in this book as a reference.
|Data Engineering Pattern
|Data Engineering Design Pattern
|Modern Data Stack
|Open Data Stack
For more details and terms on my Data Engineering Vault, simply hit
Cmd+k to search for any term.