Started writing on DEDP Dynamic-Queries and Open Data Platform Design Pattern (Lakehouse)
new CE's and DEPs like IaC and DevOps, devops-de-pattern

14th of March 2025

Published new chapter Data-Asset Reusability Pattern

13th of January 2025

Started new chapter Declarative Data Pipeline Design Pattern

10-14th of November 2024

Published new chapter Data Contract vs. Schema Evolution chapter.
NoSQL part of Data Contract vs. Schema Evolution chapter.
Big push and finalizing Schema Evolution on the Data Contract vs. Schema Evolution chapter.

18th of June 2024

Finalizing the first draft and release of the Cache Pattern chapter.

16th of June 2024

Updated a big part of the Cache pattern chapter.

28th of May 2024

Started with the first pattern called Cache.

14th of May 2024

Started with the Data Contract vs. Schema Evolution chapter.

16th of April 2024

Release of Data Warehouses vs. Master Data Management vs. Data Lakes vs. Reverse-ETL vs. CDP chapter.

2nd / 9th of April 2024

Big update on the ETL chapter, working on data lakes, reverse ETL, and master data management and finalizing a first draft of Data Warehouses vs. Master Data Management vs. Data Lakes vs. Reverse-ETL vs. CDP chapter.

27th of February 2024

Started with the ETL-ELT-MDM chapter, focusing on ETL and its related patterns.

20th of February 2024

Finalizing the first draft of the orchestration chapter, focusing on Python frameworks and underlying patterns.

15th of February 2024

I updated the chapter orchestration chapter with Bash, Stored Procedure, and Python-Script, and added a fourth term to compare among the others: Traditional ETL tools, which completes this chapter for the history of orchestration.

8th of February 2024

Added commentary feature to chat/discuss with fellow writers. Check, for example, Feedback.

5th of February 2024

I added a subscription to buy parts two and three of the book. This was a hard decision for me, as I want to share everything I write with you all. But I also had to admit that I spent a lot of time distilling all my free resources into this one condensed book.
If you find this book valuable, the small amount is, in my mind, justified to get the latest up-to-date content in data engineering. I'm sorry if you are disappointed. It's also a way to spend more time on this book and writing in general. Check out the subscription page for more information.
I updated the copyright and legal notice and privacy policy to reflect the technology used for managing members.

24th of January 2024

Started the new chapter Bash vs. Stored Procedure vs. Python-Script and wrote the initial draft bash/cron part.
Discovered a new pattern called Implicit Orchestration

20th of January 2024

We finalized and published a new chapter: Materialized View vs. One Big Table (OBT) vs. dbt Table vs. Traditional OLAP Cube vs. DWA.
Added a new sub-chapter, Data Warehouse Automation.

31st of December 2023

I updated dbt and OBT to the chapter Materialized Views vs. One Big Table (OBT) vs. dbt Tables vs. Traditional OLAP Cubes.
A new pattern occurred while writing: Open Table Formats

12th of December 2023

Beginning of Materialized Views vs. One Big Table (OBT) vs. dbt Tables vs. Traditional OLAP Cubes

7th of December 2023

Social announcements: Twitter, LinkedIn, Reddit, and hitting Hackernews for a good 18 hours.

6th of December 2023

🚨 Go live without announcements. Checking if everything is working putting up some pressure ;)
I added up to 60 new terms to the Data Engineering Vault that link out from this book.

5th of December 2023

Writing the bulk of Understanding Convergent Evolution, combining it with the factor time.

26th of September 2023

Re-structure the TOC for the 5th time, but this time it feels right. I am preparing to release the first version.

24th of April 2023

buying of domain dataengineeringdesignpatterns.com
uploading Skeleton without content

Get Updates in your Emails

Or get notified with the latest news from my newsletter by subscribing below.

Subscribe for latest Updates to this Book

Technical (Web Site)

20th of February 2024

I added a frontmatter parser to add a feature image to the frontmatter and description for each chapter, making it nicer to share online.

17th of February 2024

Upgraded mdBook to latest v0.4.37 version which includes my PR fixing input in forms.

21st of December 2023

I added RSS-Feed. But be aware that dates are always updated with the latest build, not when content changes (might be very verbose). Will check later if I can optimize it. The code is here and mdBook-issue.

11th of November 2023

added two themes by request of my daughters. Pink Rose and Tokyo Night. Looking really good. I also removed redundant themes.

21st of June 2023

Automation: auto link to my Second Brain and Data Engineering Vault with mdbook-sspaeti.

5th March 2023

Setting up website and framework with mdBook
Adding custom Theme Kanagawa

Shout-outs

Thanks to the Bluesky Community for helping me with the history of data contracts.
Thanks, Karim for the disclaimer inspiration ;) and generally a good starting point structure for me to start with too. Check out his book on Data With Rust.
Thanks, Bartosz Bednarowicz, for highlighting the Data Transformation Services (DTS) tool used in early 1998 for Traditional ETL Tools.
Thanks to Jeffrey for pointing out the missing Hadoop orchestration tools, such as Luigi, Oozie, and Azkaban, in the orchestration chapter.