Data Engineering Design Patterns

Data engineering has gained even more relevance than before, and data engineering patterns are key to the successful implementation of data engineering projects. This book enables a data engineer to not only become familiar with data engineering patterns but also understand their application in real world use cases.

This book presents a comprehensive collection of data engineering patterns, each illustrated with relevant enterprise use cases to highlight their value and simplicity. It showcases both open-source and cloud technologies, guiding readers in building data systems for on-premise and cloud environments. The book covers patterns for data ingestion, transformation, storage, and serving, while also offering insights into performance engineering for data pipelines. Once we understand fundamental data engineering patterns, we then shift focus to patterns that help us build high-performance low latency data systems. We cover data caching, partitioning, replication, and how to select the technology stack for building out the patterns in this book.

By the end of the book, readers will have a deep understanding of various data engineering use cases and will be able to map the appropriate patterns to address them. They will also be equipped to choose the right technical stack for implementing these patterns, enabling them to create robust and efficient data systems in a secure and a cost-effective manner.

WHAT YOU WILL LEARN
● Key data engineering patterns.
● Data ingestion and processing patterns.
● Modern architectures like Lambda.
● Explore time-tested data patterns of ETL and ELT.
● Modern data systems like data lake and medallion architectures.
● Domain-specific patterns and also on data orchestration, observability, and security.
● Overcoming performance challenges in building complex data systems.

WHO THIS BOOK IS FOR
This book is designed for data engineers with beginner to intermediate experience in building enterprise-grade data systems. ETL developers transitioning into data engineering roles will also find this book valuable for understanding essential data engineering patterns. The code snippets provided throughout the book are written in Python or Scala, so a basic understanding of either language will help readers more easily grasp the concepts presented.

1. Understanding Data Engineering
2. Data Engineering Patterns, Terminologies, and Technical Stack
3. Batch Ingestion and Processing
4. Real-time Ingestion and Processing
5. Micro-batching
6. Lambda Architecture
7. ETL and ELT
8. Data Fundamentals
9. Databases and Transactional Data
10. Data Warehouse and Data Analytics
11. Data Lake and Medallion Architecture
12. Data Replication and Partitioning
13. Hot Versus Cold Data Storage
14. Data Caching and Low Latency Serving
15. Data Search Patterns
16. Domain Specific Patterns
17. Data Security Patterns
18. Data Observability and Monitoring Patterns
19. Idempotency and Deduplication Patterns
20. Data Orchestration Patterns
21. Common Performance Pitfalls
22. Technology and Infrastructure Selection
23. Recap and Next Steps

● Amit Kulkarni has 14+ years of experience working in distributed systems, databases, and cloud storage systems. As a senior manager working in Couchbase India, he has gained expertise in building and managing large-scale, performant, and fault-tolerant systems. Amit also has a strong background in data protection and cloud storage solutions, having worked at industry-leading companies like Druva and NetApp. With a deep understanding of databases, cloud storage, and disaster recovery, Amit brings a wealth of knowledge to the tech community. Being an alumnus of the prestigious institute of IIT Kanpur, Amit brings in strong expertise in computer science fundamentals, required to design large- scale and high-performance data solutions.

● Santosh Hegde is a seasoned technology leader with 18+ years of experience specialising in distributed systems and data engineering. As the senior director of engineering at Couchbase India, he leads the core R&D team, focusing on transactional and analytical NoSQL database systems. Previously, as director of engineering at Visa, he advanced SQL technologies significantly. At IBM Software Lab, he made impactful contributions to various database projects. Santosh holds multiple patents in distributed systems and database internals, underscoring his innovative contributions to the field.