Data Engineering Best Practices

Data engineering is the backbone of modern business intelligence, yet navigating the complexities of roles and tools can be challenging for new and experienced professionals alike. However, data engineering sits at the core of modern analytics. As organizations scale their use of data, they need robust architecture, reliable pipelines, and strong governance to turn raw data into trusted insights.

This book follows the journey of data from source to insight. It defines the data engineering role, presents reference architectures, and explains how to model, secure, and govern data for analytics. Subsequent chapters cover CI/CD, ETL versus ELT, infrastructure operations, data quality, operations, AI, and supporting processes.

By the end of this book, the readers will possess the competency to build, design, and operate end-to-end data platforms, collaborate effectively with analysts and data scientists, and apply repeatable patterns to build secure, scalable, and high-quality data solutions.

WHAT YOU WILL LEARN
● Grasp the core responsibilities of modern data engineers.
● Design practical analytics and data platform architectures.
● Model data for performance, clarity, and governance.
● Secure, test, and automate pipelines with CI/CD.
● Design agnostic models and analyze topologies.
● Apply data operations to analytics, AI, and daily operations.

WHO THIS BOOK IS FOR
This book is designed for data engineers, analysts, BI developers, and scientists building analytics platforms and pipelines, and it also guides the professionals responsible for data strategy, governance, and reliable data-driven decisions.

1. Data Engineering's Role
2. Reference Architectures
3. Data Models
4. Permission Management
5. Governance and Cataloguing
6. Continuous Integration and Deployment
7. ETL and ELT
8. Infrastructure Operations
9. Quality Assurance
10. DataOps and AI
11. Additional Processes
12. Popular Technologies

● Luiz Fernando F Dos Santos is a Sr. manager, head of analytics at Amazon Global Logistics, leading data engineering and analytics initiatives that power global supply-chain decisions. He specializes in modern data platforms, ETL pipelines, and cost-optimized architectures, and is passionate about turning complex operations data into actionable insight for business leaders.

● Chandan Ramanna has spent over a decade working with infrastructure architecture, information security and AI. He is currently a Sr. data engineer at Amazon, where he designs scalable data systems focused on automation, security, and efficiency.

He is a strong believer that the best systems should not require constant maintenance; they should be secure and built to sustain.