Data Engineering TL;DR Cheatsheet

Learn Data Engineering 10× faster with the cleanest, zero fluff content

Data Engineering TL;DR

Core Data Engineering

Pipeline Design Patterns
Batch vs Streaming • Lambda/Kappa • Idempotency • Late Data
TL;DR
Data Modeling
Star Schema • Kimball • Data Vault • Slowly Changing Dimensions
TL;DR
Partitioning & Bucketing
Date • Region • Skew • Z-Order • Hive vs Delta
TL;DR

Lakehouse & Medallion Architecture

Medallion Architecture
Bronze • Silver • Gold • Quality Gates • DLT
TL;DR
Data Quality & Testing
Great Expectations • dbt tests • Deequ • Monte Carlo
TL;DR
Data Lineage & Observability
Unity Catalog • OpenLineage • Monte Carlo • Anomaly Detection
TL;DR

Orchestration & CI/CD

Apache Airflow
DAGs • Operators • XComs • TaskFlow • Astronomer
TL;DR
dbt (data build tool)
Models • Tests • Docs • Jinja • Snapshots • Exposures
TL;DR
DE CI/CD Best Practices
GitOps • Databricks Repos • dbt Cloud • Terraform • Promotions
TL;DR

Streaming & Real-Time

Kafka Fundamentals
Topics • Partitions • Consumer Groups • Exactly-once • Schema Registry
TL;DR
Structured Streaming
Watermarks • Triggers • ForeachBatch • Checkpointing
TL;DR
Apache Flink
Stateful • Event Time • Checkpoints • Table API
TL;DR

File Formats & Optimization

Parquet Deep Dive
Columnar • Compression • Row Groups • Predicate Pushdown
TL;DR
Delta vs Iceberg vs Hudi
ACID • Schema Evolution • Time Travel • MERGE Performance
TL;DR
Compaction & Vacuum
Small Files • OPTIMIZE • ZORDER • Retention • Bin Packing
TL;DR

Cost, Scale & Reliability

Cost Optimization
Spot • Auto-termination • Photon • Right-sizing • Storage Tiering
TL;DR
Disaster Recovery
Cross-region • Meta-store backup • Delta clones • Failover
TL;DR
Monitoring & Alerting
Ganglia • Spark UI • Datadog • Prometheus • SLOs
TL;DR