Data platforms where pipelines fail loudly — not silently
I design and build cloud data platforms that survive partial failures, schema drift, and operational scale — incremental ingestion with checkpoint recovery, contract-driven data quality, medallion lakehouse patterns, and CI-validated platform automation on AWS, Azure, Databricks, and Snowflake.
Senior data engineering, to me, is judgment under constraint: intentional trade-offs, failure-aware design, and systems the next team can actually operate.
Design for failure.
Automate repeatable work.
Measure data quality.
Document decisions.
Keep systems understandable.
| Signal | How it shows up |
|---|---|
| System design | Architecture docs, ADRs, and clear boundaries between ingestion, transform, quality, and platform layers |
| Failure handling | Checkpoint recovery, idempotent loads, quality gates before bronze, operations runbooks |
| Operational proof | pytest, GitHub Actions CI, smoke tests, alert routing, and honest trade-off documentation |
Three connected layers — not isolated demo repos:
| Layer | Project | Focus |
|---|---|---|
| 1 · Ingest & transform | production-data-pipeline | Incremental API ingestion · PostgreSQL bronze · dbt silver/gold · Airflow · webhook alerts · v0.1.0 |
| 2 · Quality & observability | data-quality-observability | YAML data contracts · schema/null/uniqueness/freshness checks · run history · alert routing |
| 3 · Platform & governance | cloud-lakehouse-blueprint | Medallion manifests · Terraform IaC · IAM governance · lineage · validate --json for CI |
Outcome: pipelines that fail loudly, contracts that catch drift early, and platform patterns teams can run without tribal knowledge.
| Domain | Technologies & practices |
|---|---|
| Data engineering | Python · SQL · incremental ingestion · ETL/ELT · API pipelines · checkpointing · idempotent loads |
| Data architecture | Medallion lakehouse · bronze/silver/gold layers · lineage · governance · cost modeling |
| Orchestration | Apache Airflow · dbt · Prefect · Spark |
| Cloud platforms | AWS · Azure · Databricks · Snowflake |
| Quality & observability | Data contracts · schema validation · freshness checks · alerting · CI/CD |
| Article | Topic |
|---|---|
| Building a Production Data Pipeline with Incremental Loading and dbt | Incremental checkpoints, idempotent loads, medallion layering, Airflow orchestration, failure modes, and scale trade-offs |
More at br413.github.io · series: Cloud Data Platform Patterns
Production operations knowledge contributed upstream:
| Project | PR | Change |
|---|---|---|
| Prefect | #22500 ✓ merged | Kubernetes readiness vs liveness probes for orchestrator deployments |
| Prefect | #22533 | Clarify global concurrency limit setup for production deployments |
| dbt docs | #9606 | Troubleshooting note for prefixed custom schema names |
I publish focused, production-style data platform projects — not toy demos. Each flagship repo includes architecture decision records, pytest coverage, GitHub Actions CI, operations runbooks, and documented trade-offs.
Open to:
- Senior data engineering and data platform architect roles
- Data architecture reviews and lakehouse design discussions
- Data quality, observability, and pipeline automation tooling
- Technical collaboration with engineering teams and OSS maintainers
Website: br413.github.io · Writing: Dev.to · Flagship: production-data-pipeline · Quality: data-quality-observability · Platform: cloud-lakehouse-blueprint
Focus areas: data engineering · data architecture · cloud data platforms · lakehouse · ETL pipelines · dbt · Airflow · data contracts · observability · Terraform · AWS · Azure · Databricks · Snowflake