Airflow Orchestrator - Shared Data Pipeline Engine

A shared Apache Airflow deployment that orchestrates BigQuery + dbt data pipelines for multiple Django apps, without owning any business logic itself: it mounts each project as a volume and runs that project's own management commands and dbt project via BashOperator.

One engine, N projects mounted on top - not a stack per project.

A separate Airflow stack per project means four more containers each time - scheduler, worker, Postgres, Redis - on the same machine. This repo exists to avoid that.

---

What it does

• Runs DAGs versioned in each project's own repo, not this one
• One custom Docker image bakes in every project's data-layer dependencies
• Mounts each project's code and credentials as a volume, keeping host environments out of the containers
• A sync script places each project's DAG where the scheduler looks for it
• Currently orchestrates two production DAGs on daily and monthly schedules

---

Screenshots

---

Stack

Apache Airflow 3.3.0 (CeleryExecutor) · PostgreSQL 16 · Redis 7.2 · Docker

Baked into the image per project: dbt-bigquery · google-cloud-bigquery

---

Architecture

airflow_repo/
├── Dockerfile            # apache/airflow:3.3.0 + each project's requirements
├── docker-compose.yaml   # apiserver, scheduler, worker, triggerer, postgres, redis
├── scripts/sync_dags.sh  # syncs each project's DAG into dags/
├── dags/                 # synced DAGs land here
├── config/
└── plugins/

Each orchestrated project lives in its own sibling repo and owns its DAG file, its dbt project, and its credentials. This repo only provides the engine that runs them.

Dependencies are baked into the image instead of relying on each project's own venv: a venv created on Windows has binaries that don't run inside the Linux container, so baking them in avoids that mismatch entirely.

The sync script prefers a symlink over a copy, and verifies the result instead of trusting the exit code: on Windows, Git Bash can silently fall back to a real copy when symlink privileges (Developer Mode) aren't available. With a copy, editing a project's DAG file later won't show up until the sync script runs again - easy to miss unless you check.

---

Extending it

Adding a new project touches three things, all mechanical:

• Dockerfile: one more dependency install
• docker-compose.yaml: one more project volume
• scripts/sync_dags.sh: one more entry

No new image, no new compose file, no new Airflow install.

---

Run locally

Requirements: Docker.

git clone https://github.com/chipap-dev/airflow-orchestrator.git airflow_repo
cd airflow_repo

.env (git-ignored):

AIRFLOW_UID=50000
FERNET_KEY=<generate one below>

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

docker compose build
docker compose up -d
scripts/sync_dags.sh

Open http://localhost:8080 (login airflow / airflow).

---

Built by Claudia Cáceres · LinkedIn · Buenos Aires, Argentina