✈️ Flight Data Engineering Pipeline

An end-to-end Data Engineering project built on the Databricks Lakehouse Platform using the Medallion Architecture.

This repository demonstrates how raw airline booking data is ingested, transformed, and modeled into an analytics-ready Star Schema using native Databricks services including Auto Loader, Lakeflow Declarative Pipelines (LDP), Delta Lake, Unity Catalog, and Databricks Workflows.

Instead of building isolated ETL notebooks, this project focuses on designing a production-oriented data pipeline with reusable ETL components, incremental processing, Slowly Changing Dimensions (SCD), and workflow orchestration.

---

🏗️ Architecture

The solution follows the Medallion Architecture pattern, separating data processing into three independent layers:

• Bronze Layer → Raw data ingestion
• Silver Layer → Data cleansing and transformation
• Gold Layer → Dimensional modeling for analytics

---

🛠️ Technologies

• Databricks
• Unity Catalog
• Delta Lake
• PySpark
• Spark SQL
• Auto Loader
• Lakeflow Declarative Pipelines (LDP)
• Databricks Workflows

---

🚀 Key Features

🥉 Bronze Layer — Auto Loader Ingestion

• Incremental data ingestion using Databricks Auto Loader
• Raw data ingestion from Unity Catalog Volumes
• Bronze Delta table creation

---

🥈 Silver Layer — Lakeflow Declarative Pipeline

• Declarative ETL pipeline using Lakeflow Declarative Pipelines
• Data cleansing and standardization
• Schema management and duplicate handling
• Bronze-to-Silver transformation

---

🥇 Gold Layer — Star Schema Modeling

• Reusable Generic Dimension Builder
• Dynamic Fact Builder
• Incremental processing with Delta Lake MERGE
• Slowly Changing Dimension (SCD) implementation
• Surrogate key generation
• Analytics-ready dimensional model

---

🔄 Workflow Orchestration

The complete ETL process is automated using Databricks Workflows.

The pipeline is separated into four workflow components:

1. Flight Bronze Ingestion

Responsible for:

• Reading raw files from Unity Catalog Volume
• Running Auto Loader ingestion
• Creating Bronze Delta tables

---

2. Flight Silver Pipeline

Responsible for:

• Running Lakeflow Declarative Pipeline
• Transforming Bronze data
• Creating Silver tables

---

3. Flight Gold Pipeline

Responsible for:

• Building dimension tables
• Processing fact tables
• Applying incremental loading logic

---

4. Flight ETL Pipeline

The master workflow orchestrates the complete end-to-end pipeline.

Execution flow:

Flight Bronze Ingestion
          │
          ▼
Flight Silver Pipeline
          │
          ▼
Flight Gold Pipeline

---

📂 Unity Catalog

This project uses Databricks Unity Catalog to organize and govern all data assets across the Medallion Architecture.

The catalog is structured into multiple schemas representing different processing layers:

flight_catalog
│
├── source
│   └── Volumes
│       └── raw_data
│           ├── airports
│           ├── bookings
│           ├── flights
│           └── passengers
│
├── bronze
│   └── Volumes
│       └── landing
│           ├── airports
│           ├── bookings
│           ├── flights
│           └── passengers
│
├── silver
│   └── Tables
│       ├── silver_airports
│       ├── silver_bookings
│       ├── silver_flights
│       └── silver_passengers
│
└── gold
    └── Tables
        ├── dim_airports
        ├── dim_flights
        ├── dim_passengers
        └── fact_bookings

---

📁 Repository Structure

flight-data-engineering

├── bronze/                                  # Bronze layer - raw data ingestion
│   ├── bronze_ingestion.ipynb               # Auto Loader ingestion notebook
│   └── bronze_parameters.ipynb              # Bronze pipeline parameters
│
├── silver/                                  # Silver layer - data cleansing and transformation
│   ├── silver_dev.ipynb                     # Silver development notebook
│   └── Pipeline/                            # Lakeflow Declarative Pipeline source code
│       ├── airports.py
│       ├── bookings.py
│       ├── flights.py
│       └── passengers.py
│
├── gold/                                    # Gold layer - dimensional modeling
|   ├── gold_dev.ipynb                       # Gold development notebook
│   ├── dimension.py                         # Generic Dimension Builder
│   ├── fact.py                              # Generic Fact Builder
│   │
│   ├── dim_airports.ipynb                   # Airport dimension implementation
│   ├── dim_flights.ipynb                    # Flight dimension implementation
│   ├── dim_passengers.ipynb                 # Passenger dimension implementation
│   └── fact_bookings.ipynb                  # Fact table implementation
│
├── datasets/                                # Source airline datasets
│
├── images/                                  # README documentation images
│
├── SETUP.md                                 # Detailed project setup guide
├── LICENSE                                  # License information
└── README.md                                # Project overview and documentation

---

⭐ Star Schema

The Gold layer provides an analytics-ready dimensional model using a Star Schema design.

Fact Table

fact_bookings

Dimension Tables

dim_airports
dim_flights
dim_passengers

---

📚 What I Learned

This project provided practical experience building a modern Data Engineering platform using Databricks native services.

Key learnings:

• Building incremental ingestion using Auto Loader
• Designing ETL pipelines with Lakeflow Declarative Pipelines
• Implementing Delta Lake MERGE operations
• Building Slowly Changing Dimensions
• Designing reusable Generic Dimension and Fact Builders
• Creating analytics-ready Star Schema models
• Orchestrating pipelines using Databricks Workflows
• Managing data assets using Unity Catalog

---

💡 Future Improvements

• Metadata-driven ETL framework
• Data Quality Expectations
• Automated testing
• CI/CD deployment pipeline
• Data lineage implementation
• Monitoring dashboard
• Parameterized workflow deployment

---

🚀 Getting Started

Detailed deployment instructions are available in:

SETUP.md

The setup guide covers:

• Unity Catalog configuration
• Volume creation
• Source data preparation
• Auto Loader deployment
• Lakeflow Declarative Pipeline configuration
• Gold layer execution
• Workflow orchestration setup