Skip to content
View AdarshDamarla-DataEngineer-Git's full-sized avatar
🎯
Focusing
🎯
Focusing

Block or report AdarshDamarla-DataEngineer-Git

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
AdarshDamarla-DataEngineer-Git/README.md
Adarsh Damarla — Cloud and Big Data Engineer

Hi, I'm Adarsh Damarla 👋

Cloud Data Engineer building reliable batch, streaming, and lakehouse platforms that turn source data into analytics-ready models.

GitHub projects

About me

I design and build end-to-end cloud data platforms for reliable batch and real-time analytics using Azure, AWS, Google Cloud, Databricks, Spark, Kafka, Airflow, and dbt—from data ingestion and orchestration to governed lakehouse layers and analytics-ready models.

What I build

  • Metadata-driven ingestion from databases, files, APIs, and event streams
  • Batch and streaming transformations with PySpark and Structured Streaming
  • Governed Bronze, Silver, and Gold lakehouse layers using Delta Lake and Unity Catalog
  • Incremental processing, CDC, and SCD Type 1/2 history management
  • Cross-platform orchestration with Azure Data Factory and Apache Airflow
  • Analytics-ready facts, dimensions, and serverless SQL datasets

Featured projects

Project Engineering highlights Stack
☁️ Real-Time AWS E-commerce Lakehouse Streams e-commerce events through MSK Serverless and MSK Connect into S3; incrementally builds Bronze, Silver, and Gold layers with Glue and Delta Lake; publishes an SCD-managed star schema to Redshift Serverless MSK Serverless, MSK Connect, S3, AWS Glue, PySpark, Delta Lake, Redshift Serverless
🎧 Spotify Azure Incremental Lakehouse Metadata-driven incremental ingestion from Azure SQL to ADLS Gen2; Auto Loader processing; SCD Type 2 dimensions and a Type 1 streaming fact ADF, ADLS Gen2, Databricks, PySpark, Delta Lake
🚕 Real-Time Uber Ride Lakehouse Streams FastAPI-generated ride events through Event Hubs; unifies historical and real-time data; publishes an enriched OBT and SCD-managed dimensional model FastAPI, Event Hubs, Structured Streaming, Lakeflow, Unity Catalog
🌬️ Airflow + dbt + Databricks Airflow 3 orchestrates Databricks ingestion and a dependency-aware dbt graph using deferrable tasks and distributed Celery workers backed by Redis and PostgreSQL Airflow, dbt, Databricks, Celery, Redis, PostgreSQL, Docker
🛰️ NASA GCN Fermi Streaming Pipeline OAuth-authenticated Kafka ingestion of NASA gamma-ray burst notices; native Spark parsing; governed analytical snowflake schema Kafka, OAuth 2.0, PySpark, Databricks, Delta Lake

More portfolio work

  • E-commerce Databricks Pipeline — Six data domains processed through Bronze, Silver, and Gold layers with data-quality expectations, CDC, SCD Type 2, and dimensional modeling.
  • Olist Azure Big Data Platform — Approximately 1.56 million marketplace records ingested from HTTP, SQL, and MongoDB, transformed with PySpark, and served through Synapse Serverless SQL.

Platform architecture

Sources flow through ingestion, cloud storage, processing, lakehouse layers, and analytics serving

Technical skills

Cloud platforms

Microsoft Azure AWS Google Cloud

Data processing and lakehouse

Databricks Apache Spark PySpark Delta Lake Unity Catalog Lakeflow

Streaming, orchestration, and transformation

Apache Kafka Apache Airflow Azure Data Factory dbt Docker

Languages and data stores

Python SQL PostgreSQL MySQL MongoDB Redis FastAPI

Cloud data services

ADLS Gen2 Event Hubs Synapse Analytics Amazon S3 AWS Glue Amazon EMR Amazon Redshift Cloud Storage Dataproc BigQuery Pub/Sub

Engineering patterns

Medallion Architecture Batch Processing Stream Processing Incremental Loading CDC SCD Type 1 & 2 Dimensional Modeling Data Quality

Current focus

  • Automated data testing, pipeline observability, and operational reliability
  • CI/CD and environment-based deployment for Databricks and cloud data platforms
  • Spark and Delta Lake performance tuning

Designing reliable paths from source systems to analytics-ready data.

Pinned Loading

  1. aws-ecommerce-streaming-lakehouse aws-ecommerce-streaming-lakehouse Public

    Real-time AWS ecommerce data pipeline using MSK Serverless, MSK Connect, S3, AWS Glue, Delta Lake, SCD Type 2, and Redshift Serverless.

    Python

  2. Spotify-Azure-Project Spotify-Azure-Project Public

    Built a metadata-driven Azure lakehouse that incrementally ingests Spotify-style SQL data with ADF, processes new Parquet files with Databricks Auto Loader, and publishes SCD-managed Delta facts an…

    Python

  3. Uber-Project Uber-Project Public

    Built a real-time Azure lakehouse that streams synthetic ride-booking events from FastAPI through Event Hubs into Databricks, unifies them with historical data, and publishes SCD-managed facts and …

    Jupyter Notebook

  4. Ecommerce-Databricks-Pipeline Ecommerce-Databricks-Pipeline Public

    Built a Databricks lakehouse pipeline that incrementally processes six e-commerce data domains through Bronze, Silver, and Gold layers, including data-quality enforcement, CDC, SCD Type 2 history, …

    Python

  5. Airflow-DBT-Databricks Airflow-DBT-Databricks Public

    Orchestrated a Databricks ingestion and dbt analytics pipeline with Apache Airflow 3.3, using deferrable remote-job execution, parallel model branches, data tests, and a containerized CeleryExecuto…

    Python

  6. Kafka-Databricks-Nasa Kafka-Databricks-Nasa Public

    Built an OAuth-authenticated Spark streaming lakehouse that consumes NASA GCN Fermi gamma-ray burst notices from Kafka, parses classic-text messages, and publishes a governed analytical snowflake s…

    Python