SmartGrid AI — Real-Time Energy Demand & Grid Optimization Platform

An end-to-end platform that ingests simulated smart-meter telemetry in real time, detects abnormal usage as it happens, forecasts electricity demand 24 hours ahead with an LSTM, and lets an LLM-powered analyst copilot answer questions grounded in the live grid state.

The second copilot exchange above is worth pointing out: asked "which region is the closest peak?", it declines to guess rather than hallucinate an answer, explaining exactly what data it does and doesn't have. That's the anti-hallucination grounding actually working, not just claimed.

This repo is a working vertical slice of the full architecture below — every box in the diagram either runs locally via Docker Compose or has a documented, concrete path to its AWS production equivalent (infrastructure/aws/README.md).

SMART METERS (simulated)
        │
        ▼
   APACHE KAFKA  ──────────────► AWS S3 data lake (prod; see infra docs)
        │                               │
        ▼                               ▼
  Stream consumer                   AIRFLOW (daily DAG)
  (online z-score alerts)          clean → score anomalies → retrain
        │                               │
        ▼                               ▼
   PostgreSQL  ◄─────────────── TensorFlow LSTM forecaster
        │                               │
        └───────────────┬───────────────┘
                         ▼
                  FastAPI backend
             /readings /forecast /alerts
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
        React dashboard  Power BI   LLM Copilot (Gemini)

What's real vs. what's a documented design

Layer	Status
Kafka producer/consumer, Postgres, FastAPI, React dashboard	Runs locally, docker compose up
TensorFlow LSTM forecaster	Actually trained — not checked into git (.keras binaries are gitignored by design), regenerate it with ml/forecasting/train.py; see Results below for real held-out numbers
IsolationForest batch anomaly detector	Implemented + unit tested, run manually or via the Airflow DAG — see Results
LLM copilot	Fully wired, needs your own GEMINI_API_KEY to actually call the model (free tier: https://aistudio.google.com/apikey)
Airflow orchestration	DAG is written and correct (airflow/dags/); not included as a service in docker-compose.yml since a full Airflow stack is heavy for local dev — see airflow/README.md to run it
AWS (MSK, RDS, EKS, MWAA, S3)	Design doc, not deployed — infrastructure/aws/README.md
Power BI	Spec, not a .pbix file (binary + credentials don't belong in a repo) — powerbi/dashboards/README.md
Kubernetes manifests	Written and ready to kubectl apply against a real cluster, not applied here

Being upfront about this split is deliberate — it's the honest story for a portfolio project: a real, running core, with the rest of the stack designed to the same standard rather than faked.

Live demo

Deploy your own copy on Render using render.yaml (free tier, no credit card).

One thing the hosted demo fakes, on purpose: the real pipeline (kafka/producers/meter_simulator.py -> Kafka -> kafka/consumers/db_writer.py) needs Kafka and Zookeeper running continuously, which no free host supports (they sleep idle services; a sleeping consumer stalls the stream). The hosted demo instead runs backend/app/demo_seed.py -- the exact same load-curve simulation and rolling z-score alerting math as the real pipeline, as a background thread that writes straight to Postgres instead of through Kafka. Set DEMO_MODE=true to enable it; local docker compose up doesn't set this, so local dev still exercises the real Kafka path. Everything downstream of the data layer -- the LSTM forecaster, the IsolationForest detector, every FastAPI route, the copilot -- is identical code in both places.

Quick start (local)

Requires Docker and Docker Compose.

cp .env.example .env
# Optionally set GEMINI_API_KEY in .env to enable the copilot

# Train the forecaster first -- forecaster_central.keras is gitignored
# (binaries don't belong in git), so without this step the API falls
# back to a naive seasonal forecast instead of the trained LSTM.
# Runs in a throwaway container so it works regardless of your host
# Python version -- TensorFlow doesn't support the newest CPython yet.
docker run --rm -v "$(pwd)/ml/forecasting:/app" -w /app python:3.11-slim \
  bash -c "pip install -r requirements.txt && python train.py --region central --epochs 15"

docker compose up --build

• Dashboard: http://localhost:5173
• API docs (Swagger): http://localhost:8010/docs
• Postgres: localhost:5433 (smartgrid / smartgrid)

What happens on startup:

1. Kafka + ZooKeeper come up.
2. meter-simulator starts streaming 25 synthetic meters into the smart-meter-readings topic.
3. db-writer consumes the stream, persists readings to Postgres, and flags obvious spikes in real time via a rolling z-score.
4. backend serves the API. If you ran the training step above, it loads the trained forecasting model from ml/forecasting/artifacts/ (mounted as a volume); otherwise it transparently falls back to a naive seasonal forecast so the API never 500s.
5. frontend polls the API every 5s and renders the control-room dashboard.

Give it a minute or two after startup for enough readings to accumulate before the live chart looks interesting.

Retraining the forecasting model

cd ml/forecasting
pip install -r requirements.txt   # numpy, pandas, tensorflow-cpu
python train.py --region central --epochs 15

Python version note: TensorFlow doesn't ship wheels for the newest CPython yet (as of this writing, 3.13 is the latest it officially supports) — pip install -r requirements.txt will fail with "no matching distribution" on Python 3.14+. Rather than juggling a second Python install on your host, run this same step inside a throwaway container instead (from the repo root, not ml/forecasting/):

docker run --rm -v "$(pwd)/ml/forecasting:/app" -w /app python:3.11-slim \
  bash -c "pip install -r requirements.txt && python train.py --region central --epochs 15"

The bind mount means artifacts/forecaster_central.keras lands on your host filesystem exactly like a normal local run — docker-compose.yml already expects it there. Same fix applies to eval_forecast.py.

This regenerates forecaster_central.keras from a synthetic 90-day hourly history with realistic daily/weekly seasonality. In production, Airflow runs this same script nightly against real aggregated consumption instead of the synthetic generator (see the DAG in airflow/dags/).

Then check how well it actually did:

python eval_forecast.py --region central

Running the tests

pip install -r tests/requirements.txt -r ml/anomaly_detection/requirements.txt
cd backend && pip install -r requirements.txt && cd ..
pytest tests/ -v

Covers the IsolationForest anomaly detector (catches injected outliers, degrades gracefully on tiny regions) and the forecaster's fallback path (naive seasonal forecast when no trained model is present, so the API never 500s even before the first training run).

Results

Numbers below are measured, not estimated -- reproduce them yourself with python ml/forecasting/eval_forecast.py and python ml/anomaly_detection/eval_anomaly.py. All figures are on the synthetic benchmark data this repo ships with (see the "what's real" table above); they describe model behavior, not a production deployment.

LSTM forecaster (ml/forecasting/train.py, 15 epochs, 168h lookback -> 24h horizon, 1,673 train / 296 held-out validation windows over a 90-day synthetic series):

Model	MAE	MAPE	RMSE
LSTM (trained)	8.22 MW	8.96%	10.53 MW
Naive seasonal baseline	53.00 MW	63.18%	65.17 MW

84.5% lower MAE than the naive seasonal fallback on held-out windows.

Anomaly detector (ml/anomaly_detection/detector.py, IsolationForest, contamination=0.01), evaluated against a synthetic anomaly rate matched to that contamination setting:

Metric	Value
Precision	0.91
Recall	0.95
F1	0.93

Recall drops sharply if the true anomaly rate exceeds the configured contamination -- e.g. at a 3% injected rate against the same 1% setting, recall falls to ~0.35 because the model is capped at flagging its top 1% as anomalies regardless of how many true anomalies exist. Worth tuning contamination to your actual expected anomaly rate rather than leaving the default, and worth mentioning in an interview as a known limitation you understand rather than one you missed.

Project structure

smartgrid-ai/
├── kafka/
│   ├── producers/          # smart meter simulator -> Kafka
│   └── consumers/          # Kafka -> Postgres + inline anomaly flagging
├── backend/                # FastAPI: readings, forecast, alerts, copilot
│   └── app/
│       ├── routers/
│       └── ml/             # forecaster loader (trained LSTM + fallback)
├── ml/
│   ├── forecasting/        # LSTM training + eval_forecast.py (accuracy vs. naive)
│   └── anomaly_detection/  # IsolationForest scorer + eval_anomaly.py (precision/recall)
├── airflow/dags/           # daily retrain/clean/score pipeline
├── llm/prompts/            # copilot system prompt
├── frontend/                # React + Vite control-room dashboard
├── infrastructure/
│   ├── docker/postgres/    # schema (init.sql)
│   ├── kubernetes/         # Deployment/Service/HPA/Ingress manifests
│   └── aws/                # production architecture design doc
├── powerbi/dashboards/     # BI report spec (data sources, refresh cadence)
├── tests/
└── docker-compose.yml

Design notes worth calling out in an interview

• Two-tier anomaly detection: a cheap single-feature rolling z-score runs inline on the Kafka stream for sub-second alerting (kafka/consumers/db_writer.py), while a heavier, multivariate IsolationForest (ml/anomaly_detection/detector.py) runs in Airflow batch pass to catch subtler anomalies the online check would miss. This is a standard streaming-vs-batch tradeoff, not an oversight.
• Graceful degradation: the forecast endpoint never fails — if no trained model is present, it falls back to a seasonal-naive forecast (backend/app/ml/forecaster.py), so the API and dashboard stay usable through a fresh deploy or before the first training run completes.
• LLM copilot is grounded, not generative-only: every chat request pulls live region totals, active alerts, and the current forecast from Postgres and injects them into the prompt, with an explicit system-prompt instruction not to state numbers absent from that snapshot (llm/prompts/copilot_system_prompt.txt).