A production ML system that scores loan applications in real time using an XGBoost + LSTM ensemble served via ONNX INT8, routed through a LangGraph multi-agent pipeline for edge-case analysis, and monitored with automated drift detection and retraining.
| Metric | Target |
|---|---|
| p95 inference latency | < 150ms end-to-end |
| Concurrent request capacity | 10,000+ |
| SLA uptime | 99.9% |
| Feature train-serve skew incidents | -40% vs naive baseline |
| False-positive review rate reduction | 20%+ via agent routing |
| AUC-ROC improvement | 8%+ over rule-based baseline |
flowchart LR
A[Loan Application] --> B[Kafka\nloan-applications-raw]
B --> C[Schema Validator\nPydantic v2]
C -->|invalid| D[DLQ\nloan-applications-dlq]
C -->|valid| E[PySpark Pipeline\nAWS EMR]
E --> F[Feast Feature Store\nS3 offline / Redis online]
F --> G[ONNX Ensemble\nXGBoost + LSTM INT8]
G -->|confidence >= 0.75| H[Auto Decision\napprove / decline]
G -->|confidence < 0.75| I[LangGraph Agents\nretrieval - classification - escalation]
I -->|resolved| H
I -->|unresolved| J[Human Review Queue]
G --> K[FastAPI /v1/score\nEKS + HPA]
K --> L[Prometheus + Grafana]
F --> M[Evidently Drift Detector]
M -->|PSI > threshold| N[SNS Alert\nSageMaker Pipeline retrain]
Layer 1 - Event ingestion. Kafka MSK topic loan-applications-raw. Pydantic v2 schema validation on every message. Invalid messages routed to loan-applications-dlq with structured error metadata.
Layer 2 - Feature engineering. PySpark on AWS EMR computes credit history aggregates, bureau signal ratios, behavioral velocity features, and application metadata. Output stored in Feast (S3 offline, Redis online). Great Expectations validation blocks materialization on data quality failures.
Layer 3 - Model training. XGBoost on tabular credit features + LSTM on 12-month payment history sequences. Weighted ensemble calibrated with isotonic regression. All runs tracked in MLflow. Both models exported to ONNX and quantized to INT8. Promotion requires AUC-ROC improvement >= 5% on holdout and quantization degradation < 0.5% AUC.
Layer 4 - Model serving. SageMaker endpoint backed by ONNX Runtime. EKS deployment with HPA (min 2, max 20 pods). Circuit breaker on Redis latency - falls back to cached features with feature_staleness_flag in the response. Rolling deployments only.
Layer 5 - Agent layer. LangGraph StateGraph with three nodes: retrieval (bureau data + ChromaDB similarity), classification (Claude structured output -> RiskDecision), escalation (compliance rules + human queue). Any decision with confidence < 0.75 is auto-escalated. Every state transition is audit-logged.
Layer 6 - Drift monitoring. Evidently runs daily PSI and KS tests against the last 24h of scoring inputs vs training distribution. Threshold breach publishes to SNS, which triggers a Lambda that starts a SageMaker Pipeline retraining job. Time from breach to trigger: under 5 minutes.
Layer 7 - API layer. FastAPI with async handlers. Rate limiting at 1000 req/s per service. OpenTelemetry distributed tracing. Structured JSON request logging. Prometheus metrics at /metrics. Grafana dashboard committed to infra/grafana/.
credit-risk-platform/
- .github/workflows/ # CI (lint + test), CD staging, CD prod
- infra/
- terraform/ # MSK, EMR, Redis, EKS, SageMaker modules
- k8s/ # deployment, hpa, service, configmap
- grafana/ # dashboard JSON (provisioned on deploy)
- src/
- ingestion/ # kafka-consumer, schema-validator, dead-letter-handler
- features/ # spark-pipeline, feast-definitions, feast-materialize, validation
- training/ # xgboost-trainer, lstm-trainer, ensemble, onnx-export, quantization
- serving/ # inference-handler, onnx-runtime, sagemaker-endpoint, quantization
- agents/ # graph, state, tools, prompts, retrieval, classification, escalation
- monitoring/ # drift-detector, retrain-trigger, alert-handler, metrics-exporter
- api/ # main, schemas, middleware, routers/score, routers/health, routers/admin
- tests/
- unit/ # 281 tests, all passing
- integration/ # train-serve parity test
- load/ # Locust load test (p95 < 150ms at 10K users)
- configs/ # base.yaml, staging.yaml, production.yaml
- notebooks/ # eda, model-evaluation, drift-analysis
- scripts/ # bootstrap-feast, run-backfill, promote-model
- requirements/ # base, dev, serving
- docs/runbook.md # operations runbook - 7 failure modes
- Dockerfile
- Makefile
- pyproject.toml
- CLAUDE.md
- Python 3.11+
- AWS CLI configured (
aws configure) - Terraform >= 1.5
- kubectl >= 1.28
- Docker >= 24
- An AWS account with permissions for MSK, EMR, ElastiCache, EKS, SageMaker, S3, SNS, Lambda, Secrets Manager
git clone <this-repo>
cd credit-risk-platform
python -m venv .venv
source .venv/bin/activate
pip install -r requirements/dev.txt
make test
# Expected: 281 passed# 1. Provision all infrastructure
cd infra/terraform/kafka && terraform init && terraform apply
cd ../emr && terraform init && terraform apply
cd ../redis && terraform init && terraform apply
cd ../eks && terraform init && terraform apply
cd ../sagemaker && terraform init && terraform apply
# 2. Bootstrap Feast feature registry
make bootstrap-feast
# 3. Run PySpark feature pipeline on EMR
make run-emr
# 4. Train models (XGBoost + LSTM) as SageMaker Training Jobs
python src/training/xgboost-trainer.py --config-path configs/production.yaml
python src/training/lstm-trainer.py --config-path configs/production.yaml
python src/training/ensemble.py --config-path configs/production.yaml
# 5. Export to ONNX and quantize to INT8
python src/training/onnx-export.py
python src/serving/quantization.py
# 6. Materialize features from S3 to Redis
python src/features/feast-materialize.py
# 7. Promote validated model to production
./scripts/promote-model.sh --stage production
# 8. Deploy API to EKS
kubectl apply -f infra/k8s/
# 9. Run load test (validates p95 < 150ms at 10K concurrent users)
make load-test
# 10. Verify monitoring
kubectl port-forward svc/grafana 3000:3000 -n monitoring
# Open http://localhost:3000 - Credit Risk Intelligence Platform dashboard| Target | Description |
|---|---|
make test |
Run all 281 unit tests |
make lint |
Ruff + Black check |
make format |
Black + isort auto-format |
make run-local |
Start FastAPI dev server on port 8000 |
make run-emr |
Submit PySpark pipeline to AWS EMR |
make load-test |
Run Locust load test against staging endpoint |
make bootstrap-feast |
Initialize Feast feature registry |
make promote-model |
Run validation gate and promote model to Staging |
All thresholds, timeouts, batch sizes, and latency targets are configuration values in configs/. Nothing is hardcoded in logic files.
configs/base.yaml- shared defaults for all environmentsconfigs/staging.yaml- staging overrides (lower traffic thresholds)configs/production.yaml- production overrides (full capacity, strict gates)
Environment-specific secrets (API keys, DB passwords, Kafka credentials) are read from AWS Secrets Manager at runtime. No secrets are stored in config files or code.
Three GitHub Actions workflows:
.github/workflows/ci.yml- runs on every PR:make lint+make test. Blocks merge if any test fails or coverage drops below 80% onsrc/..github/workflows/cd-staging.yml- runs on merge tomain: deploys to staging EKS, runs smoke test..github/workflows/cd-prod.yml- runs on Git tagv*.*.*: requires manual approval, deploys to production EKS with zero-downtime rolling update.
See docs/runbook.md for:
- p95 latency SLO breach (> 150ms)
- Feature store circuit breaker open
- Drift threshold breach - retraining not triggering
- Model validation gate failure
- Scoring API pod OOMKill
- Zero-downtime model rollout procedure
- Human review queue backlog
Why ONNX INT8 for serving? Reduces inference container size from ~500MB (full PyTorch) to ~10MB (onnxruntime only). INT8 quantization delivers 30% latency reduction with < 0.5% AUC degradation on the holdout set.
Why Feast? Eliminates train-serve skew by giving both training and serving a single feature retrieval interface backed by the same definitions. Reduces skew-related incidents by 40% vs. separate training/serving implementations.
Why LangGraph over a simple LLM chain? Every state transition is explicit, loggable, and reconstructable from audit logs. Financial regulators require that every credit decision be explainable. LangGraph's checkpointing captures all intermediate agent states, not just the final output.
Why circuit breaker on Redis? A slow Redis causes every inference request to wait for a timeout. At 10K concurrent requests, a 5-second Redis stall would exhaust the connection pool and cause a full outage. The circuit breaker contains the blast radius: fall back to cached features, continue scoring at degraded accuracy, flag the response.
MIT