Distinguished Engineer, AI & ML Systems · UBS London
Building production AI agents, LLM inference infrastructure, evaluation frameworks, and MLOps for Tier-1 financial institutions. Current focus: making AI systems that are explainable, auditable, and actually deployable in regulated environments — not just impressive in demos.
- LLM inference & GPU serving — vLLM, TensorRT-LLM, speculative decoding (γ-sweep acceptance-rate analysis), KV-cache/prefix-caching characterisation, dynamic per-token FP8 quantization, fused CUDA kernels
- Multi-agent systems — three-layer orchestration architecture: intent routing, specialist agents, reconciliation with human-in-the-loop escalation
- LLM evaluation — golden dataset frameworks, model-as-judge scoring, CI/CD regression gates aligned to FCA model risk standards
- LLM training pipelines — pretraining data quality (MinHash+LSH dedup, perplexity filtering), curriculum design, training-stability monitoring
- MLOps & CI/CD for ML — Vertex AI pipelines, shadow deployments, automated rollback, full audit trails — cut model release cycles by 60%
- RAG at scale — hierarchical chunking, query rewriting, cross-encoder re-ranking, uncertainty thresholding for sub-second inference on millions of documents
- Enterprise AI adoption — bridging the gap between what AI can do in a lab and what Risk, Legal and Compliance will actually sign off on
| Repo | Description |
|---|---|
| llm-failure-taxonomy | 6-class system-level LLM production failure taxonomy — 50 labeled incidents, rule-based + Claude API classifiers, failure budget calculator. Companion code for Evaluation Blindness (arXiv 2026) |
| gpu-llm-profiler | NVIDIA GPU inference profiler — speculative decoding acceptance-rate sweep (γ 1–8), KV cache hit rate vs context length (with/without prefix caching), AWQ / GPTQ / FP16 throughput comparison. Built on vLLM. 14 unit tests run GPU-free. |
| llm-pretraining-toolkit | Pretraining data quality, curriculum design and training stability — from-scratch MinHash+LSH near-duplicate detection, perplexity filtering (KenLM/GPT-2 backends), fastText language gating, difficulty-scored curriculum mixing with decay schedules, rolling z-score loss-spike detection with rollback flags, per-layer gradient-norm tracking, HF-compatible dataset cards. 34 tests, NumPy-only core, Apache-2.0. |
| enterprise-agent-framework | Production-grade multi-agent orchestration for regulated environments — router, specialist agents, reconciler, JSONL audit trail |
| llm-evaluation-toolkit | Golden dataset evaluation harness, model-as-judge scoring, CI/CD regression gates for production LLM releases |
| mlops-cicd-templates | GitHub Actions + Vertex AI ML pipelines — shadow deploys, canary rollouts, rollback scripts, BigQuery audit logging |
| rag-production-patterns | Hierarchical chunking, query rewriting, re-ranking, uncertainty thresholding — RAG patterns that survive production |
| financial-doc-intelligence | Document classification and extraction for financial documents with plain-English interpretability outputs |
| gcp-vertex-ai-accelerators | Vertex AI, BigQuery ML and Cloud Run utility scripts — reusable accelerators from real enterprise deployments |
Evaluation Blindness: How Silent Measurement Failures Corrupt AI Systems from Training to Deployment · arXiv preprint, August 2026 · Independent research · Sole author
Most measurement failures in AI systems produce no error signal — the failure propagates silently through training loops, evaluation pipelines, and production monitoring until downstream harm makes it visible. This paper formalises evaluation blindness as the unifying property of this failure mode and shows it operates at two stages the literature has treated separately: training time (reward hacking, importance-sampling bugs, benchmark contamination) and deployment time (six production failure classes, 53% silent across 50 real-world incidents). Introduces a detectability predicate unifying both stages, four concrete training-time case studies including a TRL PR #6594 implementation bug, and a per-use-case failure budget framework tied to risk class. Code and taxonomy: github.com/priyanka25aug/llm-failure-taxonomy
Beyond Hallucination: A System-Level Failure Taxonomy for Production LLMs · Zenodo, 2026 · Sole author
Earlier version of the taxonomy work. Introduces the FC-A/B/C/D failure budget framework with risk-tiered governance, a multi-label classifier (~88% accuracy), and a failure-budget calculator built on 50 annotated real-world production incidents.
| Project | Contribution | Status |
|---|---|---|
| TransformerLens | Jacobian Lens community-format converters — importing anthropics/jacobian-lens fit-checkpoints and neuronpedia/jacobian-lens HF artifacts into the JacobianLens artifact format, with round-trip validation and strict schema checks. Maintainer-invited (Tier 2 of the Jacobian Lens follow-up, mechanistic interpretability tooling). |
issue #1539 · scoped, in progress |
| Anthropic claude-cookbooks | 08_The_compliance_aware_agent.ipynb — 44-cell notebook: FC-A/B/C/D failure budget framework, three-layer multi-agent orchestration, confidence-based escalation, human-in-the-loop for FCA/MiFID II/Basel III regulated environments | PR #783 · open |
| Anthropic claude-cookbooks | evals/model_as_judge/ — model-as-judge evaluation pipeline: golden dataset design, rubric scoring (0.99 judge-human correlation), regression gates, shadow deployment eval, CI entry point | PR #786 · open |
| LangChain / LangGraph | compliance_checkpoint_fca_mifid2.ipynb — compliance-aware HITL checkpoint: 4-node StateGraph, append-only SQLite audit trail (tamper-proof triggers), write-intent-before-execute pattern, uuid5 idempotency key, 5 runnable scenarios — FCA SYSC / MiFID II Art. 25 / SR 11-7 | PR #8422 · awaiting maintainer assignment |
| NVIDIA NeMo Guardrails | financial_services_compliance_guardrails.ipynb — GLiNER PII detection (IBAN, sort code, account number, NI number), MiFID II topic control via NIM, FCA COBS 4 disclaimer enforcement, Consumer Duty / SR 11-7 audit trail — 5 runnable scenarios with assertions | PR #2216 · open |
| HuggingFace evaluate | financial_llm_faithfulness — rule-based metric for regulated financial AI: numerical faithfulness (% / bps / monetary values vs reference), FCA COBS 4 / MiFID II disclaimer detection, composite compliance risk score (0–1). First financial metric in the library. Deterministic — no external APIs. | PR #783 · open |
| OpenHands | Financial document intelligence utility — classifies earnings reports, loan agreements, audit reports and regulatory filings with plain-English risk outputs. No external APIs. | PR #14271 · under review |
| vLLM | benchmarks/benchmark_spec_decode_analysis.py — speculative decoding analysis script: sweeps gamma (1–8), records acceptance rate (α = (speedup−1)/γ), tok/s, p50/p95/p99 latency, GPU memory. First spec decode analysis script in vllm/benchmarks/. SSH-signed. | PR #49825 · open |
| vLLM | silu_and_mul_dynamic_per_token_quant CUDA kernel — fused SiLU+gating + dynamic per-token FP8 quantization kernel. Two-phase: Phase 1 computes per-token absmax via warp_max + shared memory reduction; Phase 2 quantizes to FP8. Fills gap left by existing per-tensor silu_and_mul_quant. Addresses Q3 2026 SIG-Quantization roadmap item #48168. | PR #49828 · open |
| flash-attention | tests/test_flash_attn_gqa_decode.py — 6 test groups for production LLM serving gaps: GQA extreme ratios (8:1/16:1/32:1 — Llama 3 70B patterns), speculative decode verify phase (seqlen_q=2–8), chunked prefill+decode varlen mixed batches, paged KV small block sizes [16,32,128], softcap+GQA decode, long context (8k/16k) GQA backward. SSH-signed. | PR #2730 · open |
