AI / ML Engineer at HypeOn AI — production LLM systems: multi-stage orchestration, RAG, NL-to-SQL, and the evals that keep them honest. Bangalore, India.

Portfolio · LinkedIn · umarfarook0yt@gmail.com

I build the messy middle of applied AI: agent orchestration, retrieval that returns the right thing, NL-to-SQL with cost guardrails, and the observability that keeps it all running in production. Strong Python (FastAPI), end-to-end ownership on GCP and AWS, and a bias toward systems that survive contact with real users.

I care about evals before scale: if you can't measure an agent's behavior, you can't improve it. Most of my open-source work exists to make LLM behavior measurable.

Agent infrastructure and evals

• mcp-bigquery-evals — MCP server that lets agents explore BigQuery safely: 7 read-only tools, mandatory dry-run cost caps, structured error codes agents can self-correct on, and a Spider/BIRD-style NL-to-SQL eval harness. Published on PyPI.
• trustbench — production-readiness harness for AI customer-support agents: versioned golden sets, eight trust metrics (five LLM-judged, three deterministic), and McNemar-tested per-intent regression detection. The Cohen's kappa function for judge-versus-human calibration is written and tested, but no hand-labelled set exists to run it against yet. 82 offline tests (green CI), zero API calls needed.
• rag-document-qa (in development) — retrieval-augmented document Q&A with citation-grounded answers; vector store, embedder and reranker are pluggable via Protocol. The retriever eval harness computes Recall@K, MRR and nDCG@10 and has not been run against a benchmark yet. 126 tests pass offline with no model download, no network and no API key.

Applied agent product

• Cargo-Concierge — agentic freight-forwarder copilot: free-form quote request in, ranked airline options and a draft response out. Hand-labelled ablation: 14/15 exact-match on the six graded fields over 15 cases, and removing the rules block from the prompt costs 33 points. Next.js + Mastra + Postgres. Live demo.

Applied ML and analysis

• street-view-plate-blurring — YOLOv8n license-plate detector (mAP@0.5 0.782, recall 0.739 on a 1,073-image val split) driving a recall-first Gaussian-blur redaction pipeline.
• youtube-shorts-performance-prediction — a rigorous negative result: no pre-publish feature predicts Shorts engagement above chance, and the one "95% accurate" model is a leakage trap (permutation test p = 0.955).
• ipl-data-analysis — 1,095 IPL matches with chronologically computed venue-history features and match-outcome models that top out at ROC-AUC 0.547, which does not separate from a coin flip on a 273-match test split.

ML from first principles (building the stack a layer down from the APIs)

• Nano-LLM-from-scratch — GPT-2 rebuilt in PyTorch with RoPE, RMSNorm, SwiGLU and a KV-cache, plus a byte-level BPE written from scratch. 257 offline tests. A 2,500-step CPU run is reported with the artifacts behind every number; the 124M config ships untrained and says so.
• Tiny-diffusion — DDPM with classifier-free guidance and a DDIM sampler, written from scratch, with 111 offline tests. The MNIST CPU smoke run is reported with the numbers it actually produced, including a deliberately unflattering smoke FID and a plain statement that the samples are not digits yet.
• Triton-attention-kernels — fused Triton kernels for the transformer hot path. Design and benchmark plan; no kernel code yet.
• DPO-on-my-LLM — SFT to DPO post-training with a position-swap LLM judge. Designed and written up; no code yet.

• Conversational research agent (HypeOn AI) — multi-stage routing (chitchat / factual / research), SSE streaming, session memory, idempotent retries, Pydantic-validated outputs, prompt-injection guardrails, Prometheus metrics.
• NL-to-SQL over BigQuery (HypeOn AI) — schema discovery, synonym matching, dry-run cost caps, multi-provider routing with fallback; built so non-technical operators can query the warehouse.
• Clinical chat assistant (Synclovis Systems) — RAG over clinical PDFs with chunking, metadata filtering, and guardrails against unsupported answers.
• AI inventory platform (freelance) — LLM invoice extraction, demand forecasting, real-time stock alerts for a retail client.

Python (FastAPI, Pydantic, SQLAlchemy) · PyTorch · TypeScript · LangChain · FAISS · sentence-transformers · PostgreSQL · Redis · BigQuery · GCP · AWS · Docker · GitHub Actions · Prometheus

• Tradeoffs over tools — pick by constraint, not hype.
• Evals before scale — a bad eval beats no eval.
• Data quality over model swapping — a new model rarely fixes bad inputs; retrieval and prompt structure compound.
• Ship narrow, then expand — one user, one workflow, working end-to-end.

Open to conversations about production LLM systems, RAG, evals, and ML systems.

umarfarook-ai.vercel.app · LinkedIn · umarfarook0yt@gmail.com