I run LLMs in production and try to make them cheap and reliable.
Right now that means owning an inference stack — vLLM on Modal behind an OpenAI-compatible gateway — and the agent platform on top of it, which walks people through filing their income tax returns. Regulated domain, real money, so most of my work is the unglamorous half: evaluation harnesses, conversation replay, guardrails over model non-determinism, and finding out why the GPU bill is what it is.
Before the LLM work I spent a few years on a Rails monolith large enough to be interesting — 200-odd models, a Postgres cluster past a terabyte, and the index audits and query rewrites that come with that. I still reach for Go when something needs to be fast and Rust when I want the compiler to argue with me.
Writing — logs.beer
- Your GPU bill is an autoscaler problem — six GPUs averaging 0.2 requests in flight, a batch-tapered speculative decoding schedule, and the experiment that made things worse.
Working with — Go · Python · Ruby · TypeScript · Rust · C · vLLM · Modal · PostgreSQL · ClickHouse · Redis · Rails · FastAPI · Fiber
Interested in — inference economics, agent evaluation, anything where the measurement turns out to be the hard part.