Flow ④'s first stage — condensing multi-turn history into a standalone retrieval query — is deliberately fail-open: on any failure it degrades to the raw question, so a broken condenser can never break chat. The cost of that posture is that a broken (or merely bad) condenser is also invisible: it silently guts retrieval on every follow-up question, and nothing measures it. Every entry in crates/b2-llm/evals/questions.json is single-turn, and groundedness.rs passes an empty history to Vault::ask, so the condense path never executes in the one harness that scores chat.
This is the same shape as the retrieval-reach ceiling the chat eval already reports (a citation can only be as good as what retrieval handed over): on a follow-up, retrieval reach is a function of the condensed query, and today that function's quality is assumed.
Ask
Add labelled multi-turn scripts to questions.json: a first question, then a follow-up that leans on pronouns/ellipsis ("does that also fix the sour taste?", "how long does that take?"), with expect naming the note the follow-up must still cite. Score, per script:
- retrieval reach after condensation vs the same follow-up asked cold (no history) — the paired readout (process rule 1) of what condensation buys or costs;
- the existing citation/refusal metrics on the follow-up's answer.
Keep the run's posture: numbers to read, exit code only for a broken pipeline. Row keys additive beside the existing per-question shape.
Flow ④'s first stage — condensing multi-turn history into a standalone retrieval query — is deliberately fail-open: on any failure it degrades to the raw question, so a broken condenser can never break chat. The cost of that posture is that a broken (or merely bad) condenser is also invisible: it silently guts retrieval on every follow-up question, and nothing measures it. Every entry in
crates/b2-llm/evals/questions.jsonis single-turn, andgroundedness.rspasses an empty history toVault::ask, so the condense path never executes in the one harness that scores chat.This is the same shape as the retrieval-reach ceiling the chat eval already reports (a citation can only be as good as what retrieval handed over): on a follow-up, retrieval reach is a function of the condensed query, and today that function's quality is assumed.
Ask
Add labelled multi-turn scripts to
questions.json: a first question, then a follow-up that leans on pronouns/ellipsis ("does that also fix the sour taste?", "how long does that take?"), withexpectnaming the note the follow-up must still cite. Score, per script:Keep the run's posture: numbers to read, exit code only for a broken pipeline. Row keys additive beside the existing per-question shape.