A Benchmark for Qualitative and Quantitative Reasoning in Consulting Cases
BACasebook is a benchmark for step-typed, multi-step business-analytics reasoning: a single case chain interleaves three step types — framework structuring, qualitative judgment, and exact quantitative derivation — the regime a professional analyst works in, and one that neither fully verifiable math suites nor open-ended judge benchmarks capture.
The corpus is 50 verbatim cases (248 questions) from three consulting-interview preparation collections (PK20 / PK21 / PK23) compiled by one experienced ex-practitioner. The collections codify the case-interview standard the MBB firms (McKinsey, Bain, BCG) use to assess candidates — they are not the firms' internal casebooks or club casebooks. Ground truth (keypoint phrasings, numerical answers, framework decompositions) is extracted verbatim from the source PDFs, so source fidelity replaces annotator subjectivity.
Headline finding. Across six frontier LLMs, quantitative business calculations are correct only ~26% of the time even though every required input is available in the case — the deficit is acquiring and integrating the inputs (requesting the on-request items, planning the calculation), not arithmetic.
cases/ 50 case files in JSON, grouped by source casebook
PK20/ 11 cases
PK21/ 24 cases
PK23/ 15 cases
eval/ evaluation harness artifacts
prompts/judge/ 8 judge prompts (interaction gates + scorers)
prompts/analyst/ analyst system prompt + 11 method skill cards
scorer/ deterministic quant scorer + panel orchestration
RUBRIC.md the R1 keypoint and R4 accept-set rubrics
README.md how a case is scored, the info-release gate, what's included
human_audit/
SUMMARY.md blinded human validation of scoring (168 items)
docs/
SCHEMA.md the case-JSON schema, field by field
DATASET_CARD.md intended use, provenance, licensing, maintenance
ANNOTATION_PROTOCOL.md how cases were extracted and validated
tests/
test_dataset.py validation suite (counts, schema, repo-wide no-CJK)
LICENSE non-commercial research license
Each case JSON is self-contained for evaluation: the opening prompt, client context, upfront exhibits, the full on-request information pool (with its release triggers), the typed questions, and the source-derived ground truth (keypoints, accept sets, tolerances) are all in the file. You can run the full evaluation without acquiring any external document. What is not redistributed is the original casebook PDFs: reproduction of source text is limited to the verbatim keypoint extracts inside the ground truth (see LICENSE and the Ethics Statement in the paper). So the benchmark is self-contained to run, and the case content is reconstructible from the JSON, but the underlying source PDFs are referenced, not shipped.
Each case is one JSON file with these top-level sections (full detail in
docs/SCHEMA.md):
| Section | What it holds |
|---|---|
source, case_arch, difficulty, industry, topic |
Case metadata |
opening_prompt |
The verbatim prompt that opens the case |
structure_1_client_basic_context |
Information given upfront |
structure_2_information_upon_request |
Facts released only when the analyst asks (trigger / concept / release-conditions / info) |
structure_3_case_questions |
The 4–5 questions, each typed framework / qualitative / quantitative, with a state machine |
structure_4_ground_truth |
The scoring ground truth: keypoints per pillar, plus quantitative exact-match answers |
scoring_summary |
Per-question keypoint counts and scoring method |
Questions are scored on a comprehensive rubric: framework and qualitative turns use a 3-level keypoint rubric (covered 1.0 / false-partial 0.5 / false-not-covered 0.0); quantitative turns use exact-match with tolerance bands.
Every quantitative question carries a machine-readable final answer for
deterministic re-scoring: under structure_4_ground_truth.<Qid>.quantitative_part,
correct_final_answer is the canonical value, accepted_values the accept-set,
and tolerance_pct / zero_tolerance the matching rule.
import json, glob
cases = [json.load(open(f)) for f in glob.glob("cases/*/*.json")]
print(len(cases), "cases")
c = cases[0]
print(c["source"]["title"] if "title" in c["source"] else c["source"])
for qid, q in c["structure_3_case_questions"].items():
if isinstance(q, dict) and "type" in q:
print(qid, q["type"], "—", q["question_text"][:80])| Cases | 50 |
| Questions | 248 |
| Source casebooks | 3 (PK20, PK21, PK23) |
| Question types | framework, qualitative, quantitative |
| Languages | English |
See docs/DATASET_CARD.md for the full breakdown,
provenance, and intended/out-of-scope use.
python3 tests/test_dataset.py # or: pytest tests/Checks the case count (50), question count (248), casebook split, schema conformance, and that no internal annotation text remains.
Released under a non-commercial research license (see LICENSE).
The source casebook copyrights remain with their respective authors;
BACasebook redistributes verbatim keypoint extracts only — the
full casebook PDFs are not redistributed.
A citation will be provided upon publication.