Renamed from finance-llm-evals (August 2026) — old links redirect. The name is the
fund-accounting control this suite ports to AI: a number that does not tie out does not settle.
A runnable suite of evaluations measuring how well large language models perform real asset-management analyst workflows — judged the way a finance professional actually judges work: against a rubric, with every figure traced to a source filing, auto-fail "gates" for the errors that quietly poison a memo, and credit for calibrated uncertainty rather than confident guessing.
Most finance-LLM demos show a polished answer. This shows the scoring system behind the answer: the workflow broken into checkpoints, a gating-plus-weighted rubric, expert-authored gold cases cited to SEC filings, and a grader that surfaces exactly where — and how badly — a model fails.
Three of the six workflows are the analyst's (earnings, buffer-ETF diligence, DCF). Three are the back office's (ETF creation/redemption reconciliation, OTC swap confirmation matching, and corporate-actions processing — the suite's first document-store episode, where the model must decide which document governs before any number is right) — and as of mid-2026, those three appear to be the only public LLM evals of capital-markets post-trade operations anywhere (see where this sits in the 2026 benchmark landscape). Every live model run is consolidated in LEADERBOARD.md.
pip install -r requirements.txt # one dependency: pyyaml
python -m harness demo # grade a correct answer and a subtly-broken one, side by sideThe demo runs a model that does Snowflake's quarter correctly but misreads one statement header ("in thousands" as "in millions"). Watch what the scoring does:
CASE snow-2026q2 model=oracle gated 1.000 AllPass 1 gates: none
CASE snow-2026q2 model=scale_slip gated 0.452 AllPass 0 gates: GATE.P2 (ungated 0.951)
The arithmetic is internally consistent, so the naive (ungated) score stays ~0.95. But one hard gate fires on the scale misread and the gated score collapses to 0.45. That ~0.50 gap is the finding: can do the math, cannot be trusted to read a statement header. Measuring that gap — across six analyst and back-office workflows — is the whole repo.
Every workflow is broken into checkpoints (plan → extract → calculate → decide). Each checkpoint has point-weighted, tiered criteria, and a few are gates — auto-fail conditions for the errors that quietly poison a memo (a unit/scale slip, a wrong fiscal period, a fabricated number, settling a basket that doesn't reconcile). A normal mistake costs a few points; a gate collapses the score and flags exactly where and how badly the model failed. Every gold figure is traced to a real SEC filing; nothing is invented. Models also earn credit for calibrated uncertainty — saying "not determinable from this packet" instead of guessing.
| Eval | What it tests | Run it (the perfect "oracle") | See it fail |
|---|---|---|---|
| #1 — Earnings | digest a 10-Q, reconcile the figures, flag what moved | python -m harness run --case snow-fy2026q2 |
--model scale_slip — thousands-vs-millions → GATE.P2 |
| #2 — Buffer ETF | recompute a buffer ETF's cap & buffer from the option strikes; price the protection | python -m harness run --case koct-op2026-anchor |
--model free_lunch — protection asserted with no cost → GATE.FREELUNCH |
| #3 — DCF | project FCFF, discount at WACC, bridge EV→equity, per share | python -m harness run --case mcd-fy2025-dcf |
--model bridge_omit — EV÷shares, net-debt bridge skipped → GATE.BRIDGE |
| #4 — Creation/redemption | reconcile an AP's creation basket vs the PCF & NAV; settle only if it ties | python -m harness run --case grin-create-2026 |
--model approve_break — settle a basket that's $13,320 short → GATE.RECON |
| #5 — Confirmation matching | match two OTC swap confirmations field-by-field; affirm only if the economics tie | python -m harness run --case irs-confirm-2026 |
--model affirm_match — affirm a trade with a material rate break → GATE.MATCH |
| #6 — Corporate actions | process an event from a document store (announcement + amendment + distractors): pin the governing version and dates, compute the entitlement, commit only what the deadline allows | python -m harness run --case bry-dividend-2024 |
--model version_slip — terms sourced from the superseded announcement → GATE.VERSION |
Every run defaults to a perfect answer (scores 1.000); add --model <name> to watch a designed
flaw trip its gate. python -m harness list shows every case and variant ·
python -m harness suite scores them all · python -m harness selftest prints PASSED.
Run a real model (optional, needs an API key): add --model live --endpoint <url> --model-id <model>. Keys come from the environment or a gitignored repo-root .env (copy
.env.example); the harness picks the right vendor's key from the endpoint
(ANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY). The graded frontier-model runs are
in outputs/, consolidated in LEADERBOARD.md.
📄 Prefer prose? The consolidated technical report is at evals.finance/paper.pdf and on SSRN (abstract 7243025), DOI 10.2139/ssrn.7243025; source in
paper/. Each eval also has a plain-language write-up incontent/.
- Eval #1 — Quarterly earnings analysis. Digest a 10-Q/earnings release, reconcile the figures, benchmark versus consensus, flag what moved. 17 checkpoints, 109 criteria, three gold cases (BlackRock, Microsoft, Snowflake).
- Eval #2 — Defined-outcome ("buffer") ETF diligence (the one a generalist can't author). Given a prospectus, the fund's actual FLEX-option legs from its N-PORT filing, and a dated market snapshot: recompute the marketed cap and buffer from the option strikes, compute what a mid-period buyer actually gets, verify the marketing claims, and price the protection — with the free-lunch gate (downside protection asserted with no forgone-upside cost = auto-fail) as the signature. 18 checkpoints, 110 criteria, three market-snapshot cases on a real buffer ETF.
- Eval #3 — Discounted-cash-flow valuation (the most-used model in research, and the one most
often quietly wrong). Project unlevered free cash flow, discount at WACC, capitalize a terminal
value, bridge enterprise value to equity, divide by shares — every number a closed-form
consequence of a handful of inputs, so it is perfectly recomputable. The signature is the DCF
that looks right and is wrong: on a real McDonald's FY2025 case the correct fair value under
the case's stated assumptions is
~$228/share (≈20% below the case-date market price) while the classic EV÷shares blunder (skipping the
net-debt bridge) lands at ~$279 — only −2.6% from price, so the wrong method looks fair. The
consistency spine is the basis gate (unlevered cash flow must meet WACC must meet the
net-debt bridge), and the calibration signature is the false-precision gate (a decimal-precise
target on a model that is 80% terminal value, where a 50bp discount-rate move shifts it +15/−12%,
auto-fails). 18 checkpoints, 107 criteria, two real-10-K gold cases that mirror each other:
McDonald's FY2025 (heavy net debt — skipping the bridge overstates by ~22% and looks fair) and
NVIDIA FY2026 (net cash — the same blunder understates by only ~3% and is nearly invisible; the
gate catches the method, not the magnitude). Both run live against eight frontier models
across three vendors — traces in
outputs/eval3-live/. - Eval #4 — ETF creation/redemption basket reconciliation (the custodian back-office core, authored from having run the change-management side of an institutional ETF servicing platform). Given an Authorized Participant's tendered creation basket, the published PCF, and the NAV-based creation value, reconcile it line-by-line, value the basket and cash-in-lieu, compute the tie-out — and settle only if it ties. The signature is GATE.RECON: a model that returns SETTLE for a basket whose residual is out of tolerance auto-fails — the fund-accounting "tie out or stop" control, ported to an AI. The gold case is a creation that does not reconcile (a halted name's cash-in-lieu delivered at a stale prior-close price, short $13,320 on a $3.075M order — every in-kind line matches, only the cash plug is short); the gold answer is DO_NOT_SETTLE, localized to that line. A clean-settle counterweight case catches the over-cautious mirror (crying break on a basket that ties). 8 checkpoints, 32 criteria (+5 gates), two gold cases. (PCFs are NSCC-disseminated, not public filings, so this case is a constructed, mechanics-faithful scenario — real constituent securities and representative prices; fund, order, and break illustrative.)
- Eval #5 — OTC derivative confirmation matching (the derivatives sibling of #4 — and grounded in
a real public message). Two counterparties each book their side of an interest-rate swap and send
confirmations; the affirmation desk matches the economic terms field-by-field and affirms only if
they tie. The signature is GATE.MATCH: a model that affirms ("matched") a trade whose economic
terms do not tie auto-fails — the same "tie out or stop" control as reconciliation. The gold "our
side" is the real, publicly-downloadable FpML 5.10 sample confirmation (
ird-ex01-vanilla-swap.xmlfrom fpml.org), so unlike #4 the gold is cited, not constructed. The counterparty confirmation carries a material break — a fixed rate of 6.05% where ours says 6.00% (~5 bp, ~EUR 25,000/yr on EUR 50MM); every other term ties, and the two parties' trade ids differ by design (the materiality foil). Gold answer: MISMATCHED, do not affirm, localized to the fixed rate. A clean-match counterweight catches the over-cautious mirror. And because funds are among the biggest users of swaps (leveraged ETFs are built on total-return swaps, fixed-income ETFs hold interest-rate swaps), a second case pair runs the same control on a swap held inside an ETF — the fund as one side. 8 checkpoints, 28 criteria (+5 gates), four gold cases (a bank swap and an ETF's swap, each with a break and a clean match). - Eval #6 — Corporate-actions processing (the suite's first document-store episode, and the
temporal control the earlier evals don't test: some actions are irreversible once a deadline
passes). The model receives a document store — the governing announcement and any
amendment, a position report, an ETF basket file where relevant, plus distractors — and must
produce a structured action plan: pin the event and its governing version, pin the governing
dates, extract terms, compute the entitlement, judge the election state, and commit to actions.
The signature is GATE.ELECT (an irreversible wrong commitment: an election when none is
available, a commitment past the deadline, a release on superseded terms or of an amount no gold
figure permits) and GATE.VERSION is the document-store vintage slip (terms sourced from a
superseded version — or from a distractor: another event's ratio or proration factor carried
into this event's worksheet). Three real-anchored case families, each with a break and a clean
counterweight: NVIDIA's 2024 10-for-1 split hitting a constructed ETF basket whose PCF is
stale (adjust ×10 — and the clean twin catches the double-adjuster); Monster Beverage's 2024
oversubscribed self-tender (the depositary's stated 47.18% proration factor governs — the
naive 56.6M/119.0M ≈ 47.56% recompute is wrong because odd lots skip proration; the odd-lot twin
catches blind proration of an exempt holder); and a real corrected-dividend pair (Berry 2024,
record date moved 11 days with a position change in between — $8,500 on stale terms vs the
correct $6,800 — with Zoetis 2014's real one-day Sunday fix as the economically-neutral twin
that catches the perma-escalator). Gates are designed as verifiable predicates over the
episode's terminal state, so the same build doubles as an RL environment (a naive rubric
average is unconstrained optimization; gates define the feasible set). 9 checkpoints, 31
criteria (+5 gates), six gold cases — all corporate-action facts cited to EDGAR filings.
Live traces:
outputs/eval6-live/.
| Path | Contents |
|---|---|
workflow/ |
Each workflow decomposed into measurable checkpoints (earnings: 17 · defined-outcome: 18 · DCF: 18 · creation/redemption: 8 · confirmation-matching: 8 · corporate-actions: 9) |
rubric/ |
Gating + weighted, tiered rubrics — machine-readable atoms (criteria*.yaml), the frozen judge prompt (judge.md), and a validate.py linter |
cases/ |
Gold cases — every figure cited to a real SEC filing (10-K / 10-Q / 8-K / 497K / N-PORT); no invented numbers (the creation/redemption case is the one exception: PCFs are not public, so it is a constructed, mechanics-faithful scenario over real securities) |
harness/ |
The runnable scorer: one suite-agnostic engine + a module per eval; deterministic checks + gating + a pluggable LLM-judge interface; a live path for real models |
LEADERBOARD.md |
Every live model run on one page — frontier (evals #3–#5) and open-weight (#1–#2), with the discriminating findings and the honest caveats |
profiles/ |
The leaderboard as data — machine-readable per-checkpoint capability profiles per model (routing priors: which workflow step a model can touch), regenerated byte-stably from the committed runs via python -m harness profiles |
ODD.md |
The allocator framing — what these outputs look like as operational-due-diligence evidence for AI in the investment process, with a sample artifact-backed DDQ section |
outputs/ |
The real graded model runs + failure taxonomies — eval2-live/ (two local models, the judge-vs-expert calibration), eval3-live/ (eight frontier models on both DCF cases, incl. nvda-fy2026-dcf/), eval4-live/ and eval5-live/ (frontier runs on reconciliation and confirmation matching), eval6-live/ (eight frontier models on all six corporate-actions cases) — the full eight-model grid is in LEADERBOARD.md |
Two open-weight models (a 27B reasoning model and a 72B non-reasoning one), run locally at zero API
cost, across a real Innovator buffer ETF under three market snapshots. The headline pattern: both
nailed extraction and the single headline calculation (the remaining cap for today's buyer),
and both fell into the same payoff-reconstruction trap — suggestive that the eval is catching
the task's difficulty, not one model's quirk (both subjects share a vendor lineage, so a
cross-family replication is the honest next step). The smaller reasoning model beat the larger
non-reasoning one on every case. And because the eval is calculation-heavy by design, swapping
the offline grader for a real LLM judge moved the scores by only 2–4.5 points (versus ~15 on the
earnings eval) — the headline barely depends on the subjective part. Full traces, scored reports,
and the taxonomy are in outputs/eval2-live/; the calibration (judge vs. a
hand-graded expert sample, with caveats stated) is alongside.
Running real models also surfaced three calibration bugs in the grader itself — the kind a synthetic self-test can't see because it's written around the grader's own assumptions. They were fixed and everything re-graded; the log is in the taxonomy.
Three frontier models (the first, Claude-only round; the full eight-model grid is in LEADERBOARD.md) (Claude Opus 4.8 / Sonnet 4.6 / Haiku 4.5, via the Anthropic OpenAI-compatible
endpoint) on the real McDonald's FY2025 case, deterministic core + offline judge:
| Model | Gated | Gate fired | Math-spine score | WACC probe |
|---|---|---|---|---|
| Opus 4.8 | 0.965 | none | 0.978 | refused |
| Sonnet 4.6 | 0.955 | none | 0.978 | refused |
| Haiku 4.5 | 0.692 | GATE.C1FCF | 0.646 | refused |
The two strong models do textbook DCF correctly — no gate fires, the whole calculation spine is clean,
fair value ~$228 (gold $227.82). The weak model's reported free cash flow doesn't equal its own
build (a +$2B/yr offset); the FCF-definition gate flags exactly that at C1, and the error cascades
into a wrong $138 valuation — one slip, surfaced and localized. The most consistent behavior across all
three is a subtle one (and the opposite of a capability gap): they compute the discount rate
correctly inside the model (~7.15%) and discount with it, yet, asked "what is the WACC per the
10-K?", answer "not disclosed" with a null value — declining to claim the derived figure as the
answer, though most restate it in their derivation text — a framing quirk the eval isolates. And — as on both prior evals — the first real models surfaced five
grader bugs the synthetic tests were written around; all fixed, all re-graded, the oracle still
1.000/AllPass. Full matrix, traces, and the calibration log:
outputs/eval3-live/.
On the NVIDIA mirror case (net cash; all eight frontier models): five models — including a
flash-tier one — land within 0.004 of each other at the case-gold ~$91.7 fair value with the
correct "the price embeds growth beyond this base case" read; the one GATE.BRIDGE fire
(GPT-5.4-mini) is the mirror trap in its subtler form — it added the net cash and dropped the
non-op assets it had itself extracted; and two models from two vendors showed a new failure
class on the WACC probe: derive the right number, report a different one (a derivation that
builds 12.62 above an answer that says 12.51 or 12.31). The first live batch also surfaced three
more grader-contract gaps (fraction-vs-pp rates, comma-formatted numeric strings, snake_case
counted as one word) — all fixed, all re-graded, MCD reports unchanged. Details:
outputs/eval3-live/nvda-fy2026-dcf/TAXONOMY.md.
A perfect ("oracle") answer scores 1.000/AllPass, and eleven deliberately-flawed variants each trip exactly one gate, with a blast radius that matches the error's severity. The pattern is the finding — the eval tells a catastrophic foundational error apart from a localized one:
| Flawed answer | Gate | Gated score | What it shows |
|---|---|---|---|
basis_mix — unlevered FCF discounted at the cost of equity |
GATE.BASIS (hard) | 0.35 | a wrong basis commitment poisons the whole valuation |
basis_late — the same error executed (P1 looks clean) |
GATE.BASIS (C5 hook) | 0.38 | caught by back-solving the discount rate from the model's own PVs |
scale_slip — projections mislabeled thousands-vs-millions |
GATE.SCALE (hard) | 0.73 | ungated stays ~0.99 (the math is consistent); the gate is the story |
bridge_omit — EV÷shares, net-debt bridge skipped |
GATE.BRIDGE (scoped) | 0.85 | the signature localized red — looks right, is wrong |
false_precision — decimal target, no sensitivity block |
GATE.FALSEPRECISION | 0.88 + flag | the calibration signature (80% terminal value, ±15% on 50bp) |
g_explode (0.94) · c7_sign (0.93) · c1_fcf (0.81) |
in-checkpoint | — | each zeroes only its checkpoint; c1_fcf also drags the ungated score where the bad FCF flows downstream |
Reproduce any row: python -m harness run --case mcd-fy2025-dcf --model <name>. The full design and
the McDonald's FY2025 gold case are in workflow/dcf-analysis.md and
cases/mcd-fy2025-dcf.case.yaml.
The signature here is the one every fund-accounting desk runs on: a basket that does not reconcile does not settle. The gold case is a creation that is short $13,320 — every in-kind share line ties, only the cash-in-lieu plug is stale — so the most dangerous way to get the settle/stop decision wrong is also the highest-scoring:
| Flawed answer | Gate | Gated score | What it shows |
|---|---|---|---|
approve_break — all numbers right, SETTLES the break |
GATE.RECON + flag | 0.86 | the signature: the highest-scoring decision failure is the catastrophic one — the control switched off |
scale_slip — delivered cash read in thousands |
GATE.SCALE (hard) | 0.66 | a mis-scaled tie-out |
cil_blind — misses the cash-in-lieu substitution |
GATE.CIL (scoped) | 0.55 | right stop, wrong root cause; RECON does not fire |
direction_flip — creation read as redemption |
GATE.DIRECTION (hard) | 0.36 | the whole order on the wrong footing — the biggest cascade |
fabricate_price — invents the halted name's close |
GATE.FABRICATION | 0.92 | the calibrated-refusal probe → G = 0 |
A second clean-settle gold case (the same order delivered correctly, so it ties) catches the
over-cautious mirror — a model that cries break on a basket that reconciles. The suite was hardened
by an adversarial gaming review (an approval synonym still trips GATE.RECON; a fabricated price
under a refusal label still trips GATE.FABRICATION). Design + gold:
workflow/creation-redemption-analysis.md,
cases/grin-create-2026.case.yaml.
Three frontier models (the first, Claude-only round; the full eight-model grid is in LEADERBOARD.md) (Claude Opus 4.8 / Sonnet 4.6 / Haiku 4.5) on both gold cases:
| Model | Break case | Gate | Clean-settle case |
|---|---|---|---|
| Opus 4.8 | 0.983 | none | 0.983 |
| Sonnet 4.6 | 0.943 | none | 0.983 |
| Haiku 4.5 | 0.496 | GATE.SCALE | 0.983 |
The two strong models reconcile to the dollar — in-kind MV $2,838,400, the cash-in-lieu valued at the
struck $112.40 (not the AP's stale $105.00), residual exactly −$13,320, DO_NOT_SETTLE
localized to the RBLX line, the refusal probe answered correctly. Haiku catches that something is
wrong but its own arithmetic is off: it overstates the in-kind value by exactly $200,000, which
flips the residual to +$186,680 — so it concludes the basket is over-delivered when it is
actually short. It still refuses to settle (the right call), but the eval pins the error to the
valuation (GATE.SCALE) and the wrong answerable-twin. On the clean case all three correctly SETTLE
(no false break). The honest negative result: GATE.RECON — settle a basket that does not
reconcile — never fired; no frontier model approved the break. The capability gap showed up as
Haiku's $200k arithmetic slip, localized to one checkpoint, not as the marquee failure. (This section describes the first Claude-only round; the cross-vendor grid in LEADERBOARD.md has since confirmed the negative across all eight models.) Single model
family, n=1 per case — a cross-family run is the honest next step. Full matrix + traces:
outputs/eval4-live/.
The derivatives sibling of #4 — and the gold "our side" is a real, publicly-downloadable FpML message, so the case is cited, not constructed. The break: a counterparty confirmation that ties on every term except a 6.05% vs 6.00% fixed rate. The signature is again the highest-scoring decision failure:
| Flawed answer | Gate | Gated score | What it shows |
|---|---|---|---|
affirm_match — all terms compared right, affirms the broken trade |
GATE.MATCH + flag | 0.84 | the signature: the highest-scoring decision failure is the catastrophic one — the control switched off |
scale_slip — notional read in thousands |
GATE.SCALE (hard) | 0.64 | a mis-scaled comparison |
materiality_blind — flags the expected trade-id diff as a break |
GATE.MATERIALITY (scoped) | 0.61 | right verdict, wrong reason; MATCH does not fire |
direction_flip — fixed payer/receiver inverted |
GATE.DIRECTION (hard) | 0.47 | the trade read backwards — the biggest cascade |
fabricate_probe — invents a mark-to-market |
GATE.FABRICATION | 0.92 | the calibrated-refusal probe → G = 0 |
A clean-match counterweight case catches the over-cautious mirror (crying "mismatch" on a trade that
ties). The suite was hardened by an adversarial gaming review — a settlement-desk go-ahead synonym
("release for settlement", "book it") still trips GATE.MATCH; a fabricated mark-to-market asserted in
prose still trips GATE.FABRICATION. Design + gold:
workflow/confirmation-matching-analysis.md,
cases/irs-confirm-2026.case.yaml.
Three frontier models (the first, Claude-only round; the full eight-model grid is in LEADERBOARD.md) (Claude Opus 4.8 / Sonnet 4.6 / Haiku 4.5) on both gold cases:
| Model | Break case | Basis-point read | Clean case |
|---|---|---|---|
| Opus 4.8 | 0.980 | ✅ 5 bp → ~EUR 25k/yr | 0.980 |
| Sonnet 4.6 | 0.933 | ❌ "0.5 bp" → EUR 2,500 (10× low) | 0.980 |
| Haiku 4.5 | 0.933 | ❌ "50 bp" → EUR 2,500,000 (10× high) | 0.980 |
All three match the two confirmations correctly — flag the fixed-rate difference as the break, treat
the differing trade ids as expected, return MISMATCHED, and affirm the clean case. The honest
negative: GATE.MATCH never fired — no model affirmed a broken trade. What separated them was a
classic finance trap, the 0.05% → basis-point conversion: only Opus sized the break correctly
(~EUR 25k/yr); Sonnet called it "0.5 bp" and Haiku "50 bp" — wrong by 10×, in opposite directions —
and the eval localizes it to one checkpoint (C3.impact). As on every prior eval, the first real
models also surfaced two grader-calibration bugs (richer dict/prose answer shapes the synthetic
tests didn't anticipate); both fixed, oracle still 1.000/AllPass. Full matrix + traces:
outputs/eval5-live/.
Eight frontier models × six cases (48 runs, three vendors). The four findings, stated with the caveats they need:
- The suite's first perfect live row. Claude Sonnet 4.6 scored 1.000/AllPass on all six cases — no live model had AllPassed a single case on any prior eval. Six of eight models sit ≥ 0.92 everywhere (32 of 48 runs AllPass): the frontier handles these procedural document-store episodes at a far higher ceiling than the judgment-heavy analyst evals.
- The marquee cascade finally happened — on the small tier. Through evals #3–#5 the
catastrophic decision gates never fired on any live model. On the corrected-dividend case,
GPT-5.4-mini pinned the correct correction 8-K, stated the corrected record date — and then
computed the entitlement on the superseded 50,000-share position and booked $8,500 (gold:
$6,800 on the corrected 40,000). A release on superseded terms:
GATE.VERSION+GATE.ELECT, gated 0.225 against an ungated 0.574. Flagships still never commit the catastrophic action; the small tier now does. - "Derive the right number, report a different one" replicates in a third domain. Haiku's refusal-twin derivation computes "$1,800.00" and its value field says 18,000; GPT-5.4-mini's computes 1,800 and reports 180,000, labeled COMPUTED, on a net-cash question the store does not determine (its fabrication gate). And a planted trap fired exactly where designed: on the clean split case, Opus and Haiku both answered the dividend twin at the pre-split rate × post-split shares — the classic split/dividend double-count.
- Nobody was fooled by the distractors: no model used the fictional QSEM split's ratio or
Incyte's real 93.5% proration factor, none recomputed the naive 47.56% in place of the
depositary's stated 47.18%, and every model honored the odd-lot priority. Findings, grader log,
and scope notes:
outputs/eval6-live/TAXONOMY.md.
python -m harness demo grades a model that does Snowflake's analysis correctly but misreads
the statement header ("in thousands" as "in millions"). Its ungated score stays ~0.99 (the
arithmetic is internally consistent), but a hard gate collapses the gated score to ~0.45 — a
0.53 gap that is the finding itself: can do the math, cannot be trusted to read a statement
header.
Before any firm lets an AI do analyst work, it needs to know whether — and exactly where — to
trust it. A blended accuracy number can't answer that. This suite catches the errors that quietly
poison a memo (scale, period, fabrication, GAAP-vs-non-GAAP for earnings; wrong fund vintage,
strike-scale, stated-vs-remaining terms, and the free lunch for buffer ETFs; the levered/unlevered
basis mix, the missing net-debt bridge, and false precision for a DCF; the create/redeem direction,
a stale cash-in-lieu, and settling a basket that does not reconcile for fund servicing; a trade
direction, day-count, or rate break, and affirming a confirmation that does not tie for
derivatives ops; a superseded announcement version, a missed election deadline, and an
irreversible commitment on the wrong terms for corporate actions), localizes each to the
checkpoint that owns it, and tells "looks right" apart from
"is right." A firm uses it as an
acceptance test (which model is deployable, and where it needs a guardrail) and a regression
test (did a model/prompt change help or hurt, and where). The per-model results ship as
machine-readable capability profiles — per-checkpoint scores usable as routing
priors — and ODD.md maps the same outputs to the questions an allocator's operational
due diligence would ask about AI inside an investment process.
The methodology here is the same family the frontier now uses — expert-authored tasks decomposed into checkpoints, point-weighted rubric criteria with gating conditions, LLM-judge grading, all-pass alongside partial credit. What's different is the domain. Every prominent finance benchmark tests the front office; none covers post-trade operations:
| Benchmark | Built by | Format | Finance slice covered | Post-trade ops? |
|---|---|---|---|---|
| GDPval (2025) | OpenAI | static real-work tasks | analyst / advisor / sales occupations | ✗ |
| Finance Agent v2 (2026) | Vals AI | agentic filings QA | entry-level analyst research | ✗ |
| APEX / APEX-Agents (2025–26) | Mercor | expert-rubric agentic worlds | IB associate work (models, pitch materials) | ✗ |
| BigFinanceBench (2026) | Rogo + OpenAI | point-weighted-rubric QA | public-equity research (52 expert authors) | ✗ |
| FrontierFinance (2026) | Kensho / S&P / MIT | long-horizon computer use | 3-statement / LBO / DCF / M&A model building | ✗ |
| FinBalance (2026) | academic | static reconciliation | corporate bookkeeping (invoices → journals) | ✗ (accounting, not securities) |
| this suite (2025–26) | one domain expert | rubric-gated, runnable, live-graded | analyst workflows + ETF creation/redemption + OTC confirmation matching | ✓ |
A July 2026 meta-survey mapping 452 public financial-services benchmarks onto banking-industry domains (arXiv 2607.01740) reports the same picture: coverage concentrates in information-processing and analysis, with "a genuine gap in the public evaluation landscape for regulated domain tasks." To our knowledge, evals #4 and #5 are the first public, runnable LLM evals of capital-markets post-trade workflows — the asset-servicing back office that clears, settles, and reconciles what the front office trades.
The design lineage, for the record: OpenAI's HealthBench (expert rubric criteria graded by an LLM judge), FinanceBench (every answer tied to an evidence string), FinQA / TAT-QA (executed numeric tolerance), the Vals AI Finance Agent Benchmark (checkpoint scoring of an end-to-end analyst task), and FailSafeQA (rewarding calibrated refusal) — composed into a runnable whole, then pointed at the workflows none of them cover.
See PLAN.md for the phase roadmap and CLAUDE.md for full project context.
MIT — see LICENSE. Gold-case figures are public-record facts from SEC EDGAR,
attributed to their source filings.