fix: compare benchmark configuration under declared equivalences (#469)

[godofecht]

Closes #469

The pipeline, since the issue guessed at it

There are no CONFIG| records. Neither runner emits one: bench_sklearn_v2.py
prints RESULT|, DETAIL|, TIMING_UNIT|, BENCHMARK_MODE|,
FIXTURE_SOURCE| and BENCHMARK_ENV|, and flow_results_v2.txt contains only
those prefixes. Configuration is declared, not measured:

benchmarks/parity_contract.json   rows[].flow  and  rows[].sklearn
        |
        v
benchmarks/generate_disparity_report.py   config_diff(flow, sklearn)
        |
        v
benchmarks/disparity_report.json   rows[].configuration_differences
        |                                  |
        |                                  +--> disparity.yml assertions
        |                                  +--> check_disparity_regression.py
        |                                       (configuration_difference_count,
        |                                        one-sided: fails on an increase)
        +--> publish_headline_v2.py --check -> docs/disparity-report.json
                                            -> docs/disparity-history.html

config_diff was a plain key-by-key dict comparison, so anything the two
projects spelled differently read as a difference. Nothing else in the repo
constructs or consumes a configuration record, so the fix is contained to the
comparator. bench_flow_v2.flow is untouched; no RESULT| or DETAIL| record
can move, and compiler bug #469 does not apply to this change.

What changed

New benchmarks/config_equivalence.py holds the equivalences the comparator is
allowed to apply. Three rules, in order, and each applies only to a parameter
present on exactly one side (or to a mapped pair each of whose names appears on
exactly its own side):

1. Cross-vocabulary mappings, verified numerically. sklearn's C and Flow's
   l2 name the same setting under alpha = 1 / (C * n_train), the relation
   fix: match sklearn's C=1.0 in the canonical LogisticRegression benchmark (#408) #430 derived when closing LogisticRegression coefficient norm is 2x sklearn's at identical accuracy, suggesting the penalty is not being applied equivalently #408. They are equivalent only when the recorded
   numbers satisfy it to 1e-5 relative. n_train is read from
   split_indices.json (iris 120, digits 1437, diabetes 353), not hardcoded. If
   the split fixture is missing or the row's dataset is not in it, n_train is
   unknown, the conversion cannot be verified and the pair stays reported.
2. Absent equals explicitly disabled. penalty: none on Flow's
   LinearRegression against a sklearn OLS with no penalty parameter. False is
   deliberately excluded: a boolean set to false still implies the knob exists.
3. Solver-private parameters. A per-parameter table, each entry commented
   with why. learning_rate (a step size only exists for a first-order
   iterative solver; the sklearn counterparts here are line-searched lbfgs, a
   direct factorization and coordinate descent) and dual (sklearn's
   primal/dual liblinear switch; Flow implements one formulation). The rule is
   gated on solver identity where the record carries it: if the counterpart
   declares an optimizer/solver that does own the knob and still omits the
   value, the omission stays reported.

A parameter both sides record is always compared and any difference reported.
That is what keeps max_iter and optimizer alive.

Applied equivalences are not dropped. Each row gains
configuration_equivalences carrying the rule, the values and the reason, so
the report reclassifies the entry rather than erasing it. Report
schema_version 3 -> 4.

max_iter is deliberately not in the solver-private table

Ridge/diabetes keeps a one-sided max_iter 1000 vs <missing> after this change.
scikit-learn's Ridge does take a max_iter; the contract simply does not
declare one, and an iteration budget is exactly the class of setting #469 asks
to keep visible. Flow solving Ridge in 1000 gradient steps where sklearn uses a
direct factorization is a real difference and should stay on the report.

Before / after, all 9 rows that carried a configuration dimension

Generated by running the origin/main copy of generate_disparity_report.py
and the new one against the same committed inputs.

Row	before	after	surviving
LogisticRegression / iris	5	2	max_iter, optimizer
LogisticRegression / digits	5	2	max_iter, optimizer
KernelSVC_RBF / iris	1	1	max_iter
PCA / iris	1	1	solver
Ridge / diabetes	2	1	max_iter
LinearSVC / iris	1	0	none
LinearSVC / digits	1	0	none
Lasso / diabetes	1	0	none
LinearRegression / diabetes	1	0	none

Full entries:

LogisticRegression / iris / accuracy   (5 -> 2)
    before:
        C              flow='<missing>'              sklearn=1.0
        l2             flow=0.008333333              sklearn='<missing>'
        learning_rate  flow=0.1                      sklearn='<missing>'
        max_iter       flow=200                      sklearn=1000
        optimizer      flow='lbfgs_no_line_search'   sklearn='lbfgs'
    after:
        max_iter       flow=200                      sklearn=1000
        optimizer      flow='lbfgs_no_line_search'   sklearn='lbfgs'
    equivalences applied:
        l2             cross-vocabulary mapping
        learning_rate  solver-private parameter

LinearSVC / iris / accuracy   (1 -> 0)
    before:
        dual           flow='<missing>'              sklearn='auto'
    after:
        (none)
    equivalences applied:
        dual           solver-private parameter

KernelSVC_RBF / iris / accuracy   (1 -> 1)
    before:
        max_iter       flow=200                      sklearn=1000
    after:
        max_iter       flow=200                      sklearn=1000

PCA / iris / explained_var_ratio   (1 -> 1)
    before:
        solver         flow='power_iteration'        sklearn='auto'
    after:
        solver         flow='power_iteration'        sklearn='auto'

LogisticRegression / digits / accuracy   (5 -> 2)
    before:
        C              flow='<missing>'              sklearn=1.0
        l2             flow=0.000695894              sklearn='<missing>'
        learning_rate  flow=0.1                      sklearn='<missing>'
        max_iter       flow=200                      sklearn=1000
        optimizer      flow='lbfgs_no_line_search'   sklearn='lbfgs'
    after:
        max_iter       flow=200                      sklearn=1000
        optimizer      flow='lbfgs_no_line_search'   sklearn='lbfgs'
    equivalences applied:
        l2             cross-vocabulary mapping
        learning_rate  solver-private parameter

LinearSVC / digits / accuracy   (1 -> 0)
    before:
        dual           flow='<missing>'              sklearn='auto'
    after:
        (none)
    equivalences applied:
        dual           solver-private parameter

Ridge / diabetes / r2   (2 -> 1)
    before:
        learning_rate  flow=0.01                     sklearn='<missing>'
        max_iter       flow=1000                     sklearn='<missing>'
    after:
        max_iter       flow=1000                     sklearn='<missing>'
    equivalences applied:
        learning_rate  solver-private parameter

Lasso / diabetes / r2   (1 -> 0)
    before:
        learning_rate  flow=0.01                     sklearn='<missing>'
    after:
        (none)
    equivalences applied:
        learning_rate  solver-private parameter

LinearRegression / diabetes / r2   (1 -> 0)
    before:
        penalty        flow='none'                   sklearn='<missing>'
    after:
        (none)
    equivalences applied:
        penalty        absent equals explicitly disabled

Counts:

before: {"rows": 19, "rows_with_configuration_difference": 9,  "rows_with_model_state_diagnostics": 19, "rows_with_semantic_difference": 1, "rows_with_substantive_disparity": 14, "rows_with_tracked_disparity": 18, "strict_final_status_disagreements": 0}
after:  {"rows": 19, "rows_with_configuration_difference": 5, "rows_with_configuration_equivalence": 7, "rows_with_model_state_diagnostics": 19, "rows_with_semantic_difference": 1, "rows_with_substantive_disparity": 14, "rows_with_tracked_disparity": 18, "strict_final_status_disagreements": 0}

disparity_dimensions drops configuration on exactly four rows: LinearSVC
iris, LinearSVC digits, Lasso diabetes, LinearRegression diabetes. No other
dimension moves on any row. Every row field outside
configuration_differences / configuration_equivalences /
disparity_dimensions is identical between the two reports, verified field by
field.

Proof the equivalence check bites

parity_contract.json was temporarily patched so the iris LogisticRegression
row records l2: 0.01 against sklearn's C: 1.0, then the report was
regenerated:

LogisticRegression / iris / accuracy   (5 -> 4)
    after:
        C              flow='<missing>'              sklearn=1.0
            ! l2 = 1 / (C * n_train) does not hold: n_train=120 and C=1.0 give l2=0.008333333333333333, recorded 0.01
        l2             flow=0.01                     sklearn='<missing>'
            ! l2 = 1 / (C * n_train) does not hold: n_train=120 and C=1.0 give l2=0.008333333333333333, recorded 0.01
        max_iter       flow=200                      sklearn=1000
        optimizer      flow='lbfgs_no_line_search'   sklearn='lbfgs'

Both entries come back, each carrying the value the relation demanded. The
contract was restored with git checkout -- and git status is clean of it;
the committed parity_contract.json is unchanged in this diff.

benchmarks/test_config_equivalence.py pins the same property without touching
the contract: the pre-#430 value (l2=0.001 against C=1.0 on iris, the 8.3x
mismatch #408 was about) is reported, an unknown n_train is reported, and the
digits n_train applied to the iris row is reported.

Verification

Everything below was run locally on this branch.

$ python3 benchmarks/test_config_equivalence.py
config equivalence fixtures: PASS                    (30 assertions)

$ python3 benchmarks/test_disparity_regression_policy.py
disparity regression policy fixtures: PASS

$ python3 benchmarks/test_generate_headline_summary.py
headline summary fixture: PASS

$ python3 benchmarks/publish_headline_v2.py --check
canonical benchmark, architecture, disparity, and available history evidence are internally consistent

$ python3 benchmarks/model_state_coverage.py --require-complete --disparity <new report>
{"canonical_rows": 19, "covered_rows": 19, "missing_rows": 0}

$ <the disparity.yml assertion block, against the new report>
{'disparity': {...}, 'model_state_coverage': {...}}     all assertions pass
$ publish_headline_v2.validate_disparity(new_report, 19)
validate_disparity OK

$ python3 benchmarks/check_disparity_regression.py --current <new report> \
      --baseline benchmarks/disparity_report.json --history <scratch copy>
runtime rule: same measurement host; a row fails when it slows by more than 1 log2 units (2.00x) relative to sklearn
disparity regression gate passed; history snapshots=23

The regression gate's configuration rule is one-sided (new_n - old_n > max_increase), so counts going down cannot trip it. Row keys, score fields and
runtime fields are untouched, and the compact history snapshot keeps the same
shape.

benchmarks/test_config_equivalence.py is added to the syntax-and-imports
job, next to the two existing benchmark fixture tests, and both new files are
added to that job's py_compile list.

No generated artifact is in this diff. disparity_report.json,
parity_diagnostics.json, disparity_history.json and
model_state_coverage.json were all written to a scratch path outside the
repository; git status is clean apart from the six files below.

.github/workflows/flow.yml                     |   3 +
benchmarks/README.md                           |  24 +++
benchmarks/config_equivalence.py               | 269 +++++++++++++++++++++
benchmarks/generate_disparity_report.py        |  52 +++--
benchmarks/test_config_equivalence.py          | 172 ++++++++++++++
benchmarks/test_disparity_regression_policy.py |   2 +-

I have not run CI; gh pr checks has not been consulted at the time of writing.

🤖 Generated with Claude Code