Skip to content

fix: centre multinomial softmax parameters to the sum-to-zero gauge - #476

Merged
godofecht merged 1 commit into
mainfrom
fix/470-multinomial-centering
Aug 21, 2026
Merged

fix: centre multinomial softmax parameters to the sum-to-zero gauge#476
godofecht merged 1 commit into
mainfrom
fix/470-multinomial-centering

Conversation

@godofecht

@godofecht godofecht commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Closes #470

lin433_multinomial_fit now centres the multinomial parameters before it
returns: for every feature column it subtracts the mean across classes from
each class's coefficient, and it subtracts the mean intercept from every
intercept. The means are accumulated in f64 before the f32 store, for the
reason PR #443 moved its line-search objective to f64.

That is the fix the issue asks for, and it is correct. It does not move the
diagnostic the issue was filed against, and the measurements below say why.

What the issue expected

LogisticRegression/digits reports intercepts_max_relative_diff = 0.508 and
coef_row_l2_norms_max_relative_diff = 0.0125. The hypothesis was that Flow
reports the raw point LBFGS reached while sklearn reports the sum-to-zero
representative, so most of that gap is a gauge artifact.

What the parameters actually look like

Flow was already producing sum-to-zero parameters before this change.
Measured on the canonical fits, largest absolute sum across classes:

coefficient column intercept vector largest parameter
iris, before 1.01e-06 9.24e-07 2.49
iris, after 7.45e-08 8.94e-08 2.49
digits, before 1.10e-05 6.30e-06 2.10
digits, after 1.38e-07 1.49e-07 2.10

sklearn's own digits fit sits in the same place: 4.62e-06 for coefficients,
1.11e-05 for intercepts. Both sides were already reporting the same gauge.

This is not an accident. theta starts at zero, and the softmax
cross-entropy gradient sums to zero across classes for every feature and for
the intercept because sum_c (p_ic - t_ic) = 0. The L2 term is linear in
theta and preserves that. Every LBFGS direction is a linear combination of
gradients and earlier steps, so in exact arithmetic each iterate stays in the
sum-to-zero subspace it started in. The residual above is f32 accumulation
drift, and the centring is what removes it.

The comment already in tests/test_multinomial_logistic.flow says the same
thing for the intercept-only fixture: "The iterates stay in the zero-sum
subspace because the residuals sum to zero across classes."

So what is the digits gap

Flow minus sklearn on the digits intercepts, class order 0..9:

0.04835  -0.13273  0.01106  0.00637  -0.11226
0.10810  -0.11728  0.10993  0.11350  -0.03506

Spread 0.246, standard deviation 0.092, mean -1.7e-06. A gauge shift is one
constant added to every class, so it would show up here as a constant vector.
This is not one.

Recomputing the fitted objective off-line on the same scaled training split,
mean cross-entropy plus 0.5 * alpha * ||W||^2 with alpha = 1/(C m):

objective mean CE gradient norm
Flow 0.066468071 0.028899868 1.25e-04
sklearn 0.066514768 0.029139766 4.82e-04

Flow's objective is 4.7e-05 below sklearn's, and its gradient norm is
smaller. Neither side is exactly at the optimum, and the digits softmax
problem at C=1 is flat enough that two points 0.13 apart in an intercept
differ by 5e-05 in objective. The residual intercepts_max_relative_diff is
that flatness plus the solver-path difference the row already declares
(max_iter 200 capped at 100 with no line search, against sklearn's 1000
with one). It is a genuine difference in where each solver stopped, not a
parameterization artifact. #470 can close, but the diagnostic number will
stay where it is.

Why keep the change

It makes the gauge a property of the returned model rather than of the
starting point. A warm start, a non-zero initialisation, or a solver whose
step is not a pure combination of gradients would leave the subspace, and
nothing downstream would notice. It also removes the drift measured above.

Prediction invariance

benchmarks/bench_flow_v2.flow before and after, same host, 165
RESULT|/DETAIL| records compared field by field with timings excluded:
160 byte-identical, 5 changed. Both LogisticRegression accuracies are
unchanged.

RESULT|LogisticRegression|iris|accuracy|0.933333337     (before and after)
RESULT|LogisticRegression|digits|accuracy|0.975000024   (before and after)

The five records that move, all LogisticRegression, all at 1e-06:

DETAIL|LogisticRegression|digits|coef_row_l2_norms
  before 2.3551352,3.59787488,3.59507728,3.70378041,3.4379251,3.37079287,2.85742688,2.80640364,3.00237036,3.82105875
  after  2.3551352,3.59787488,3.5950768,3.70378017,3.43792486,3.37079382,2.8574264,2.80640292,3.00236988,3.8210597
DETAIL|LogisticRegression|digits|intercepts
  before -0.249675035,-0.966414332,-0.153456986,0.0822226033,-0.617585778,0.454551131,-0.667063475,0.216171026,2.09950805,-0.198263496
  after  -0.24967441,-0.966413677,-0.15345636,0.0822232366,-0.617585123,0.454551756,-0.667062819,0.216171652,2.09950876,-0.19826287
DETAIL|LogisticRegression|iris|coef_abs_sum
  before 13.221365            after 13.221364
DETAIL|LogisticRegression|iris|coef_row_l2_norms
  before 2.88320565,1.05400169,3.31365442
  after  2.88320541,1.05400145,3.3136549
DETAIL|LogisticRegression|iris|intercepts
  before -0.305306703,1.90892518,-1.60361755
  after  -0.305307001,1.90892482,-1.60361791

Element-wise on the held-out sets, comparing logistic_predict and the
softmax probabilities of the same fitted model before and after centring it:

test rows label flips probability entries not bit-equal largest movement
iris 30 0 90 38 1.19e-07
digits 360 0 3600 2560 8.34e-07

Zero labels move. The probability movement is one f32 ulp at that magnitude.
After the change the same comparison over an already-centred model gives
2.98e-08 (iris) and 2.38e-07 (digits) with 6 and 276 entries not bit-equal,
which is the floor of the f32 round trip.

Diagnostics, before and after

Recomputed with benchmarks/generate_disparity_report.py's own
enrich_state_from_raw_details against the committed
sklearn_results_v2.txt, so both sides are the same sklearn run:

row field before after
digits intercepts_max_relative_diff 0.5079057403 0.5079071654
digits intercepts_max_abs_diff 0.132488846 0.132488191
digits coef_row_l2_norms_max_relative_diff 0.01249790243 0.01249806031
digits coef_frobenius_norm_relative_diff 0.002576328709 0.002576328709
iris intercepts_max_relative_diff 0.0009052359034 0.0009042607194
iris coef_row_l2_norms_max_relative_diff 0.0005097107817 0.0005099383693

No drop, as the sections above predict. Whatever remains is the solver-path
difference, and it should be read as that.

Tests

New: tests/test_multinomial_gauge.flow. Three checks on an 8-class,
32-feature fixture: the fitted parameters sum to zero across classes within
nc * ulp(max); an arbitrary per-class shift leaves every softmax
probability and every predicted label unchanged; removing the class mean from
the shifted model returns it to the fitted parameters.

It bites. With the centring block deleted the fixture reports a worst
coefficient class sum of 1.74e-06 against a bound of 1.10e-06 and an
intercept class sum of 6.24e-07 against a bound of 6.92e-08, and the file
exits 1.

Everything touching logistic regression run individually, all exit 0:
test_multinomial_logistic, test_multinomial_gauge,
test_logistic_newton_solver, test_logistic_binary_decide,
test_logistic_inference, test_opt_logisticregression_fit,
test_opt_logisticregression_predict,
test_opt_logisticregression_predict_proba,
test_opt_logisticregression_decision_function, test_penalty_l2_from_c,
test_parity_gap, test_new_features, test_advanced_estimators,
test_cv_pipeline_label_values, test_estimator_persistence,
test_model_selection, test_sparse_estimators.

No pinned value in any of them moved.
tests/test_multinomial_logistic.flow pins the intercepts through the
closed-form b = log(pi) - mean(log(pi)), which is already the centred
representative, so it passes unchanged.

linear.flow is over 150KB and compiler bug #469 makes exit codes an
insufficient check there, so the verification above is on printed values:
165 benchmark records compared field by field, not a green exit.

python tools/run_all.py: passed 113/113 failed 0/113, including the new
tests/test_multinomial_gauge.flow. Other agents were compiling into the shared
~/.local/bin/build during the run; nothing needed a re-run.

lin433_multinomial_fit now subtracts the class mean from every feature
column and from the intercept vector before returning, so the model it
reports is the sum-to-zero representative scikit-learn reports. The means
are accumulated in f64 before the f32 store.

The change is small on purpose. theta starts at zero and the softmax
cross-entropy gradient sums to zero across classes, so every LBFGS iterate
already lies in the sum-to-zero subspace in exact arithmetic. What the
centring removes is f32 accumulation drift out of that subspace: measured
at 1.1e-05 on digits and 1.0e-06 on iris before, 1.4e-07 and 7.5e-08
after. It also makes the gauge a property of the returned model rather
than of the starting point.

Predictions do not move. On the canonical benchmark both LogisticRegression
accuracies are identical (0.933333337 iris, 0.975000024 digits) and 160 of
the 165 RESULT/DETAIL records are byte-identical; the five that move are
LogisticRegression coefficient records at the 1e-06 level. On a direct
before/after comparison over the test sets, zero of 30 iris and zero of 360
digits predicted labels change, and the largest softmax probability
movement is 8.3e-07.

tests/test_multinomial_gauge.flow pins the gauge, the invariance and the
recentring. Verified to bite by deleting the centring block and rerunning.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@godofecht
godofecht merged commit 704cf97 into main Aug 21, 2026
8 checks passed
@godofecht
godofecht deleted the fix/470-multinomial-centering branch August 21, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multinomial softmax parameters are unidentified up to a per-class shift; sklearn centres them, Flow does not

1 participant