fix: centre multinomial softmax parameters to the sum-to-zero gauge - #476
Merged
Conversation
lin433_multinomial_fit now subtracts the class mean from every feature column and from the intercept vector before returning, so the model it reports is the sum-to-zero representative scikit-learn reports. The means are accumulated in f64 before the f32 store. The change is small on purpose. theta starts at zero and the softmax cross-entropy gradient sums to zero across classes, so every LBFGS iterate already lies in the sum-to-zero subspace in exact arithmetic. What the centring removes is f32 accumulation drift out of that subspace: measured at 1.1e-05 on digits and 1.0e-06 on iris before, 1.4e-07 and 7.5e-08 after. It also makes the gauge a property of the returned model rather than of the starting point. Predictions do not move. On the canonical benchmark both LogisticRegression accuracies are identical (0.933333337 iris, 0.975000024 digits) and 160 of the 165 RESULT/DETAIL records are byte-identical; the five that move are LogisticRegression coefficient records at the 1e-06 level. On a direct before/after comparison over the test sets, zero of 30 iris and zero of 360 digits predicted labels change, and the largest softmax probability movement is 8.3e-07. tests/test_multinomial_gauge.flow pins the gauge, the invariance and the recentring. Verified to bite by deleting the centring block and rerunning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #470
lin433_multinomial_fitnow centres the multinomial parameters before itreturns: for every feature column it subtracts the mean across classes from
each class's coefficient, and it subtracts the mean intercept from every
intercept. The means are accumulated in f64 before the f32 store, for the
reason PR #443 moved its line-search objective to f64.
That is the fix the issue asks for, and it is correct. It does not move the
diagnostic the issue was filed against, and the measurements below say why.
What the issue expected
LogisticRegression/digitsreportsintercepts_max_relative_diff = 0.508andcoef_row_l2_norms_max_relative_diff = 0.0125. The hypothesis was that Flowreports the raw point LBFGS reached while sklearn reports the sum-to-zero
representative, so most of that gap is a gauge artifact.
What the parameters actually look like
Flow was already producing sum-to-zero parameters before this change.
Measured on the canonical fits, largest absolute sum across classes:
sklearn's own digits fit sits in the same place: 4.62e-06 for coefficients,
1.11e-05 for intercepts. Both sides were already reporting the same gauge.
This is not an accident.
thetastarts at zero, and the softmaxcross-entropy gradient sums to zero across classes for every feature and for
the intercept because
sum_c (p_ic - t_ic) = 0. The L2 term is linear intheta and preserves that. Every LBFGS direction is a linear combination of
gradients and earlier steps, so in exact arithmetic each iterate stays in the
sum-to-zero subspace it started in. The residual above is f32 accumulation
drift, and the centring is what removes it.
The comment already in
tests/test_multinomial_logistic.flowsays the samething for the intercept-only fixture: "The iterates stay in the zero-sum
subspace because the residuals sum to zero across classes."
So what is the digits gap
Flow minus sklearn on the digits intercepts, class order 0..9:
Spread 0.246, standard deviation 0.092, mean -1.7e-06. A gauge shift is one
constant added to every class, so it would show up here as a constant vector.
This is not one.
Recomputing the fitted objective off-line on the same scaled training split,
mean cross-entropy plus
0.5 * alpha * ||W||^2withalpha = 1/(C m):Flow's objective is 4.7e-05 below sklearn's, and its gradient norm is
smaller. Neither side is exactly at the optimum, and the digits softmax
problem at C=1 is flat enough that two points 0.13 apart in an intercept
differ by 5e-05 in objective. The residual
intercepts_max_relative_diffisthat flatness plus the solver-path difference the row already declares
(
max_iter200 capped at 100 with no line search, against sklearn's 1000with one). It is a genuine difference in where each solver stopped, not a
parameterization artifact. #470 can close, but the diagnostic number will
stay where it is.
Why keep the change
It makes the gauge a property of the returned model rather than of the
starting point. A warm start, a non-zero initialisation, or a solver whose
step is not a pure combination of gradients would leave the subspace, and
nothing downstream would notice. It also removes the drift measured above.
Prediction invariance
benchmarks/bench_flow_v2.flowbefore and after, same host, 165RESULT|/DETAIL|records compared field by field with timings excluded:160 byte-identical, 5 changed. Both
LogisticRegressionaccuracies areunchanged.
The five records that move, all
LogisticRegression, all at 1e-06:Element-wise on the held-out sets, comparing
logistic_predictand thesoftmax probabilities of the same fitted model before and after centring it:
Zero labels move. The probability movement is one f32 ulp at that magnitude.
After the change the same comparison over an already-centred model gives
2.98e-08 (iris) and 2.38e-07 (digits) with 6 and 276 entries not bit-equal,
which is the floor of the f32 round trip.
Diagnostics, before and after
Recomputed with
benchmarks/generate_disparity_report.py's ownenrich_state_from_raw_detailsagainst the committedsklearn_results_v2.txt, so both sides are the same sklearn run:intercepts_max_relative_diffintercepts_max_abs_diffcoef_row_l2_norms_max_relative_diffcoef_frobenius_norm_relative_diffintercepts_max_relative_diffcoef_row_l2_norms_max_relative_diffNo drop, as the sections above predict. Whatever remains is the solver-path
difference, and it should be read as that.
Tests
New:
tests/test_multinomial_gauge.flow. Three checks on an 8-class,32-feature fixture: the fitted parameters sum to zero across classes within
nc * ulp(max); an arbitrary per-class shift leaves every softmaxprobability and every predicted label unchanged; removing the class mean from
the shifted model returns it to the fitted parameters.
It bites. With the centring block deleted the fixture reports a worst
coefficient class sum of 1.74e-06 against a bound of 1.10e-06 and an
intercept class sum of 6.24e-07 against a bound of 6.92e-08, and the file
exits 1.
Everything touching logistic regression run individually, all exit 0:
test_multinomial_logistic,test_multinomial_gauge,test_logistic_newton_solver,test_logistic_binary_decide,test_logistic_inference,test_opt_logisticregression_fit,test_opt_logisticregression_predict,test_opt_logisticregression_predict_proba,test_opt_logisticregression_decision_function,test_penalty_l2_from_c,test_parity_gap,test_new_features,test_advanced_estimators,test_cv_pipeline_label_values,test_estimator_persistence,test_model_selection,test_sparse_estimators.No pinned value in any of them moved.
tests/test_multinomial_logistic.flowpins the intercepts through theclosed-form
b = log(pi) - mean(log(pi)), which is already the centredrepresentative, so it passes unchanged.
linear.flowis over 150KB and compiler bug #469 makes exit codes aninsufficient check there, so the verification above is on printed values:
165 benchmark records compared field by field, not a green exit.
python tools/run_all.py:passed 113/113 failed 0/113, including the newtests/test_multinomial_gauge.flow. Other agents were compiling into the shared~/.local/bin/buildduring the run; nothing needed a re-run.