KernelSVC per-pair intercepts differ from sklearn by a sign convention; iris max_iter declared 200 vs 1000

[godofecht]

From the substantive-disparity survey. Two entangled items on the KernelSVC rows:

1. intercept_per_pair_max_relative_diff is 1.99 on iris and 2.0 on digits. A relative difference of ~2 on max-normalisation is the signature of a sign flip: |a - (-a)| / |a| = 2. Dual-coefficient sums match to 6.5e-5 (iris), so the models agree; the per-pair intercept convention differs — almost certainly which class of the (i,j) pair is treated as positive. Predictions compose correctly on both sides. Determine the convention difference and either align Flow's or transform at emission so the diagnostic compares like with like. Do not change decision values.

2. Configuration: iris declares max_iter 200 vs sklearn 1000 (digits declares 1000). Established on Close the Digits KernelSVC quality gap and isolate the libsvm performance deficit #206/SMO working-set selection: 98.4% of _smo_step calls make no progress, and the solver is 96% of KernelSVC fit #412: the solver converges in ~26 outer sweeps, so the cap is a backstop on both, but the declared configs should match or the difference should be shown to be inert (verify iris converges well under 200 sweeps, then align the declared value).

svm.flow is 76KB and the known victim of compiler bug #469: after any edit run tests/test_multiclass_svm.flow and tests/test_multiclass_kernel_svm.flow and diff printed accuracies against main.

[godofecht]

Diagnosis: it is pair orientation, and the sign is a consequence

The relationship is not a global sign flip. On iris only one of three pairs is negated; on digits, 19 of 45. The rule behind which ones is exact.

What libsvm/sklearn does

Confirmed against sklearn 1.9.0 rather than recollection, with decision_function_shape='ovo' on iris (gamma=0.25, C=1.0):

pair(0,1) mean dec for class0:  1.1554846880   class1: -1.1553219431
pair(0,2) mean dec for class0:  1.0876953861   class2: -1.0603767484
pair(1,2) mean dec for class1:  1.1814879710   class2: -1.1994213056

For pair (i, j) with i < j, the numerically smaller label is the +1 side. intercept_[p] is the constant term of that same decision function.

What Flow does

kernel_svc_multi_fit in lib/scikit/svm.flow builds pairs over class-discovery slots, not labels:

if y[i] == classes[a] { y_dual[idx] = 1.0 } else { y_dual[idx] = -1.0 }

with a < b indexing into classes, which distinct_f32 fills in order of first appearance in y_train. kernel_svc_multi_predict votes for pair_a on a positive decision value, so Flow is internally consistent with itself. It is only consistent with libsvm when the discovery order happens to be ascending by label.

Instrumented fit on the canonical iris split:

class_slot 0 label 0.0
class_slot 1 label 2.0
class_slot 2 label 1.0

raw_pair 0  plus_label 0.0  minus_label 2.0  b -0.134948671
raw_pair 1  plus_label 0.0  minus_label 1.0  b  0.012464760
raw_pair 2  plus_label 2.0  minus_label 1.0  b -0.017334666   <-- +1 side is the HIGHER label

Discovery order on this split is [0, 2, 1]. Pairs (0,2) and (0,1) come out with the lower label as +1 and agree with libsvm. Pair (2,1) comes out with 2 as +1, so its whole decision function, and therefore its intercept, is the negative of libsvm's.

The numbers

msd207_ksvc_pair_order in benchmarks/bench_flow_v2.flow already sorts the emitted pairs into ascending (lo, hi) label order, which is why pair_class_a/pair_class_b match exactly. It does not reorient the pair, so the intercept ships with Flow's orientation.

iris, emitted order (0,1), (0,2), (1,2):

idx	pair	sklearn	Flow	relationship
0	(0,1)	0.012601528	0.012464718	same sign, abs diff 1.368e-4
1	(0,2)	-0.13494871	-0.134948775	same sign, abs diff 6.5e-8
2	(1,2)	0.0172924496	-0.0171961244	negated, abs diff 3.4489e-2

max_abs_diff 0.0344885740 and max_relative_diff 1.9944296382 both come entirely from index 2.

digits, 45 pairs. Negated at indices:

4, 12, 14, 19, 21, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 39, 40, 41, 44

Agreeing at the other 26. max_abs_diff 1.521018926, max_relative_diff 1.9998502327.

Comparing magnitudes instead of signed values:

dataset	max_abs_diff signed	max_abs_diff on |x|	max_relative_diff signed	max_relative_diff on |x|
iris	0.0344885740	1.3681e-4	1.9944296382	0.0010137921
digits	1.521018926	1.1383578e-3	1.9998502327	0.0012719343

So the residual once orientation is accounted for is 1.4e-4 / 1.1e-3 absolute, which is the same f32-solver-vs-f64-libsvm noise floor already visible in dual_coef_abs_sum_per_pair (6.5e-5 iris, 2.4e-4 digits). No model disagreement remains.

Item 2: iris max_iter

_kernel_svc_smo_precomputed sets budget = max_iter * n and counts individual SMO pair updates, so max_iter is a sweep-equivalent multiplier rather than a sweep count. Instrumented on the canonical iris split (n = 80 per pair), with the cap swept over {1, 2, 3, 5, 10, 26, 50, 200, 1000, 5000}:

budget     80 -> steps 39, 30, 80
budget    160 -> steps 39, 30, 80
budget    240 -> steps 39, 30, 80
budget    400 -> steps 39, 30, 80
budget    800 -> steps 39, 30, 80
budget   2080 -> steps 39, 30, 80
budget   4000 -> steps 39, 30, 80
budget  16000 -> steps 39, 30, 80    (max_iter = 200, the declared value)
budget  80000 -> steps 39, 30, 80    (max_iter = 1000, sklearn's declared value)
budget 400000 -> steps 39, 30, 80

Every pair exits on the libsvm KKT rule gmax + gmax2 < tol, never on the budget. The worst pair needs 80 steps, one sweep-equivalent, against a declared budget of 16000. Accuracy and all three intercepts are bit-identical across the entire sweep, 0.966666639 and -0.134948671 / 0.012464760 / -0.017334666. The 200 vs 1000 difference is inert with 200x headroom.

Plan

Option (b): reorient at emission in msd207_ksvc_state, negating the intercept for pairs where Flow's pair_a carries the higher label, and align the declared iris cap to 1000. The solver stays untouched, which keeps svm.flow out of the blast radius of #469.

[godofecht]

Follow-up: the mechanism reproduces in closed form. Reading class-discovery order out of the committed split fixtures, forming Flow's slot pairs, sorting them the way msd207_ksvc_pair_order does, and marking every pair whose pair_a label exceeds its pair_b label predicts both flip sets exactly, with no Flow run involved:

iris   discovery order [0, 2, 1]                      flipped emitted indices [2]
digits discovery order [5, 0, 7, 1, 2, 9, 8, 6, 3, 4] flipped emitted indices
       [4, 12, 14, 19, 21, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 39, 40, 41, 44]

Same 1 and 19 indices the benchmark diff moves.

PR #475 takes option (b).