Skip to content

Config comparator flags identical settings expressed in different vocabularies #469

Description

@godofecht

From the substantive-disparity survey after #468. 9 of 19 rows carry a "configuration" dimension, and most entries are the same setting under two names:

  • {"parameter": "C", "flow": "<missing>", "sklearn": 1.0} next to {"parameter": "l2", "flow": 0.008333333, "sklearn": "<missing>"} — these are the same configuration: fix: match sklearn's C=1.0 in the canonical LogisticRegression benchmark (#408) #430 established alpha = 1/(C * n_samples), and 0.008333 = 1/(1.0 * 120) exactly.
  • {"parameter": "penalty", "flow": "none", "sklearn": "<missing>"} on LinearRegression — sklearn OLS has no penalty parameter; identical semantics.
  • {"parameter": "learning_rate", "flow": 0.1, "sklearn": "<missing>"} — sklearn lbfgs has no learning rate; comparing it is meaningless.
  • {"parameter": "dual", "flow": "<missing>", "sklearn": "auto"} on LinearSVC — Flow has no such switch.

Real differences are hiding in the noise: max_iter 200 vs 1000 on LogisticRegression and KernelSVC iris is genuine and currently drowned out.

Fix: teach the comparator declared equivalences (C ↔ alpha via the #430 mapping with the row's n_train, absent ↔ explicit-none, solver-specific parameters ignored when the counterpart solver has no such knob), so surviving entries are genuine divergences. Find where CONFIG records are emitted and compared first; do not hardcode row names.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions