From 56740a0d506e6e8e4d7dc1e3a057bf552ec818b4 Mon Sep 17 00:00:00 2001 From: godofecht Date: Fri, 21 Aug 2026 12:33:53 +0100 Subject: [PATCH 1/2] docs: trajectory panel shows Flow-vs-Python speedup per freeze Replaces the six absolute before/now columns with the speedup at the previous and latest freeze, plus a column attributing any >10% move to whichever side's own time moved more. Env-change banner retained. Co-Authored-By: Claude Fable 5 --- docs/benchmarks.html | 45 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/docs/benchmarks.html b/docs/benchmarks.html index 74136a9..d005b12 100644 --- a/docs/benchmarks.html +++ b/docs/benchmarks.html @@ -1,39 +1,38 @@ -Benchmarks — flow-scikit
flow~scikit

canonical v2 / parity + disparity benchmark

Eligibility never means identity.

All 19 canonical rows are measured and currently eligible for comparison, but numerical, semantic and runtime disparities remain first-class evidence. This page renders the committed benchmark and disparity artifacts directly so differences cannot disappear merely because a row passes its contract.

Flow wins—

End-to-end fit + predict comparisons won by Flow.

sklearn wins—

End-to-end comparisons won by scikit-learn.

parity eligible—

Rows admitted to the competitive denominator.

substantive disparities—

Rows whose fitted state, score, configuration or semantics genuinely diverge, above float-noise floors. Runtime differences are tracked per row but not counted here.

TIMING_UNIT|msend-to-endseed=4280/20 persisted split2% practical tie thresholddisparity retained after eligibility
KMeans note: Digits KMeans is eligible under the same declared contract as every other clustering row. Its seeded k-means++ initialization now matches scikit-learn's, so the strict diagnostic and the final eligibility decision agree. The convergence statistic, the point at which inertia is reported, empty-cluster relocation and the n_init selection rule still differ and stay visible in the disparity artifact.

runtime overview

runtime overview

The plots are generated from the canonical JSON.

Each runtime plot shows end-to-end fit + predict time on a log scale. The plots use the same rows as the table below and therefore update whenever the frozen canonical result changes.

All 19 speed ratios

scikit-learn total time divided by Flow total time. The vertical 1× line separates Flow wins from scikit-learn wins.

Iris total runtime

scikit-learnFlow

Digits total runtime

scikit-learnFlow

Diabetes total runtime

scikit-learnFlow

persistent disparity

Passing parity does not erase the gap.

The disparity plot normalizes each row's principal numerical difference against its effective tolerance where a tolerance is available. A value near 1 means the row is close to the acceptance boundary. Semantic/configuration differences are tracked in the same artifact and remain visible in the table.

Numerical disparity relative to tolerance

The dashed line is the acceptance boundary. Values can remain non-zero even for eligible rows.

all canonical rows

No selected-win table.

Every row is shown below. Speedup is sklearn_ms / flow_ms; values above 1× favor Flow. Strict diagnostic status is kept separate from final eligibility.

AlgorithmDatasetFinal parityStrict diagnosticWinnerscore |Δ|sklearn msFlow msspeedup

methodology

Correctness, disparity and timing are separate dimensions.

The benchmark consumes the same persisted train/test indices in Python and Flow. Python uses high-resolution adaptive timing and the canonical runner aggregates repeated process measurements with medians and IQR. Flow timings are emitted in milliseconds and aggregated by the same runner.

Supervised rows compare predictive metrics under declared tolerances. PCA additionally checks explained variance, singular values, reconstruction error and sign-aligned components. KMeans uses permutation-invariant clustering quality and inertia. The persistent disparity artifact preserves raw numerical gaps and known semantic/configuration differences even after the estimator-specific eligibility contract succeeds.

historical deployment evidence

Footprint and startup remain separate experiments.

The repository also contains a historical deployment comparison recording a roughly 1.4 MB Flow native executable and a roughly 65× cold-start advantage (33 ms versus 2160 ms). Those figures come from a different deployment experiment and are intentionally not mixed into the canonical estimator timing denominator.

trajectory

Each implementation's own timings, across freezes.

Every other figure on this page is a ratio of Flow to scikit-learn. A ratio cannot distinguish Flow getting faster from scikit-learn getting slower, and it hides the case where both move together. These are the absolute per-row totals recorded at each freeze.

reproduce

Read the source artifacts.

Canonical result ↗ Disparity report ↗

\ No newline at end of file From ffc3fdb21e6110619e8ae13b409bc93c4b9a7bcf Mon Sep 17 00:00:00 2001 From: godofecht Date: Fri, 21 Aug 2026 12:34:42 +0100 Subject: [PATCH 2/2] docs: key the environment banner on runtime_environment_id environment_id covers only the software stack. Freeze 42b61f5 ran on a different machine (runtime_environment_id fbf1f8c6 vs def1cc47) with the same software env, sklearn's BLAS-heavy rows ran up to 10x faster there, and the banner stayed silent. The regression gate already uses runtime_environment_id for runtime comparability; the banner now matches. Co-Authored-By: Claude Fable 5 --- docs/benchmarks.html | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/benchmarks.html b/docs/benchmarks.html index d005b12..491e881 100644 --- a/docs/benchmarks.html +++ b/docs/benchmarks.html @@ -6,9 +6,10 @@ if(!host) return; if(snaps.length<2){ host.innerHTML='

Only one freeze recorded so far; a trajectory needs at least two.

'; return; } const last=snaps[snaps.length-1], prev=snaps[snaps.length-2]; - if(warn && last.environment_id!==prev.environment_id){ + const hostOf=s=>s.runtime_environment_id||s.environment_id; + if(warn && hostOf(last)!==hostOf(prev)){ warn.innerHTML='

'+ - 'The measurement environment changed between the last two freezes. '+ + 'The last two freezes ran on different machines. '+ 'Flow’s own times are stable across runners; scikit-learn’s swing with the runner’s BLAS threading '+ '(up to 10× on some rows), so a speedup change between these two freezes may come from the '+ 'denominator rather than from either implementation’s code.

';