test(conformance): run the network, and retract the gate it contradicts - #588
Merged
Conversation
Every earlier benchmark here ended with the same refusal: no network was run and no task accuracy was measured. This runs three -- GPT-2 124M, pythia-160m and opt-125m -- on WikiText-2, quantising transformer-block weights per-tensor into each candidate split, and reports perplexity. At 8 bits, e3m4 (the split the golden-section rule picks) comes first on gpt2 and opt-125m and second by 0.1% on pythia-160m. Against e4m3 that is a tie; against e5m2 it is a consistent advantage. e2m5 is bad everywhere. At 16 bits every split is lossless. That retracts the 1%-flushed gate from dot_product_bench.py. On GPT-2, e3m4 zeroes 1.75% of weights on average and still gives the LOWEST perplexity of the four candidates, while e2m5 zeroes 12.65% and costs 16%. Moderate flushing is cheap; the threshold was calibrated on a proxy. The number is still reported -- it is what relative error cannot see -- but it no longer disqualifies, because a task metric outranks a proxy gate. The sweep cannot call gf_ref.py 85M times per configuration, so there is a vectorised quantiser beside it, verified against the oracle bit for bit on 10800 values across nine splits. Mutation-checked: rounding mode and bias each fail that test. On opt-125m, e3m4 scores 0.998x -- quantisation slightly beat the baseline. That is noise, and it is the scale at which differences between the top splits should be read.
# Conflicts: # conformance/dot_product_bench.py # conformance/dot_product_bench_test.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every benchmark in this directory ended with the same refusal: no network was run and no task accuracy was measured. This runs three.
GPT-2 124M, pythia-160m, opt-125m on WikiText-2 test, transformer-block weights quantised per-tensor into each candidate split, metric = perplexity.
8 bits
e3m4— the split the golden-section rule picks — comes first on two models and second by 0.1% on the third. Againste4m3that is a tie, not a win; againste5m2it is a consistent advantage. At 16 bits every split is lossless (1.000×).On opt-125m,
e3m4scores 0.998× — quantisation slightly beat the baseline. That is noise, and it is the scale at which differences between the top splits should be read.This retracts the gate from #587
That PR introduced a hard 1%-flushed feasibility gate and called
e3m4UNUSABLE. The task metric contradicts it:Moderate flushing is cheap. The threshold was calibrated on a proxy, so the number is still reported — it is the thing relative error cannot see — but it no longer disqualifies. A task metric outranks a proxy gate, and leaving both verdicts in the repository would have been two truths side by side.
The fast path is verified, not assumed
The sweep cannot call
gf_ref.py85M times per configuration, so there is a vectorised numpy quantiser beside it — checked bit for bit against the oracle on 10,800 values across nine splits. Mutation-checked: rounding mode (both call sites) and the bias formula each fail that test. One narrow gap is recorded in the file rather than papered over.What this still does not establish
Weights only, three small models, one dataset, 24 windows of 512 tokens rather than the full test split. No training, no activation or gradient quantisation, no larger architecture. The e5m2 regime — activations and gradients, where outliers run ~100× typical — remains untouched.