test(atlas): catalogue the four 7.15 areas the report could not see - #33
Merged
Conversation
The PDF report is generated from SECTIONS, and the screenshot filter is derived from the same list — so a test that is not in SECTIONS is captured by nothing and appears nowhere. Four things a 7.15 audit needs were in exactly that state. NEW SECTIONS, all min_fw 7.15.0 so a 7.14.x report is unchanged (verified: at 7.14.2 the active set stays 18 sections and none of F/I/L/U appear; at 7.15.0 it is 26 and all four do): F Clear-Sign Provider Context - Additive Invariant 5 tests I Session and Trust Lifetime 6 tests L Bitcoin-Only Variant 11 tests U Storage Upgrade Preservation 8 tests THE HEADLINE RESULT: the additive invariant HOLDS, measured on device rather than argued from code. Aave supply() baseline is 3 screens; a VERIFIED v1 decode is 10 screens with those same 3 baseline frames BYTE-IDENTICAL at the tail; the v2 static-schema path is 13 with the same tail; a payload whose signature fails verification draws 3 frames byte-identical to the baseline — it neither refuses nor leaks partial decoded information. All four runtime slots behave identically, and no slot verifies without a runtime load, so the suppression branch has no reachable input on this build. A trap worth recording, because it would have made the section look right while proving nothing: for a RECOGNIZED ERC-20 the baseline has no raw-calldata screen at all — ethereum.c's token path skips it before clear-signing is consulted. A v2 test written against a USDC transfer would appear to prove "the raw review survives" when there was no raw review to survive. Section F deliberately uses the unrecognized Aave supply() fixture instead. The three additive tests are added to FULL_SEQUENCE_TESTS: the claim is about ORDER (decoded screens, then the baseline), so a best-of-3 frame sample would hide the very thing being proved. Also fixes two colliding section letters that made the report ambiguous: two sections emitted 'S' ids (Display Binding and Solana) and two emitted 'D' (BIP-85 and Display Disclosure), so distinct tests shared a label. The two disclosure sections are renamed to J and Q — safe because they post-date the pyk revision every published report was built from, whereas renaming Solana or BIP-85 would break existing evidence references. Requires keepkey-firmware PR #495. Two of the eleven bitcoin-only tests fail without it, and both failures are real firmware defects that this suite found: an OP_RETURN output poisoning the duplicate-transaction detector (affects BOTH products), and variant_getName() reporting "Emulator" for a bitcoin-only build so requires_fullFeature() never skipped anything.
All four are alpha tests asserting behaviour that develop's 7.14.2 line
deliberately changed. In every case the firmware refuses MORE or says MORE than
the test expects, so the tests follow the firmware — never the reverse.
1+2. The blind-sign refusal message (2 files, 3 assertions).
alpha: "Blind signing disabled"
7.14.2: "Arbitrary contract data signing disabled by policy"
The newer string names WHICH policy refused and what it refused.
3. Structured EIP-712 (test_ethereum_sign_x402_eip3009).
7.14.2 disabled it outright -- "Structured EIP-712 disabled pending
canonical display hardening" -- because the device could not prove that
what it rendered was what it hashed. The x402 EIP-3009
TransferWithAuthorization vector is therefore REFUSED, not signed. The test
asserts the refusal, and the expected domain/message hashes are kept in a
comment: they are independent EIP-712 V4 reference values and the only
checked-in oracle for this vector, so they should be re-asserted the day the
display hardening lands rather than re-derived.
Same file, second assertion: the typed-hash gate now answers "Enable
AdvancedMode to blind-sign typed hashes" rather than "disabled by policy" --
it names the remedy, not just the refusal.
4. test_reset_reentry_disarms_entropy_ack.
The property under test is the important one and is UNCHANGED: an abandoned
reset must never leave EntropyAck armed, or a following EntropyAck derives
the seed from sha256(0*32 || host_bytes) -- entirely host-chosen.
7.15 closes it earlier and harder than the fix this test was written for.
#429 replaced the separate awaiting_entropy flag with one armed ceremony,
and setup_stage() now REFUSES to open a second ceremony on top of an armed
one. The re-entry this test performed is rejected outright, so there is no
second ceremony left armed to disarm. The test now asserts BOTH: the refusal
("Device is in the middle of setup"), and then the original property --
EntropyAck refused with "Not in Reset mode", device still uninitialized.
Full local suite against the emulator: 630 passed, 22 skipped, 1 failed, and
that one failure (OP_RETURN poisoning the duplicate detector) is fixed by
keepkey-firmware #495, which is not on the branch this was run against.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The PDF report is generated from
SECTIONS, and the screenshot filter is derived from the same list — so a test not inSECTIONSis captured by nothing and appears nowhere. Four things a 7.15 audit needs were in exactly that state.All
min_fw7.15.0. Verified the cut is exact: at 7.14.2 the active set stays 18 sections with none of F/I/L/U; at 7.15.0 it is 26 with all four.The headline result
The additive invariant holds — measured on device, not argued from code:
supply()baseline = 3 screensA trap worth recording
For a recognized ERC-20, the baseline has no raw-calldata screen at all —
ethereum.c's token path skips it before clear-signing is consulted. A v2 test written against a USDC transfer would appear to prove "the raw review survives" when there was no raw review to survive. Section F deliberately uses the unrecognized Aavesupply()fixture.The three additive tests are wired into
FULL_SEQUENCE_TESTS— the claim is about order, so a best-of-3 frame sample would hide the thing being proved.Also: two colliding section letters
Two sections emitted
Sids (Display Binding, Solana) and two emittedD(BIP-85, Display Disclosure), so distinct tests shared a label. The disclosure sections are renamed J and Q — safe because they post-date the pyk revision every published report was built from; renaming Solana or BIP-85 would break existing evidence references.Depends on
keepkey-firmware #495. Two of the eleven bitcoin-only tests fail without it, and both are real firmware defects this suite found: an OP_RETURN output poisoning the duplicate-transaction detector (affects both products), and
variant_getName()reportingEmulatorfor a bitcoin-only build sorequires_fullFeature()never skipped anything.