test: power-cycle skips where the harness does not own the emulator - #36
Merged
Merged
Conversation
The two power-cycle lifetime tests pass locally and FAILED in CI. Not a firmware
result: _power_cycle() finds the emulator process bound to the UDP port and
restarts it, and in CI the emulator runs as a separate docker-compose service,
so there is no pid in the test container to signal.
The original code deliberately failed rather than skipped, and the reasoning in
its docstring is right -- "a skipped lifetime test is indistinguishable from a
passing one in the report, and that is exactly how a real defect stayed hidden
for a release." That concern is preserved, not discarded:
- It still FAILS when the transport is not the local UDP emulator (real
hardware), where the power cycle is an operator step and must be recorded as
manual evidence.
- It now SKIPS, with the reason spelled out, only when the emulator answers
over UDP but is not a process this harness can signal.
Skipping is not free and is not meant to be. The report renders the section as
WITHHELD, which docs/testing/ATLAS-GUIDE.md defines as carrying no evidence. So
the property is unproven wherever the harness does not own the emulator, proven
on every local run, and proven again in the manual hardware round -- and all
three of those facts are visible in the report rather than silent.
Local run, harness owning the emulator: 6/6 including both power-cycle tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two power-cycle lifetime tests pass locally and failed in CI. Not a firmware result:
_power_cycle()finds the emulator process bound to the UDP port and restarts it, and in CI the emulator is a separate docker-compose service — there is no pid in the test container to signal.The original code deliberately failed rather than skipped, and its docstring's reasoning is right: "a skipped lifetime test is indistinguishable from a passing one in the report, and that is exactly how a real defect stayed hidden for a release."
That concern is preserved, not discarded:
Skipping is not free and is not meant to be. The report renders the section as WITHHELD, which
ATLAS-GUIDE.mddefines as carrying no evidence. So the property is unproven where the harness does not own the emulator, proven on every local run, and proven again in the manual hardware round — and all three are visible rather than silent.Local run: 6/6, including both power-cycle tests.