Make the fail-under summary agree with the exit code - #751
Conversation
pytest-cov decides the --cov-fail-under verdict twice with two different comparisons. pytest_sessionfinish uses coverage.py's should_fail_under, which compares the total rounded to [report] precision. The terminal summary compares the raw total with a bare `<`. When the raw total is below the threshold but rounds up to meet it, the two disagree: the run exits 0 while the summary prints a red "FAIL Required test coverage of N% not reached". Use the same predicate in both places. The exit code is unchanged; only the banner and its colour move, so no currently-passing build starts failing. Adds two regression tests alongside the existing test_cov_min_float_value group. Both fail on master and pass with this change; the rest of the fail-under tests are unaffected because their thresholds do not sit in the divergent band. Refs pytest-dev#638. pytest-dev#728 reports the same inconsistency from the opposite direction -- see the PR description.
|
Why does this pr show a huge removal/add change on plugin.py? Is that a display bug in github or what's going on? |
The file was committed with CRLF endings, which rewrote all 477 lines and made this pull request look like a full-file rewrite (+477 -471) instead of the seven-line change it is. That is what the review question was about. Endings are back to LF, matching upstream. The diff against master is now only the fail-under predicate change and its two comments; nothing else in this file differs.
|
Line endings, not a rewrite. The file had been committed with CRLF, so all 477 lines of The actual change in Thanks for looking at it. |
The defect
--cov-fail-underis decided twice, with two different comparisons.plugin.pypytest_sessionfinishshould_fail_under(self.cov_total, cov_fail_under, cov_precision)— coverage.py's own, compares the total rounded to[report] precisionplugin.pypytest_terminal_summaryfailed = self.cov_total < self.options.cov_fail_under— the raw totalWhen the raw total is below the threshold but rounds up to meet it, the two disagree. The run exits
0and prints a redFAILbanner at the same time.Reproduced on pytest 9.1.1 / pytest-cov 7.1.0 / coverage 7.15.2, with the suite's own
SCRIPTfixture (88.888…% covered) at the default precision of 0:round(88.888…, 0)is89, soshould_fail_undersays pass and the exit code is0— correctly, given the configured precision. Only the banner disagrees.The change
One line: the summary uses the predicate the exit-code path already uses.
The exit code does not change. Only the banner text and its colour move, so no build that passes today starts failing.
On the direction of the fix
The inconsistency is not in dispute, but the two open issues want it resolved in opposite directions, so this is a maintainer call and I want to be explicit about which way I went.
FAILshown when precision says the threshold was met. This PR fixes it.FAIL …" and that the problem is the exit code consulting the rounded value. That would be the mirror fix: changepytest_sessionfinishto use the raw comparison instead.I chose to align the banner rather than the exit code because:
should_fail_underis coverage.py's canonical predicate, andpytest_sessionfinishalready delegates to it deliberately — the semantic choice has been made in this codebase.[report] precision/--cov-precisionexists to declare the granularity that matters; at precision 0, 88.888…% is 89%.If you prefer #728's reading, the change belongs on the other line and I am happy to redo it that way.
Not changed
The banner still hardcodes
Total coverage: {actual:.2f}%rather than usingcov_precision, so at precision 3 it can still print a differently-rounded number than the table above it. Fixing that changes an expected string intest_cov_min_float_value_not_reached, so I have left it alone rather than fold an unrelated behaviour change into this PR. Happy to follow up.Tests
Two regression tests alongside the existing
test_cov_min_float_valuegroup — one at default precision, one mirroring #638 withprecision = 1. Both fail onmasterand pass with the change.Full suite on Windows / Python 3.13: 205 passed, 6 skipped (all pre-existing platform skips).
ruff checkandruff format --diffclean.The remaining fail-under tests are unaffected: their thresholds (88.88 @ p0, 88.89 @ p3, 100 @ p0, 50 @ p0) do not sit in the divergent band, so both predicates return the same boolean.