Description
CTs sometimes fail for (mostly its timing issues), and the tests are red, even though the functionality is green. This is wasteful
Environment
github hosted runners, see config in .github (stable for last year)
Steps To Reproduce
run the CTs, see that some sometimes fail
Expected behavior
CTs should fail when what the tests assert is broken
Actual Behavior
CTs sometimes fail due to flakyness and rerunning the tests makes them green.
Additional context
It might be simply helpful to categorize which types of flakyness we currently have. A zoology of test-devils, so to say.
This bug is not high prio, just me being impatient with waiting for test runs
BTW: the benchmark has similar issues
Description
CTs sometimes fail for (mostly its timing issues), and the tests are red, even though the functionality is green. This is wasteful
Environment
github hosted runners, see config in .github (stable for last year)
Steps To Reproduce
run the CTs, see that some sometimes fail
Expected behavior
CTs should fail when what the tests assert is broken
Actual Behavior
CTs sometimes fail due to
flakynessand rerunning the tests makes them green.Additional context
It might be simply helpful to categorize which
typesof flakyness we currently have. Azoology of test-devils, so to say.This bug is not high prio, just me being impatient with waiting for test runs
BTW: the benchmark has similar issues