e2e: wait for dashboard QR content before final screenshot#714
Merged
Conversation
The final E2E screenshot captured the desktop wallpaper (stddev ~0.20) instead of the welcome dashboard with the QR code (stddev ~0.31) when test_translation_disabled.sh ran last: its restore_url() killed chromium right before screenshot.sh, and the paint threshold (0.08) was satisfied by the wallpaper before the relaunched kiosk's welcome iframe painted. - screenshot.sh: raise PAINT_STDDEV_THRESHOLD to 0.26 (above the wallpaper/shell baseline, below the full QR dashboard) and require two consecutive passing captures so the screenshot is taken only after the welcome iframe has rendered and settled. - test_translation_disabled.sh: restore_url() now waits for the dashboard kiosk process to relaunch on the original URL before returning, so screenshot.sh starts from a relaunched kiosk instead of the bare desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The final E2E screenshot captured the desktop wallpaper / logo (stddev
~0.20) instead of the welcome dashboard with the QR code (stddev~0.31). Tests still reportedALL TESTS PASSED, so the regression was silent.Root cause (deterministic, not flaky — reproduced twice with identical stddev
0.20752):test_translation_disabled.shis the last test beforescreenshot.sh.restore_url()writes the dashboard URL back tofullpageos.txt,killall chromium, then returns without waiting for the kiosk to relaunch.run_onepageosrespawns chromium, but the relaunched kiosk first paints the desktop shell/wallpaper and only later loads the welcome iframe (the QR code).screenshot.shaccepts a candidate as soon asdashboard_is_paintedis true, butPAINT_STDDEV_THRESHOLDwas0.08— and the shell/wallpaper-only state already scores0.20752, so it passed on the very first candidate before the QR iframe painted.Stddev measurements (cropped
1280x620+0+80, same as the hook):0.207520.1230930.3120330.08Fix
testing/hooks/screenshot.shPAINT_STDDEV_THRESHOLDdefault0.08→0.26(above the wallpaper/shell baseline0.20, below the full QR dashboard0.31).PAINT_STABLE_PASSES=2) so the screenshot is taken only after the welcome iframe has rendered and settled, not on a mid-load transition. On timeout the hook still fails and savesscreenshot-not-painted.png.testing/tests/test_translation_disabled.shrestore_url()now waits for the dashboard kiosk process to relaunch on the original URL (--kiosk --app=http://localhost/FullPageDashboard) before returning, soscreenshot.shstarts from a relaunched kiosk rather than the bare desktop.Evidence
Verified against CustomPiOS PR #274 (
pr-274-localimage) + FullPageOSdevel(7c9b93d), QEMU aarch64.Before (
e2e_results/pr274-devel-rerun-20260713/, unchangeddevel):screenshot.png= desktop wallpaper / logo, stddev0.20752Display screenshot saved after dashboard paint (1x2s)TEST_RESULT=0,exit-code 0(false pass)After (
e2e_results/pr274-devel-qrfix-20260713/, this branch):restore_url:Dashboard kiosk restored (pid: 8480) after 2x2sstddev 0→0.312033→Painted but waiting for stabilization (1/2)→0.312033→Display screenshot saved after dashboard paint (2 stable passes, 3x2s)screenshot.png= welcome dashboard with QR code + setup speech bubble, stddev0.312033TEST_RESULT=0,exit-code 0,ALL TESTS PASSEDThe fix is on the FullPageOS side only; no CustomPiOS changes.
Test plan
develto confirm the wallpaper screenshot reproduces (not flaky)0.312033)