test: add Flutter remote push release check (MBL-2302) - #401
Conversation
Sample app builds 📱Below you will find the list of the latest versions of the sample apps. It's recommended to always download the latest builds of the sample apps to accurately test the pull request.
|
cec15e6 to
cf363bb
Compare
cf363bb to
42b9dbc
Compare
42b9dbc to
35f1630
Compare
35f1630 to
5561fd1
Compare
5561fd1 to
e82a9aa
Compare
e82a9aa to
1c8ae4a
Compare
1c8ae4a to
eaa27d7
Compare
…-flutter-remote-push-e2e
…-flutter-remote-push-e2e
…-flutter-remote-push-e2e
…-flutter-remote-push-e2e
…-flutter-remote-push-e2e
…-flutter-remote-push-e2e
…-flutter-remote-push-e2e
…r-remote-push-e2e # Conflicts: # .github/workflows/ios-toolchain-compatibility.yml # apps/flutter_sample_spm/.maestro/README.md # apps/flutter_sample_spm/.maestro/scene_push_prepare.yaml
mrehan27
left a comment
There was a problem hiding this comment.
Approving the design. On a credentialed workflow the parts that matter are right: pull_request rather than pull_request_target plus the same-repo if: guard means forks never receive secrets, and the artifact export at run_remote_push.sh:246-248 sits behind redaction and --check under set -e, so unredacted evidence can't reach the upload step. Run 33107750971 exercised that path end to end. Harness pinned by SHA and verified with rev-parse, Maestro pinned with a checksum, disposable simulator per run.
Still red on this head, though the failure has moved: the campaign preflight now passes and it fails at the first backend assertion (device_change / devices.0.platform within 45s). Worth confirming that's environment timing rather than the assertion itself before this lands.
| run_started_at_seconds="$(date +%s)" | ||
| identify_payload="$(jq -nc --arg id "$run_email" \ | ||
| '{userId: $id, traits: {name: "Maestro Campaign Tester"}}')" | ||
| identify_status="$(curl -sS --retry 3 --retry-all-errors --retry-delay 2 --connect-timeout 10 \ |
There was a problem hiding this comment.
P2 — every run identifies a new profile in the production Mobile: Flutter workspace and nothing removes it; there's no delete or suppression call anywhere in the script. Each run also triggers a real campaign send against that profile.
Label-gated so the rate is low, but profiles accumulate without bound and campaign 18's metrics collect synthetic sends. Worth deleting the profile in cleanup, or documenting a retention plan if these are deliberately kept.
| die "App API key cannot access campaign 18 in the Mobile: Flutter workspace (HTTP ${campaign_status:-unreachable})" | ||
| campaign_name="$(printf '%s' "$campaign_body" | jq -r '.campaign.name // .name // empty' 2>/dev/null || true)" | ||
| campaign_event="$(printf '%s' "$campaign_body" | jq -r '.campaign.event_name // .event_name // empty' 2>/dev/null || true)" | ||
| [[ "$campaign_name" == "send_push" && "$campaign_event" == "send_push" ]] || \ |
There was a problem hiding this comment.
P2 — campaign identity is hardcoded three ways: id 18 in the URL at :51, plus both name and event_name required to equal send_push. No override.
The validation itself is worth keeping — it fired correctly and caught a real mismatch on run 33105862946. The coupling is the problem: any of the three drifting yields one message that reads as a credential fault, and there's no way to point the check at a non-production workspace. Making the id and expected name configurable addresses both.
Summary
send_pushautomation, and require backend deliveryOpenaction, and requireopenedfor the exact delivered messageWhen it runs
Apply the
run-remote-push-e2elabel to a trusted, same-repository pull request to run the check once for its current head. It does not run for ordinary pull-request updates or on a schedule. Deterministic client routing remains covered by #400 without backend credentials.Validation