Skip to content

test: add Flutter remote push release check (MBL-2302) - #401

Draft
Shahroz16 wants to merge 54 commits into
mainfrom
codex/mbl-2233-flutter-remote-push-e2e
Draft

test: add Flutter remote push release check (MBL-2302)#401
Shahroz16 wants to merge 54 commits into
mainfrom
codex/mbl-2233-flutter-remote-push-e2e

Conversation

@Shahroz16

@Shahroz16 Shahroz16 commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a trusted Flutter FCM delivery and opened-attribution release check
  • identify a fresh profile, wait for iOS device registration, trigger the Mobile: Flutter send_push automation, and require backend delivery
  • create a disposable iPhone 17 Pro simulator so stale notifications cannot affect SpringBoard grouping
  • locate the Flutter notification through accessibility, reveal its system Open action, and require opened for the exact delivered message
  • keep credentials and raw artifacts out of git

When it runs

Apply the run-remote-push-e2e label to a trusted, same-repository pull request to run the check once for its current head. It does not run for ordinary pull-request updates or on a schedule. Deterministic client routing remains covered by #400 without backend credentials.

Validation

  • Bash syntax, ShellCheck, actionlint, Maestro YAML parsing, and diff checks pass on the current head
  • the label starts only the dedicated Xcode 27 remote-push workflow
  • the coordinate-free SpringBoard interaction was proven on a terminated sample using a disposable iPhone 17 Pro simulator
  • the current-head credentialed run is waiting on the correct Mobile: Flutter App API credential; the existing local credential was rejected because it belongs to a different workspace

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Sample app builds 📱

Below you will find the list of the latest versions of the sample apps. It's recommended to always download the latest builds of the sample apps to accurately test the pull request.


  • flutter_sample_spm: 401.54.0 (29797565)

@Shahroz16
Shahroz16 force-pushed the codex/mbl-2233-flutter-remote-push-e2e branch from cec15e6 to cf363bb Compare August 26, 2026 02:37
@Shahroz16
Shahroz16 force-pushed the codex/mbl-2233-flutter-remote-push-e2e branch from cf363bb to 42b9dbc Compare August 26, 2026 02:47
@Shahroz16
Shahroz16 force-pushed the codex/mbl-2233-flutter-remote-push-e2e branch from 42b9dbc to 35f1630 Compare August 26, 2026 02:53
@Shahroz16
Shahroz16 force-pushed the codex/mbl-2233-flutter-remote-push-e2e branch from 35f1630 to 5561fd1 Compare August 26, 2026 03:30
@Shahroz16
Shahroz16 force-pushed the codex/mbl-2233-flutter-remote-push-e2e branch from 5561fd1 to e82a9aa Compare August 26, 2026 03:55
@Shahroz16
Shahroz16 force-pushed the codex/mbl-2233-flutter-remote-push-e2e branch from e82a9aa to 1c8ae4a Compare August 26, 2026 04:03
@Shahroz16
Shahroz16 force-pushed the codex/mbl-2233-flutter-remote-push-e2e branch from 1c8ae4a to eaa27d7 Compare August 26, 2026 04:26
Base automatically changed from codex/mbl-2277-flutter-maestro-e2e to main August 26, 2026 23:34
…r-remote-push-e2e

# Conflicts:
#	.github/workflows/ios-toolchain-compatibility.yml
#	apps/flutter_sample_spm/.maestro/README.md
#	apps/flutter_sample_spm/.maestro/scene_push_prepare.yaml
@Shahroz16 Shahroz16 added run-remote-push-e2e Run the trusted real push delivery and opened-attribution E2E check and removed run-remote-push-e2e Run the trusted real push delivery and opened-attribution E2E check labels Aug 27, 2026
@Shahroz16
Shahroz16 marked this pull request as ready for review August 27, 2026 19:26
@Shahroz16
Shahroz16 requested a review from a team as a code owner August 27, 2026 19:26

@mrehan27 mrehan27 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the design. On a credentialed workflow the parts that matter are right: pull_request rather than pull_request_target plus the same-repo if: guard means forks never receive secrets, and the artifact export at run_remote_push.sh:246-248 sits behind redaction and --check under set -e, so unredacted evidence can't reach the upload step. Run 33107750971 exercised that path end to end. Harness pinned by SHA and verified with rev-parse, Maestro pinned with a checksum, disposable simulator per run.

Still red on this head, though the failure has moved: the campaign preflight now passes and it fails at the first backend assertion (device_change / devices.0.platform within 45s). Worth confirming that's environment timing rather than the assertion itself before this lands.

run_started_at_seconds="$(date +%s)"
identify_payload="$(jq -nc --arg id "$run_email" \
'{userId: $id, traits: {name: "Maestro Campaign Tester"}}')"
identify_status="$(curl -sS --retry 3 --retry-all-errors --retry-delay 2 --connect-timeout 10 \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 — every run identifies a new profile in the production Mobile: Flutter workspace and nothing removes it; there's no delete or suppression call anywhere in the script. Each run also triggers a real campaign send against that profile.

Label-gated so the rate is low, but profiles accumulate without bound and campaign 18's metrics collect synthetic sends. Worth deleting the profile in cleanup, or documenting a retention plan if these are deliberately kept.

die "App API key cannot access campaign 18 in the Mobile: Flutter workspace (HTTP ${campaign_status:-unreachable})"
campaign_name="$(printf '%s' "$campaign_body" | jq -r '.campaign.name // .name // empty' 2>/dev/null || true)"
campaign_event="$(printf '%s' "$campaign_body" | jq -r '.campaign.event_name // .event_name // empty' 2>/dev/null || true)"
[[ "$campaign_name" == "send_push" && "$campaign_event" == "send_push" ]] || \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 — campaign identity is hardcoded three ways: id 18 in the URL at :51, plus both name and event_name required to equal send_push. No override.

The validation itself is worth keeping — it fired correctly and caught a real mismatch on run 33105862946. The coupling is the problem: any of the three drifting yields one message that reads as a credential fault, and there's no way to point the check at a non-production workspace. Making the id and expected name configurable addresses both.

@Shahroz16
Shahroz16 marked this pull request as draft August 28, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-remote-push-e2e Run the trusted real push delivery and opened-attribution E2E check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants