Reconcile the iLert operator alert stream

[morgaesis]

Operator alert monitoring no longer depends on a renewable local login to configure iLert. The manual production-monitor canary uses the existing GitHub OIDC identity to reconcile one fail-closed webhook action, verifies successful create and resolve deliveries through iLert action history, and always cleans up its unique alert; routine production failures retain their existing alert lifecycle.

[postil-dev]

1 blocking finding open
1 finding posted inline

1 suppressed

• Manual alert tests no longer trigger notifications at .github/workflows/production-monitor.yml:452: below the configured confidence threshold; severity error, confidence 0.04. The notification job now runs only when the smoke job fails, so manual runs with test_alert=true skip the external alert. Preserve the operator-requested test-alert branch and its delivery verification.

_{Review details}

[postil-dev]

1 blocking finding open
1 finding posted inline

3 suppressed

• Allow the manual canary to run when smoke fails at .github/workflows/production-monitor.yml:540: cites a line the named construct does not sit on; severity warn, confidence 0.94. Because this job needs smoke and its condition is not wrapped in always(), GitHub Actions skips the job when smoke fails or is cancelled, even when inputs.test_alert is true. This regresses the prior manual test-alert path, which could validate delivery during monitor failures. Remove the dependency or use an always-evaluated condition that explicitly permits the manual canary to run.
• Preserve external alerts for manual test runs at .github/workflows/production-monitor.yml:598: cites a line the named construct does not sit on; severity warn, confidence 0.98. This condition removed the inputs.test_alert == true branch, so a manually requested test run no longer reaches the external alert action even when the test is intended to verify alert delivery. Retain the test-alert condition or otherwise route test runs to the alert action without affecting production failure handling.
• Restore test-alert notification delivery at .github/workflows/production-monitor.yml:598: below the configured confidence threshold; severity error, confidence 0.04. The notification job no longer runs for operator-requested test alerts, so successful smoke and recovery checks cannot exercise the external alert path. Restore the test-alert condition or invoke an equivalent explicit alert action.

_{Review details}

[postil-dev]

1 blocking finding open
1 finding posted inline

Before the next push

Run postil review --staged.

_{Review details}

[morgaesis]

@postil re-review

[postil-dev]

review incomplete · 1 blocking finding open
1 finding posted inline

1 suppressed

• Preserve operator-requested test alerts at .github/workflows/production-monitor.yml:598: cites a line the named construct does not sit on; severity error, confidence 0.94. This condition removed the inputs.test_alert == true branch. When an operator requests a test alert and both prerequisite jobs pass, notify is skipped, so the alerting path cannot be tested and the requested external alert is never sent. Include || inputs.test_alert == true in the condition, while preserving the existing failure and cancellation cases.

_{Review details}