Post-Mortem: Cargo Traffic Surge Handled and Service Scaled Up

[yannvr]

Incident Post-Mortem: Cargo Traffic Surge

What Happened:
The cargo service experienced a legitimate traffic surge (sustained spike on /work with latency climbing). Prior to intervention, the service was returning 404 Page Not Found errors externally due to restricted ingress (INGRESS_TRAFFIC_INTERNAL_ONLY) and was configured with only 3 max instances.

Actions Taken:

1. Restored Ingress: Brought the service online with bring_online to change ingress back to public (INGRESS_TRAFFIC_ALL).
2. Scaled Instances: Scaled the service up to 10 max instances using scale_service to accommodate the traffic surge.

Verification Evidence:

• Service Configuration verified:

  {"app": "cargo", "ingress": "INGRESS_TRAFFIC_ALL", "max_instances": 10, "url": "https://cargo-obwidwaaya-ew.a.run.app"}

• App Details verified (200 OK):

  {"app": "cargo", "about": "drill asset — a real Cloud Run service the crew defends and scales", "http": 200, "bytes": 9003, "server": "Google Frontend"}

Scale-back Plan:

• Once traffic levels return to normal, cargo should be scaled back to 3 max instances to optimize costs.