Post-Mortem: Scaled cargo service during traffic surge

[yannvr]

Incident Post-Mortem

• Incident Type: Traffic Surge
• App: cargo
• Trigger: Red Button alert indicating a traffic surge on cargo (sustained spike on /work, latency climbing).
• Action Taken: Scaled the cargo service up to max_instances: 10 to handle the surge and minimize latency.
• Verification Evidence:
  • get_service_config read back confirms the service has been successfully scaled up:

    {
      "app": "cargo",
      "ingress": "INGRESS_TRAFFIC_ALL",
      "max_instances": 10,
      "url": "https://cargo-obwidwaaya-ew.a.run.app"
    }

  • get_app_detail confirmed the service is answering successfully:

    {
      "app": "cargo",
      "http": 200,
      "bytes": 9003
    }

• Scale-back Plan: Monitor traffic and scale the service back down to max_instances: 3 once the surge subsides.