Post-Mortem: Traffic Surge on cargo Service Scaled to 10 Max Instances

[yannvr]

Incident Summary

A legitimate traffic surge was detected on the cargo service, manifesting as a sustained spike on the /work endpoint with climbing latency.

Action Taken

• Scaling Action: Scaled the cargo service's Cloud Run configuration up from 3 to 10 max instances using the scale_service tool to handle the surge and maintain performance.
• Future Scale-Back Note: The service should be scaled back to its baseline of 3 max instances once traffic returns to normal.

Verification Evidence

1. Cloud Run Config verification:

{"app": "cargo", "ingress": "INGRESS_TRAFFIC_ALL", "max_instances": 10, "url": "https://cargo-obwidwaaya-ew.a.run.app"}

2. App HTTP check verification:

{"app": "cargo", "http": 200, "bytes": 9003, "final_url": "https://cargo-294160018950.europe-west1.run.app"}