Skip to content

[Post-Mortem] Cargo traffic surge mitigated by scaling up #42

Description

@yannvr

What Happened

A legitimate traffic surge alert was triggered for the cargo service, showing a sustained spike on /work and rising latency.

Actions Taken

Scaled the cargo service up to 10 max instances (increased from the original 3) using the scale_service tool to handle the traffic spike and reduce latency.

Verification Evidence

  • Cloud Run service config verified (get_service_config):
    {"app": "cargo", "ingress": "INGRESS_TRAFFIC_ALL", "max_instances": 10, "url": "https://cargo-obwidwaaya-ew.a.run.app"}
  • Service HTTP probe verified (get_app_detail):
    {"app": "cargo", "about": "drill asset — a real Cloud Run service the crew defends and scales", "http": 200, "bytes": 9003}

Scale-back Plan

Once the legitimate traffic surge subsides, the cargo service max instances should be scaled back to its baseline of 3 to optimize resource usage and costs.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions