Skip to content

Post-Mortem: sandbox 5xx Outage Recovered #36

Description

@yannvr

Incident Post-Mortem

What Happened

On August 31, 2026, the sandbox service experienced an outage and was returning HTTP 500 errors to users. The watch officer diagnosed and confirmed the incident with the following evidence:

  • get_fleet_status reported HTTP 500 for sandbox with a latency of 23ms.
  • get_app_detail returned HTTP 500 with the body snippet: "sandbox: internal error. [armor: quarantined]...".

Action Taken

The Engineer executed the healing runbook for the affected service:

  • Called heal_service(app="sandbox") which successfully completed ({"healed":true,"was_broken":true,"service":"sandbox"}).

Verification Evidence

We verified that the service has fully recovered and is now healthy:

  • get_app_detail(app="sandbox") returned HTTP 200 with response body "sandbox: ok".

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions