ops: operational runbooks

[jonmatum]

Priority: P1
Parent: #34
Promotes: OPS-1 from #28

Context

No runbook exists for the common failure modes. On-call (once wired, see #38) needs a deterministic first response.

Required runbooks (docs/ops/runbooks/)

• bedrock-throttle.md — symptoms, detection (alarm names), mitigation (back-off, fall back to Haiku), escalation
• lambda-timeout.md — per-function timeout settings, how to extend safely, how to inspect X-Ray
• ddb-throttle.md — symptoms, check consumed capacity, escalate to on-demand limits if needed
• dlq-backlog.md — how to inspect, replay, and drain each DLQ
• waf-block-spike.md — how to tell attack from false positive, how to temporarily relax rules
• enrich-stuck.md — symptoms (nodes stuck on "Generating content…"), how to trigger retry, how to rollback
• agent-runaway.md — how to detect an agent session exceeding MAX_WRITES_PER_SESSION and how to rollback (scripts/rollback-agent.sh)

Each runbook uses the same template: Symptoms → Detection → Immediate Mitigation → Investigation → Long-term Fix → Related alarms / ADRs.

Acceptance criteria

• All 7 runbooks committed
• Linked from docs/ops/README.md and from each corresponding CloudWatch alarm description (where feasible)

References

• Production readiness — consolidated plan (tracking) #34 (parent)
• AWS Well-Architected Framework assessment — gaps and remediation plan #28 OPS-1