[infra] RabbitMQ hardening: ports, probes, graceful restart

[henzigo]

Context

The RabbitMQ StatefulSet is a 1:1 port of the legacy manifest and carries its quirks:

• containerPorts only declare 15672 + 15692; the actual AMQP port 5672 is not declared
  (works, but is undocumented and confusing)
• the containerPort named rabbitmq is actually the management UI port (15672)
• the container has no probes — wait: true on the infra release therefore only
  guarantees a Running pod, not a broker that accepts AMQP connections (see the
  discussion in Drop the legacy sleep 30 and bound the hook Jobs with activeDeadlineSeconds #44: the migration hook's "infra is ready" rationale is only fully true
  once a readinessProbe exists)
• no graceful shutdown: no preStop hook and the default 30 s grace period — an unclean
  broker stop risks slower recovery on the persisted data dir

Proposal

Adopt the probe + graceful-restart design from the upstream PR
shopsys/deployment#75 (never merged there):

• declare all three ports with accurate names: amqp (5672), management (15672),
  prometheus (15692); align the Service port names
• probes as in deployment#75: startupProbe exec rabbitmq-diagnostics ping
  (failureThreshold 30 × 10 s — RabbitMQ starts slowly on small nodes), livenessProbe
rabbitmq-diagnostics ping (30 s period, 15 s timeout, failureThreshold 5),
  readinessProbe rabbitmq-diagnostics -q check_running (10 s period)
• graceful restart (from deployment#75): preStop: rabbitmqctl stop_app && sleep 5
  • terminationGracePeriodSeconds: 90 so the broker flushes and closes connections
    cleanly before SIGTERM/SIGKILL
• memory request/limit defaults (deployment#75 used requests 256Mi / limits 512Mi) —
  review against real usage before adopting
• keep everything overridable through the existing standard component keys
  (livenessProbe/readinessProbe, lifecycle, terminationGracePeriodSeconds,
  resources)

Port names are safe to change (Services target numeric ports), but regenerate golden
snapshots and note the rename in the deviations list.

Tasks

• Ports + names in statefulset-rabbitmq.yaml and service-rabbitmq.yaml
• Probes incl. startupProbe (design per deployment#75)
• preStop rabbitmqctl stop_app + terminationGracePeriodSeconds: 90
• Memory request/limit defaults (validate the 256Mi/512Mi numbers first)
• Unit tests, golden snapshots, deviations entry
• Update deviation entry 9 wording in docs/migrating-from-shopsys-deployment.md
  (after this lands, wait: true again means "broker accepts connections")

Reference: shopsys/deployment#75; Related: #44 (review thread on the readiness guarantee)

[henzigo]

Concrete design exists upstream in shopsys/deployment#75 — port it into this issue's scope:

• ports: add containerPort: 5672 named rabbitmq (AMQP), rename 15672 to rabbitmq-management
• probes: startup rabbitmq-diagnostics ping (failureThreshold: 30, periodSeconds: 10),
  liveness rabbitmq-diagnostics ping (periodSeconds: 30, timeoutSeconds: 15, failureThreshold: 5),
  readiness rabbitmq-diagnostics -q check_running (periodSeconds: 10, timeoutSeconds: 10, failureThreshold: 3)
• graceful shutdown: preStop rabbitmqctl stop_app && sleep 5, terminationGracePeriodSeconds: 90
• resources: requests cpu: 20m, memory: 256Mi, limits memory: 512Mi
• explicit updateStrategy: RollingUpdate on the StatefulSet

All of it maps to existing values keys (probes/lifecycle/resources are standard component keys),
so this is mostly a defaults change + golden regeneration.

[henzigo]

Consider softening the liveness budget from PR #75
(failureThreshold: 5 × periodSeconds: 30 = 150 s) to failureThreshold: 3, timeoutSeconds: 10
(= 90 s) — Erlang VM restarts are expensive, but 150 s of a dead broker is a long outage window.
Also, while renaming ports, adopt the
protocol-prefix convention (amqp, http-management, http-metrics) and align the Service
port names with the container port names.

[henzigo]

Body extended (2026-08-20): adopted the graceful-restart design from shopsys/deployment#75 (preStop rabbitmqctl stop_app, 90 s grace, startup/liveness/readiness probes, memory requests) and linked the #44 review discussion — the migration hook's "infra wait ⇒ broker ready" rationale becomes fully true only once the readinessProbe from this issue lands.