Skip to content

Add default readiness/liveness probes to the RabbitMQ StatefulSet #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
henzigo wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add default readiness/liveness probes to the RabbitMQ StatefulSet #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion charts/shopsys-infra/templates/statefulset-rabbitmq.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,45 @@ spec:
{{- end }}
resources:
{{- toYaml .Values.rabbitmq.resources | nindent 12 }}
{{- include "shopsys.containerSettings" .Values.rabbitmq | nindent 10 }}
{{- with .Values.rabbitmq.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if .Values.rabbitmq.livenessProbe }}
livenessProbe:
{{- toYaml .Values.rabbitmq.livenessProbe | nindent 12 }}
{{- else }}
{{- /* Stage-1 health check (https://www.rabbitmq.com/docs/monitoring#health-checks):
validates the runtime is up and CLI authentication works. rabbitmq-diagnostics
spawns an Erlang VM, so the budget is deliberately generous - a false positive
here costs a full broker restart. */}}
livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
{{- end }}
{{- if .Values.rabbitmq.readinessProbe }}
readinessProbe:
{{- toYaml .Values.rabbitmq.readinessProbe | nindent 12 }}
{{- else }}
{{- /* The AMQP listener opens as one of the last boot steps, so a TCP check on 5672
is exactly "the broker accepts connections" - the upstream Kubernetes
Operator's default and the practice recommended by the RabbitMQ docs (CLI
checks are explicitly discouraged as readiness probes). */}}
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
{{- end }}
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
70 changes: 70 additions & 0 deletions charts/shopsys-infra/tests/infra_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,76 @@ tests:
name: rabbitmq-credentials
key: user

- it: gives rabbitmq default probes so infra readiness means an accepting broker
template: templates/statefulset-rabbitmq.yaml
asserts:
- equal:
path: spec.template.spec.containers[0].readinessProbe
value:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
- equal:
path: spec.template.spec.containers[0].livenessProbe
value:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3

- it: replaces the default rabbitmq probes wholesale when they are set
template: templates/statefulset-rabbitmq.yaml
set:
rabbitmq:
livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- check_running
periodSeconds: 20
readinessProbe:
httpGet:
path: /api/health/checks/alarms
port: 15672
asserts:
- equal:
path: spec.template.spec.containers[0].livenessProbe
value:
exec:
command:
- rabbitmq-diagnostics
- -q
- check_running
periodSeconds: 20
- equal:
path: spec.template.spec.containers[0].readinessProbe
value:
httpGet:
path: /api/health/checks/alarms
port: 15672

- it: renders the rabbitmq container securityContext next to the probes
template: templates/statefulset-rabbitmq.yaml
set:
rabbitmq:
securityContext:
runAsNonRoot: true
asserts:
- equal:
path: spec.template.spec.containers[0].securityContext.runAsNonRoot
value: true
- exists:
path: spec.template.spec.containers[0].readinessProbe

- it: derives the management hostname from the first domain
template: templates/ingress-rabbitmq.yaml
set:
Expand Down
3 changes: 3 additions & 0 deletions charts/shopsys-infra/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,9 @@ rabbitmq:
extraEnv: []
extraVolumes: []
extraVolumeMounts: []
# Empty = the chart defaults render (liveness: `rabbitmq-diagnostics -q ping`,
# readiness: TCP check on the AMQP port 5672). Setting a probe REPLACES the default
# wholesale - there is no deep merge, so an override must be complete.
livenessProbe: {}
readinessProbe: {}
priorityClassName: ""
Expand Down
11 changes: 11 additions & 0 deletions docs/migrating-from-shopsys-deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,3 +124,14 @@ Intentional differences of the phase-1 rewrite; everything else is a 1:1 port.
env vars (works with any registry — GCR/GAR via username `_json_key` and the service
account JSON as the password); the GitLab-flavored `CI_REGISTRY`/`DEPLOY_REGISTER_*`
variables keep working as a fallback.
21. **RabbitMQ has default probes** (legacy declared none, so the pod counted as Ready the
moment the container process started): `readinessProbe` is a TCP check on the AMQP port
5672 — the upstream Kubernetes Operator's default and the practice recommended by the
[RabbitMQ docs](https://www.rabbitmq.com/docs/monitoring#health-checks), because the AMQP
listener opens as one of the last boot steps — and `livenessProbe` is the stage-1
`rabbitmq-diagnostics -q ping` with a deliberately generous budget (60 s initial delay,
3 × 30 s). This makes the infra release's `wait: true` mean "the broker accepts
connections" — the premise the migration hook's ordering relies on. Both are overridable
wholesale via `rabbitmq.livenessProbe` / `rabbitmq.readinessProbe` (set = replace, no
deep merge). Consequence: a broker that never finishes booting now fails the deploy
instead of letting the migration run against an unreachable broker.
2 changes: 1 addition & 1 deletion docs/values.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Every workload component (`webserver`, `storefront`, `cron`, `consumers.defaults
| `nodeSelector` / `tolerations` / `affinity` / `priorityClassName` | scheduling |
| `extraEnv` | extra env entries (raw list, supports `valueFrom`) |
| `extraVolumes` / `extraVolumeMounts` | additional volumes |
| `livenessProbe` / `readinessProbe` | probe overrides |
| `livenessProbe` / `readinessProbe` | probe overrides; empty = the chart default where one exists (`redis`, `rabbitmq`), and setting one REPLACES the default wholesale (no deep merge) |
| `securityContext` / `podSecurityContext` | security contexts |
| `terminationGracePeriodSeconds`, `lifecycle` | shutdown behavior |

Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/basic-production/expected/continuous.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: 20m

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/basic-production/expected/first-deploy-with-demo-data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: 20m

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/basic-production/expected/first-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: 20m

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/development-single-domain/expected/continuous.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: "0.01"

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/development-single-domain/expected/first-deploy-with-demo-data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: "0.01"

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/development-single-domain/expected/first-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: "0.01"

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/development-with-cloudflare/expected/continuous.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: "0.01"

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/development-with-cloudflare/expected/first-deploy-with-demo-data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: "0.01"

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
18 changes: 17 additions & 1 deletion tests/golden/scenarios/development-with-cloudflare/expected/first-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,23 @@ spec:
resources:
requests:
cpu: "0.01"

livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- -q
- ping
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 5672
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
Expand Down
Loading
Loading