Observed behaviour
scripts/warmup waits indefinitely for each WARMUP_ENABLED=true container to report a healthy state:
until [[ "$(docker inspect -f '{{.State.Health.Status}}' "$container")" == "healthy" ]]; do
sleep 2
done
There is no timeout or other stopping condition. If one service never becomes healthy, warmup never progresses past that service. Because imageup runs warmup synchronously, imageup also does not complete.
For services that do become healthy, the script then waits a fixed 10 seconds plus 0–4 seconds of random jitter before invoking /warmup. These waits occur for each applicable service and can make routine imageup runs unnecessarily slow, particularly during development.
Expected behaviour
A service that fails to become healthy should not be able to block the warmup and imageup workflows indefinitely.
Warmup should have a bounded strategy for waiting for service readiness and should clearly report services that could not be warmed.
Successful warmup runs should also avoid unnecessary fixed waiting once a service is sufficiently ready for its warmup request.
Reproduction
- Start the IMAGE stack with a service that has
WARMUP_ENABLED=true but never reaches a healthy state.
- Run
scripts/warmup, or run imageup and allow it to reach the warmup stage.
- Observe that the script repeatedly checks the service health with no timeout and never progresses.
- With normally healthy warmup-enabled services, observe that each service waits an additional 10–14 seconds after becoming healthy before its
/warmup request is sent.
Runtime context
- Script:
scripts/warmup
- Calling workflow:
scripts/imageup
- Applies to any deployment containing a
WARMUP_ENABLED=true service.
Additional context
The current warmup process:
- discovers running containers on the
image network;
- selects containers with
WARMUP_ENABLED=true;
- waits without a timeout for each selected container to become healthy;
- waits another 10 seconds plus 0–4 seconds of jitter after health is reported;
- then invokes the service's
/warmup endpoint.
The current multilang-support failure on the development server exposed the unbounded-wait behaviour; #1214 works around that immediate service-specific problem by disabling multilang-support in the test deployment, but does not address the general failure mode in scripts/warmup.
Possible implementation details should be determined during the fix, but the result should satisfy the following, at the least:
- A service that never becomes healthy cannot make
warmup wait indefinitely.
- The affected service and timeout/failure are clearly reported.
- Other eligible services can still be handled appropriately rather than silently becoming unreachable because an earlier service stalled.
- The current fixed post-health waiting is reduced or replaced where it is not needed for service readiness.
imageup reaches a bounded outcome when a warmup-enabled service is unhealthy.
Checks
Observed behaviour
scripts/warmupwaits indefinitely for eachWARMUP_ENABLED=truecontainer to report a healthy state:There is no timeout or other stopping condition. If one service never becomes healthy,
warmupnever progresses past that service. Becauseimageuprunswarmupsynchronously,imageupalso does not complete.For services that do become healthy, the script then waits a fixed 10 seconds plus 0–4 seconds of random jitter before invoking
/warmup. These waits occur for each applicable service and can make routineimageupruns unnecessarily slow, particularly during development.Expected behaviour
A service that fails to become healthy should not be able to block the warmup and
imageupworkflows indefinitely.Warmup should have a bounded strategy for waiting for service readiness and should clearly report services that could not be warmed.
Successful warmup runs should also avoid unnecessary fixed waiting once a service is sufficiently ready for its warmup request.
Reproduction
WARMUP_ENABLED=truebut never reaches a healthy state.scripts/warmup, or runimageupand allow it to reach the warmup stage./warmuprequest is sent.Runtime context
scripts/warmupscripts/imageupWARMUP_ENABLED=trueservice.Additional context
The current warmup process:
imagenetwork;WARMUP_ENABLED=true;/warmupendpoint.The current
multilang-supportfailure on the development server exposed the unbounded-wait behaviour; #1214 works around that immediate service-specific problem by disablingmultilang-supportin the test deployment, but does not address the general failure mode inscripts/warmup.Possible implementation details should be determined during the fix, but the result should satisfy the following, at the least:
warmupwait indefinitely.imageupreaches a bounded outcome when a warmup-enabled service is unhealthy.Checks