Skip to content

Warmup can block imageup indefinitely and adds unnecessary service delays #1216

Description

@aidanbruneel

Observed behaviour

scripts/warmup waits indefinitely for each WARMUP_ENABLED=true container to report a healthy state:

until [[ "$(docker inspect -f '{{.State.Health.Status}}' "$container")" == "healthy" ]]; do
    sleep 2
done

There is no timeout or other stopping condition. If one service never becomes healthy, warmup never progresses past that service. Because imageup runs warmup synchronously, imageup also does not complete.

For services that do become healthy, the script then waits a fixed 10 seconds plus 0–4 seconds of random jitter before invoking /warmup. These waits occur for each applicable service and can make routine imageup runs unnecessarily slow, particularly during development.

Expected behaviour

A service that fails to become healthy should not be able to block the warmup and imageup workflows indefinitely.

Warmup should have a bounded strategy for waiting for service readiness and should clearly report services that could not be warmed.

Successful warmup runs should also avoid unnecessary fixed waiting once a service is sufficiently ready for its warmup request.

Reproduction

  1. Start the IMAGE stack with a service that has WARMUP_ENABLED=true but never reaches a healthy state.
  2. Run scripts/warmup, or run imageup and allow it to reach the warmup stage.
  3. Observe that the script repeatedly checks the service health with no timeout and never progresses.
  4. With normally healthy warmup-enabled services, observe that each service waits an additional 10–14 seconds after becoming healthy before its /warmup request is sent.

Runtime context

  • Script: scripts/warmup
  • Calling workflow: scripts/imageup
  • Applies to any deployment containing a WARMUP_ENABLED=true service.

Additional context

The current warmup process:

  • discovers running containers on the image network;
  • selects containers with WARMUP_ENABLED=true;
  • waits without a timeout for each selected container to become healthy;
  • waits another 10 seconds plus 0–4 seconds of jitter after health is reported;
  • then invokes the service's /warmup endpoint.

The current multilang-support failure on the development server exposed the unbounded-wait behaviour; #1214 works around that immediate service-specific problem by disabling multilang-support in the test deployment, but does not address the general failure mode in scripts/warmup.

Possible implementation details should be determined during the fix, but the result should satisfy the following, at the least:

  • A service that never becomes healthy cannot make warmup wait indefinitely.
  • The affected service and timeout/failure are clearly reported.
  • Other eligible services can still be handled appropriately rather than silently becoming unreachable because an earlier service stalled.
  • The current fixed post-health waiting is reduced or replaced where it is not needed for service readiness.
  • imageup reaches a bounded outcome when a warmup-enabled service is unhealthy.

Checks

  • I searched existing issues and kept this report focused on one independently fixable bug.
  • I removed secrets, credentials, personal information, and confidential data.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: deploymentAffects deployment, promotion, rollback, production startup, or server operation.

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions