Skip to content

fix(http_server): register disabled endpoint synchronously to avoid stuck 503 on restart - #470

Open
masnnuller wants to merge 1 commit into
redpanda-data:mainfrom
masnnuller:fix/http-server-softstop-503-race
Open

fix(http_server): register disabled endpoint synchronously to avoid stuck 503 on restart#470
masnnuller wants to merge 1 commit into
redpanda-data:mainfrom
masnnuller:fix/http-server-softstop-503-race

Conversation

@masnnuller

Copy link
Copy Markdown

Problem

Fixes #469.

When an http_server input runs on the service-wide HTTP server (no dedicated address) and its stream is restarted/updated in stream mode (streams manager Update = Delete + Create), the endpoint can very rarely get stuck returning 503 Service Unavailable forever and never recover.

loop()'s teardown registered a disabled 503 "Endpoint disabled." handler from an asynchronous goroutine that waited on the same HasStopped signal as WaitForClose. TriggerHasStopped() released both concurrently, so the 503 registration raced with the new instance's normal-handler registration on restart. Since RegisterEndpoint is last-writer-wins per path, the 503 handler could occasionally win, leaving the endpoint permanently stuck at 503.

Fix

Register the disabled handler synchronously, after handlerWG.Wait() drains in-flight requests and before TriggerHasStopped(). This guarantees the ordering:

old input: drain -> register 503 -> TriggerHasStopped
        -> WaitForClose returns -> Stop -> Delete -> Create
        -> new input registers normal handler   (always wins)

so the new instance's registration always happens last and the endpoint recovers.

The separate goroutine is no longer needed for hard-stop: in-flight handlers already observe HardStopChan() and unblock handlerWG.Wait(), and new requests still receive 503 from the existing handler's soft-stop check during draining, so behavior is preserved for the full-shutdown case.

Same fix applied to the wasm build (input_http_server_wasm.go).

output_http_server is not affected: it does not re-register a disabled 503 handler on stop.

Testing

  • Added TestHTTPServerSharedServerRestartRecovers, which repeatedly creates/uses/stops an http_server input on a shared service-wide server and asserts the endpoint recovers (200) each time. It uses the real api.Type registry so RegisterEndpoint overwrite semantics match production.
  • Verified the new test fails on the pre-fix code (times out on a stuck 503) and passes with the fix.
  • go build (default + GOOS=js GOARCH=wasm), go vet, and the full internal/impl/io/ suite pass with -race.

…tuck 503

When an http_server input uses the service-wide HTTP server (no dedicated
address), loop()'s teardown registered a disabled 503 "Endpoint disabled."
handler from an asynchronous goroutine that waited on the same HasStopped
signal as WaitForClose. TriggerHasStopped released both concurrently, so the
503 registration raced with the new instance's normal-handler registration on
restart (streams manager Update = Delete + Create). Since RegisterEndpoint is
last-writer-wins per path, the 503 handler could occasionally win, leaving the
endpoint permanently returning 503.

Register the disabled handler synchronously, after handlerWG.Wait() drains
in-flight requests and before TriggerHasStopped(). This guarantees the new
instance's registration always happens after ours and wins, so the endpoint
recovers. Hard-stop no longer needs a separate goroutine because in-flight
handlers already observe HardStopChan and unblock handlerWG.Wait(), and new
requests still receive 503 from the existing handler's soft-stop check during
draining.

Applies the same fix to the wasm build. Adds a regression test exercising
repeated create/use/stop cycles on a shared service-wide server.

Fixes redpanda-data#469

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

http_server input can get permanently stuck returning 503 after a stream restart (stream mode)

2 participants