Skip to content

Fix deadlock in Manager.Stop/StopReaders - #288

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
mainfrom
theop-dd/fix-race-stop-readers-waiting-reader
Sep 4, 2026
Merged

Fix deadlock in Manager.Stop/StopReaders#288
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
mainfrom
theop-dd/fix-race-stop-readers-waiting-reader

Conversation

@theop-dd

@theop-dd theop-dd commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Split stopReaders() into 3 phases:

  1. Close all reader file descriptors (under manager.stateLock) — signals goroutines to stop
  2. Temporarily release manager.stateLock — lets handler callbacks complete, goroutines exit
  3. Re-acquire manager.stateLock — clean up underlying maps
    stop() now sets m.state = reset before calling stopReaders() to prevent concurrent NewPerfRing/NewRingBuffer from adding readers during the unlock window.

Known limitations

  • PerfMap.Stop() / RingBuffer.Stop() still hold their own stateLock during wgReader.Wait(). This is fine for current callers (only used for cleanup when Start() fails, so no goroutine is running). A warning comment documents the risk.
  • StopReaders() does not set manager state during the unlock window, so a concurrent NewPerfRing call could theoretically slip through. This is documented as a usage error.

Motivation

Manager.Stop() and Manager.StopReaders() can deadlock when a handler callback (DataHandler, RecordHandler, etc.) calls Manager methods like GetMap() or GetProbe().
The deadlock occurs because:

  1. Stop holds manager.stateLock (write lock) and waits for the reader goroutine to exit (wgReader.Wait())
  2. The reader goroutine's handler needs manager.stateLock.RLock() to call Manager methods
    Neither can make progress → deadlock.
    This is a race condition: it only triggers if Stop takes the lock while a handler is mid-execution. In production it's masked by SIGKILL at shutdown, but it surfaces in tests that stop and restart the manager in the same process.

This deadlock occurs when we call Stop on the manager. We never saw it in prod when the agent runs because if it happens, the process is eventually killed. This was however causing incidents on CI, where the hang caused every subsequent test to time out.

@theop-dd
theop-dd requested a review from a team as a code owner September 3, 2026 15:09

@brycekahle brycekahle left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this will fix StopReaders for now, but it appears any code which calls perf/ring Stop while holding the manager stateLock could continue to deadlock. I can explore how to fix that in a more permanent and less brittle way.

@theop-dd

theop-dd commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

exactly @brycekahle , that's why I put a comment on top of the function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants