Skip to content

fix: deliver events under gevent monkey-patching with an SDK-owned lane queue - #866

Closed
ThaddKara wants to merge 5 commits into
PostHog:mainfrom
ThaddKara:fix/gevent-queue-compat
Closed

fix: deliver events under gevent monkey-patching with an SDK-owned lane queue#866
ThaddKara wants to merge 5 commits into
PostHog:mainfrom
ThaddKara:fix/gevent-queue-compat

Conversation

@ThaddKara

@ThaddKara ThaddKara commented Aug 11, 2026

Copy link
Copy Markdown

💡 Motivation and Context

Fixes #865.

gevent's monkey.patch_all() replaces queue.Queue with an implementation that lacks the private synchronization attributes CPython's pure-Python queue exposes: mutex, not_empty, not_full, all_tasks_done, unfinished_tasks, _qsize(), _get(). The consumer loop (_DrainSignal) and Client.flush() synchronize on exactly those attributes, so in a gunicorn --worker-class gevent worker:

  • the consumer thread dies immediately with AttributeError: 'gevent._gevent_cqueue.Queue' object has no attribute 'not_empty',
  • flush() raises error flushing queue: 'gevent._gevent_cqueue.Queue' object has no attribute 'all_tasks_done',
  • and every captured event buffers forever and is silently dropped — capture() succeeds, nothing ships. In our production deployment this produced a multi-day, alert-free server-side telemetry blackout.

The fix: lanes now use LaneQueue (posthog/_queue.py), the pure-Python CPython queue implementation carried by the SDK itself. It builds only on threading primitives, which gevent patches compatibly, so behavior is identical on stock CPython and under monkey-patching — and its private surface can't be swapped out from under the SDK. _DrainSignal, flush(), and rebuild_after_fork() work unchanged; no call sites change beyond the two queue constructions.

💚 How did you test it?

  • New posthog/test/test_gevent_compat.py:
    • LaneQueue interface + semantics tests (the private-attribute list is asserted as a contract).
    • Client capture→flush round-trip asserting the lane queue is SDK-owned.
    • A subprocess test that mirrors a real gevent worker (monkey.patch_all() first, import the SDK after), asserting events are delivered and the queue drains. On the unfixed code this test reproduces the exact production failure, including the error flushing queue log line and zero deliveries.
  • gevent added to the test extra (CPython-only marker) so CI exercises the subprocess test; it skips cleanly where gevent is unavailable.
  • Full test_consumer.py + test_client.py suites pass (250 tests), ruff format --check / ruff check clean on the locked ruff (0.12.2), python -W error -c 'import posthog' clean.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file (hand-written in the repo format: .sampo/changesets/gevent-queue-compat.md)

🤖 Generated with Claude Code

…ne queue

gevent's monkey.patch_all() replaces queue.Queue with an implementation
that lacks the private synchronization attributes CPython exposes (mutex,
not_empty, not_full, all_tasks_done, unfinished_tasks, _qsize, _get). The
consumer loop and Client.flush() synchronize on those attributes, so in a
gevent gunicorn worker the consumer thread dies with AttributeError and
every captured event buffers forever and is silently dropped.

Give lanes SdkQueue, the pure-Python CPython queue implementation carried
by the SDK itself. It builds only on threading primitives, which gevent
patches compatibly, so behavior is identical on stock CPython and under
monkey-patching - and its private surface cannot be swapped out from
under the SDK.

The regression test runs a real gevent worker scenario in a subprocess
(patch first, import the SDK after); on the previous code it reproduces
the exact production failure, including the 'error flushing queue' log.

Fixes PostHog#865

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ThaddKara
ThaddKara requested a review from a team as a code owner August 11, 2026 18:18
ThaddKara and others added 4 commits August 11, 2026 11:30
CPython's queue.Queue stores its deque as .queue; matching the name keeps
SdkQueue attribute-compatible for callers that reach into that internal
via the backwards-compatible Client.queue property.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Role-based name matching _Lane, which is the only consumer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…se, test rigor

- regenerate references/public_api_snapshot.txt for the Client.queue
  annotation change (public_api_check now passes)
- implement Python 3.13 shutdown()/ShutDown on LaneQueue, matching
  CPython semantics; document the deliberate isinstance non-goal and
  flag the concrete-type change in the changeset
- retain the PSF-2.0 license text (LICENSE-PSF-2.0.txt) for the
  CPython-derived queue implementation; ships via the default
  LICEN[CS]E* packaging glob
- run the consumer/_DrainSignal suite against LaneQueue, the queue
  production delivery rides on, instead of stdlib Queue
- raise the gevent test floor to 25.4.1 (first version whose
  monkey.patch_all() replaces queue.Queue) and assert that premise
  inside the subprocess regression test
- join() clients in the new tests so no consumer thread outlives the
  mock.patch scope
- redo uv.lock with current uv: purely additive now (gevent + deps),
  exclude-newer sentinel preserved, no unrelated churn
- use a relative import for LaneQueue in client.py per AGENTS.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ThaddKara

Copy link
Copy Markdown
Author

Follow-up commit after an internal review pass (68c3b5f):

  • public_api_snapshot regenerated for the Client.queue annotation change — make public_api_check passes.
  • Python 3.13 shutdown()/ShutDown implemented on LaneQueue with CPython semantics, so the backwards-compat Client.queue surface keeps 3.13 parity. isinstance(client.queue, queue.Queue) remains deliberately False — inheriting would re-import the bug under monkey-patching (base class would resolve to gevent's Queue), and queue.Queue isn't an ABC; documented in the module docstring and flagged in the changeset.
  • PSF-2.0 license text retained (LICENSE-PSF-2.0.txt) for the CPython-derived implementation; ships via the default LICEN[CS]E* glob. Happy to fold it into whatever third-party-license convention you prefer.
  • Consumer/_DrainSignal suite now runs against LaneQueue (one-line alias in test_consumer.py) so the production queue class is what the delivery tests exercise.
  • gevent test floor raised to 25.4.1 — the first version whose monkey.patch_all() replaces queue.Queue — and the subprocess test now asserts that premise, so the regression test can't pass vacuously.
  • Test hygiene: clients are join()ed so no consumer thread outlives the tests.
  • uv.lock redone with current uv: purely additive (+123/-0, gevent + deps only), exclude-newer sentinel preserved.
  • Relative import per AGENTS.md.

@marandaneto

Copy link
Copy Markdown
Member

superseeded by #867
thanks @ThaddKara

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants