fix: deliver events under gevent monkey-patching with an SDK-owned lane queue - #866
Closed
ThaddKara wants to merge 5 commits into
Closed
fix: deliver events under gevent monkey-patching with an SDK-owned lane queue#866ThaddKara wants to merge 5 commits into
ThaddKara wants to merge 5 commits into
Conversation
…ne queue gevent's monkey.patch_all() replaces queue.Queue with an implementation that lacks the private synchronization attributes CPython exposes (mutex, not_empty, not_full, all_tasks_done, unfinished_tasks, _qsize, _get). The consumer loop and Client.flush() synchronize on those attributes, so in a gevent gunicorn worker the consumer thread dies with AttributeError and every captured event buffers forever and is silently dropped. Give lanes SdkQueue, the pure-Python CPython queue implementation carried by the SDK itself. It builds only on threading primitives, which gevent patches compatibly, so behavior is identical on stock CPython and under monkey-patching - and its private surface cannot be swapped out from under the SDK. The regression test runs a real gevent worker scenario in a subprocess (patch first, import the SDK after); on the previous code it reproduces the exact production failure, including the 'error flushing queue' log. Fixes PostHog#865 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CPython's queue.Queue stores its deque as .queue; matching the name keeps SdkQueue attribute-compatible for callers that reach into that internal via the backwards-compatible Client.queue property. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Role-based name matching _Lane, which is the only consumer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…se, test rigor - regenerate references/public_api_snapshot.txt for the Client.queue annotation change (public_api_check now passes) - implement Python 3.13 shutdown()/ShutDown on LaneQueue, matching CPython semantics; document the deliberate isinstance non-goal and flag the concrete-type change in the changeset - retain the PSF-2.0 license text (LICENSE-PSF-2.0.txt) for the CPython-derived queue implementation; ships via the default LICEN[CS]E* packaging glob - run the consumer/_DrainSignal suite against LaneQueue, the queue production delivery rides on, instead of stdlib Queue - raise the gevent test floor to 25.4.1 (first version whose monkey.patch_all() replaces queue.Queue) and assert that premise inside the subprocess regression test - join() clients in the new tests so no consumer thread outlives the mock.patch scope - redo uv.lock with current uv: purely additive now (gevent + deps), exclude-newer sentinel preserved, no unrelated churn - use a relative import for LaneQueue in client.py per AGENTS.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Follow-up commit after an internal review pass (68c3b5f):
|
Member
|
superseeded by #867 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
Fixes #865.
gevent's
monkey.patch_all()replacesqueue.Queuewith an implementation that lacks the private synchronization attributes CPython's pure-Python queue exposes:mutex,not_empty,not_full,all_tasks_done,unfinished_tasks,_qsize(),_get(). The consumer loop (_DrainSignal) andClient.flush()synchronize on exactly those attributes, so in a gunicorn--worker-class geventworker:AttributeError: 'gevent._gevent_cqueue.Queue' object has no attribute 'not_empty',flush()raiseserror flushing queue: 'gevent._gevent_cqueue.Queue' object has no attribute 'all_tasks_done',capture()succeeds, nothing ships. In our production deployment this produced a multi-day, alert-free server-side telemetry blackout.The fix: lanes now use
LaneQueue(posthog/_queue.py), the pure-Python CPython queue implementation carried by the SDK itself. It builds only onthreadingprimitives, which gevent patches compatibly, so behavior is identical on stock CPython and under monkey-patching — and its private surface can't be swapped out from under the SDK._DrainSignal,flush(), andrebuild_after_fork()work unchanged; no call sites change beyond the two queue constructions.💚 How did you test it?
posthog/test/test_gevent_compat.py:LaneQueueinterface + semantics tests (the private-attribute list is asserted as a contract).monkey.patch_all()first, import the SDK after), asserting events are delivered and the queue drains. On the unfixed code this test reproduces the exact production failure, including theerror flushing queuelog line and zero deliveries.geventadded to thetestextra (CPython-only marker) so CI exercises the subprocess test; it skips cleanly where gevent is unavailable.test_consumer.py+test_client.pysuites pass (250 tests),ruff format --check/ruff checkclean on the locked ruff (0.12.2),python -W error -c 'import posthog'clean.📝 Checklist
If releasing new changes
sampo addto generate a changeset file (hand-written in the repo format:.sampo/changesets/gevent-queue-compat.md)🤖 Generated with Claude Code