docs: Document jobs-funneled-to-one-worker fix (SDK 1.10.1)#698
Conversation
![promptless[bot]](https://avatars.githubusercontent.com/in/979718?s=60&v=4){kind=small}
|
|
||
| If your endpoint has multiple workers but nearly all jobs run on one worker while the others sit idle, and jobs stay `IN_QUEUE` even though workers are available, you may be running an affected version of the Runpod Python SDK. | ||
|
|
||
| Versions 1.9.1 through 1.10.0 could corrupt per-worker job tracking on endpoints that use a network volume, causing most workers to stop pulling new jobs. This most often appeared on network volume endpoints such as ComfyUI workers. |
There was a problem hiding this comment.
Sourced the affected version range (1.9.1–1.10.0), the network-volume root cause, and the 1.10.1 fix from PR #517, which reverted JobsProgress in worker_state.py from a shared pickle file at os.getcwd()/.runpod_jobs.pkl back to an in-memory set. Network volumes made os.getcwd() a shared mount across workers, corrupting per-worker occupancy so workers stopped pulling jobs — the ComfyUI-on-network-volume symptom in the linked issue #432.
Source: runpod/runpod-python#517
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
Just a reminder: If you'd like me to act on any feedback you have via Github comments, just type @Promptless in your suggestion and I'll get right on it! (I won't show up in the user dropdown, but I'll process any request that has @Promptless in the comment body.) |
lavanya-gunreddi
deleted the
promptless/serverless-jobs-funneled-single-worker
branch
1 participant
Open this suggestion in Promptless to view citations and reasoning process
Adds a Serverless troubleshooting entry for the SDK job-funneling regression: on SDK 1.9.1–1.10.0, endpoints with a network volume corrupted per-worker job tracking, funneling nearly all jobs onto one worker while others sat idle and jobs stayed IN_QUEUE. Points users to upgrade to runpod>=1.10.1 (fixed via runpod-python #517, released in 1.10.1). Common for ComfyUI-on-network-volume endpoints.
Trigger Events
Tip: Tell your friends working on non-commercial open-source projects to apply for free Promptless access at promptless.ai/oss ❤️