cuda.core: defer graph user-object payload cleanup

[Andy-Jost]

Summary

Move graph attachment payload destruction off CUDA's internal user-object callback thread. The callback now performs only an intrusive queue handoff and one coalesced Py_AddPendingCall; resource handles and Python owners are released later from Python's main thread, where their deleters may safely call CUDA.

Changes

• Add a process-lifetime lock-free deferred-cleanup queue with one pending-call wakeup regardless of payload count.
• Preserve queued payloads when pending-call scheduling fails and retry from later safe graph-attachment entries.
• Stop cleanup during interpreter finalization and intentionally leak intact payloads when safe cleanup is no longer possible.
• Initialize shutdown state explicitly during module import and document the callback/threading contract.
• Add stress coverage for more than 32 callbacks, pending-call queue saturation, main-thread finalization, and interpreter shutdown.

Related Work

• Part of cuda.core: retain graph node attachments as per-node CUDA user objects #2350.
• Prerequisite for cuda.core: retain graph attachments with per-node user objects #2357.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[Andy-Jost]

PR #2371 separates “CUDA says this payload is no longer needed” from actually cleaning it up.

Parts

1. DeferredCleanupItem

A small queue item allocated before CUDA sees the user object. It contains:

• The payload, currently GraphSlotTable*
• The function that cleans up that payload
• A next pointer for the intrusive queue

The process-wide cleanup queue pointer is published atomically once during module initialization and loaded by the CUDA callback; it is not repeated in each item. Because the item is preallocated, CUDA’s callback does not allocate memory.

2. DeferredCleanupQueue

A process-lifetime object containing:

• head_: atomic head of an intrusive multiple-producer, single-consumer queue
• scheduled_: whether a Python pending call is already scheduled
• accepting_: disabled during Python finalization

It is intentionally never freed, so a late CUDA callback can never access destroyed queue state.

3. CUDA callback

CUDA invokes enqueue_cleanup() when the user-object reference count reaches zero.

The callback only:

1. Atomically enqueues its DeferredCleanupItem.
2. Attempts to schedule one pending call.

It does not release a handle, call CUDA, allocate, or wait for cleanup.

4. Pending-call scheduling

schedule() atomically changes scheduled_ from false to true. Therefore, many CUDA callbacks still produce at most one outstanding Py_AddPendingCall.

If CPython’s pending-call queue is full:

• Scheduling returns -1.
• scheduled_ returns to false.
• The payload remains intact in cuda-core’s cleanup queue.
• A later enqueue or safe graph-attachment entry retries scheduling.

Retrying schedules cleanup; it does not enqueue the item again.

5. Main-thread drain

The pending callback runs on Python’s main thread. It atomically exchanges head_ with null, taking exclusive ownership of the entire queued list.

It then invokes each item’s cleanup function and deletes the item. Any resulting Python finalizer or CUDA-backed handle deleter runs from this permitted host context.

Items arriving during a drain are either picked up by its next loop iteration or cause another pending call to be scheduled after scheduled_ is cleared.

6. Shutdown

A Py_AtExit callback sets accepting_ false. The drain also checks whether Python is finalizing.

After that point:

• CUDA callbacks may still enqueue safely because the cleanup queue lives for the process lifetime.
• No new Python callback is scheduled.
• Queued payloads are intentionally leaked rather than risking Python/CUDA use during shutdown.

Why cleanup happens once

Each payload follows one of two mutually exclusive paths:

CUDA user-object creation fails:
    item allocated → cleanup executed directly

CUDA user-object creation succeeds:
    CUDA-owned → callback → enqueued → atomically detached → cleanup executed

After successful cuUserObjectCreate, cuda-core never executes the item’s cleanup directly. CUDA invokes its destructor callback once, when the user object’s total reference count reaches zero.

Within the cleanup queue:

• The CUDA callback enqueues that item once.
• Scheduling retries never enqueue it again.
• head_.exchange(nullptr) transfers each queued item to the single consumer exactly once.
• The cleanup function is noexcept, so processing cannot abandon an item halfway through.
• The item is deleted immediately after its payload is cleaned up.

Thus safety relies on CUDA’s user-object contract providing one final destructor invocation. The cleanup queue then preserves that exactly-once transition through atomic ownership transfer; it does not need a separate “already cleaned” flag.

[mdboom]

My initial reaction was that this might be more complex than it needs to be. Py_AddPendingCall already functions as a queue (you can call it multiple times and it will run all of the calls in order the next chance it calls any of them), so it might not be necessary to also have a queue on our side.

/However/ Py_AddPendingCall is a FIFO queue, with a maximum of 300 elements (and it appears to silently evict elements if it's more than 300). All that is "probably fine", but it's an implementation detail that could change.

So I think what's here is preferable -- we can be in charge of our own destiny on the queueing. (Say, for example, we discover that the ordering needs to change, this gives us a place to do so).

My other comments are just around improving comments and docs.

[mdboom]

This is thread-safe only as long as the GIL is held -- it is in practice (also it's run once at import under normal conditions) but maybe just add a comment.

[mdboom]

Maybe add a little context about why retrying would be necessary? I don't think it's necessarily unsound, just could use a little motivating statement.

[Andy-Jost]

Thanks for the review, @mdboom. I'll wrap the touch-ups into the next PR (#2357).

As for the complexity, I largely agree. My initial idea was very similar to yours: simply make a call to Py_AddPendingCall from CUDA's destructor thread for each user object. That ran into a few snags:

• Py_AddPendingCall only accepts 32 queued callbacks. I had my agent follow up after you mentioned 300 entries and it seems to stand by that number (details below).
• When Py_AddPendingCall fails for any reason on the CUDA thread, we're totally stuck without any recourse other than to leak that object. Introducing a queue we manage where insertion never fails fixes that.

So I agree, the complexity is higher than anticipated, but I don't think it can be reduced.

Queue limit details

My “32” was substantively right for Py_AddPendingCall; Mike found the newer 300-entry backing array but not the effective limit of this public API. Exact answer:

• Python 3.10–3.12: array size 32, but the ring buffer reserves one slot, so only 31 callbacks fit.
• Python 3.13–3.15, including free-threaded builds: backing array size is 300, but Py_AddPendingCall targets pending_mainthread, whose configured maximum remains 32.
• Full queues do not evict callbacks. Py_AddPendingCall returns -1.

Source findings
• Python 3.10 defines NPENDINGCALLS 32 (https://github.com/python/cpython/blob/3.10/Include/internal/pycore_interp.h) and its push function returns -1 when advancing would meet first (https://github.com/python/cpython/blob/3.10/Python/ceval.c#L584-L598). That leaves 31 usable entries.
• Python 3.13 introduces PENDINGCALLSARRAYSIZE 300, but also explicitly defines MAXPENDINGCALLS_MAIN 32 (https://github.com/python/cpython/blob/3.13/Include/internal/pycore_ceval_state.h).
• The runtime initializes pending_mainthread.max from that 32-entry limit in pycore_runtime_init.h (https://github.com/python/cpython/blob/3.13/Include/internal/pycore_runtime_init.h).
• Py_AddPendingCall (https://github.com/python/cpython/blob/3.13/Python/ceval_gil.c#L805-L819) explicitly selects _Py_PENDING_MAINTHREADONLY; when that queue is full, it returns -1.
• Python 3.15 retains MAXPENDINGCALLS_MAIN 32 (https://github.com/python/cpython/blob/3.15/Include/internal/pycore_ceval_state.h).
• The public documentation (https://docs.python.org/3.14/c-api/threads.html#c.Py_AddPendingCall) promises -1 on failure but intentionally does not document a capacity.

[github-actions]

Doc Preview CI
Preview removed because the pull request was closed or merged.