[BUG] Wayland backend keeps allocating GPU memory after stop_capture()

[lonk42]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I'm running linuxserver/webtop:fedora-kde-e46b460a-ls291 with the Wayland backend, Leaving certain GPU enabled processes open I get GPU memory for the process keeps growing after stop_capture(), for as long as clients on the nested compositor go on creating and destroying buffers. It is never released while capture is stopped. The next start_capture() releases all of it at once.

It looks like device memory only. Across 1158 samples of one climb (167 -> 5747 MiB), fds to /dev/nvidia* stayed at 25, fds to /dev/dri/* stayed at 4, and RSS was byte-identical throughout.

I may have this wrong, but reading the source it looks like the deferred-destruction queue never gets drained while capture is stopped:

• The calloop render tick early-outs at src/lib.rs:2390 while !is_capturing, so no Frame is finished.
• DmabufHandler::dmabuf_imported still imports client buffers, since it is protocol-driven rather than tick-driven.
• Smithay only issues glDeleteTextures / eglDestroyImageKHR inside GlesRenderer::cleanup(), whose only callers are finishing a Frame and Renderer::cleanup_texture_cache(). I couldn't find a call to the latter anywhere in the crate, and Smithay appears to call it automatically only from drm::compositor.

If that's right, calling cleanup_texture_cache() in the early-out branch might be enough, possibly throttled to every Nth tick since the tick is 16 ms and the drain looks idempotent.

This may be the same underlying thing as Smithay/smithay#1747, which is open, and which cosmic-comp seems to have worked around in pop-os/cosmic-comp#1442.

Expected Behavior

GPU memory released when the objects holding it are dropped, rather than only on the next start_capture().

Steps To Reproduce

Using example/screen_to_browser.py with PIXELFLUX_WAYLAND=true and a GPU:

1. Run the example and connect a browser, so start_capture() runs.
2. Open several application windows on the nested compositor.
3. Disconnect the browser, so stop_capture() runs.
4. Start any GL client (glxgears), leave it 60 s, then kill it.
5. Watch the process in nvidia-smi — the type G row. Memory rises and stays up for as long as I've waited.
6. Reconnect, and it drops back to baseline.

One thing that might help confirm the diagnosis without a patched build: with PIXELFLUX_CU set, a screenshot sets pending_screenshot, which is the other half of the early-out condition, so it forces a single render tick. With capture stopped and no resolution change:

baseline (capture stopped, idle):  107 MiB
after starting a GL client:        149 MiB
after killing it:                  189 MiB
after 90 s idle:                   189 MiB
POST {"action":"screenshot"}       http 200
after screenshot:                    3 MiB

I haven't found anything else that releases it without restarting capture, which is partly why I suspect the cleanup path.

Environment

• pixelflux 2.0.0, smithay pinned ca932e042fa9ad150605c150a86275b85f9ad5b3
• Python 3.14, Fedora 44, kernel 7.1.3
• Backend: Wayland (PIXELFLUX_WAYLAND=true), kwin_wayland --xwayland nested, NVENC, zero-copy path active
• GPU: RTX 3070 (8 GB), driver 610.43.03. Also seen on an RTX 2070 SUPER, driver 610.43.02.
• Found via selkies, which embeds pixelflux, so the capture start/stop above is driven by browser clients connecting and disconnecting rather than by direct API calls.