feat: add H.264 hardware decoding

[DatCaptainHorse]

PR changes

Adds decoding support, scoped to H.264 for now so there's not too much to review at once.. though I feel it already is 😅

LLM's summary:

Adds a stream-driven H.264 decoder alongside the existing encoder, plus a
shared Vulkan Video layer (src/video.rs, src/decoder/codec.rs) used by both
directions.

- Full reference management: IDR, sliding-window, and MMCO (B-pyramid).
- Display-order output by default via an internal reorder buffer, with a
  decode-order opt-out for lowest latency; DecodeConfig gains OutputOrder
  and Decoder gains flush().
- Readback and reorder copies run on the transfer queue; fixes transfer
  queue-family selection to avoid the video engine.
- Encoder B-frame guards return errors instead of panicking.
- Rejects unsupported streams (interlaced, scaling matrices) with clear errors.

Verified byte-identical to ffmpeg across AMD (RADV) and NVIDIA, including
B-pyramid, mid-stream resolution changes, and encode->decode round trips.

Comment

Whew.. this has been atleast a few weeks worth of work, I ended up getting a claude subscription since I was losing my mind over H.264 codec parsing and dealing with all the reference frame management, it's just not fun and goes over my brain's capacity to handle 😓

This is very much a draft, meaning WIP, RFC and all. API could be better IMO, and I'd like some feedback and guidance on what you wish for @hgaiser !

Currently codec parsing and actual decode are done separately for API user:

for au in access_units(&stream) {
    let decoded = match decoder.decode(au, frame_count as u64) {
...

incase someone wants to do their own parsing or do some magic by intercepting parsed data. When adding AV1 decoding in future, the parsing API needs changing to be more codec-agnostic though.

I've verified it works though in a real use-case, here's a picture showing pixelforge decoding integrated into a native app for playing Nestri streams, H.264 is decoded and shown perfectly fine

It's a lot to unpack, despite trying to limit the scope to H.264 decoding for now, there's a lot of boilerplate and logic needed for even just that, I apologize for the size of this PR 😅

Certain code parts are somewhat oriented towards H.26X codecs, so when adding AV1 decoding in future, it will need adjusting to be more clean and sane.

Aside from decoder changes, the encoder pub fn new.. was poked a bit to change the "b-frames unsupported" assertion into an Err return instead, allowing to gracefully handle that. Though I can revert that one since it's overreaching a bit here.

[hgaiser]

I laughed when I read ... scoped to H.264 for now so there's not too much to review at once.. and looked at the amount of diff lines :p

Thanks for submitting it though, cool stuff. I had a rough implementation of decoding working a long time ago but decided to keep the scope relatively smaller by focusing only on encoding. That said, I'm completely open to adding decoding support. It will probably take some time to go over it.

For full transparency: I'll probably review the public API and use LLMs to review the internals.

[DatCaptainHorse]

Take your time @hgaiser - and feel free to be strict on demands/wants!

Also a random 🎉 - pixelforge decoding works under Windows as well :P
(excuse my cursed setup of running Windows in VM with GPU passthrough, connecting to it via Parsec.. yep)

[DatCaptainHorse]

@hgaiser Sorry for the bump, I'd just like to know when you believe you'll have the time and energy to look atleast into surface-level changes (i.e. the public/dev-facing API) 🤔

It doesn't have to be reviewed all at once, see what immediately "sticks out" or even irks you and I'll make the changes 🫡

I can also rebase against current main if it'd help, some conflicts seem to exist now.

[hgaiser]

No worries, fair question. I started looking into this yesterday, it's high on my to-do list :)

I'll try to get to it "soon" 🙄

[hgaiser]

I was wondering: would it make sense to have a similar asynchronous behavior here, comparable to the encode pipeline? It could look something like:

// Producer thread: feeds packets, never blocks on output
let producer = std::thread::spawn(move || {
    for au in access_units(&stream) {
        decoder.decode(au, pts).unwrap();   // just accepts the packet
    }
    decoder.flush().unwrap();               // signal end of stream
});

// Main/consumer thread: awaits decoded frames
loop {
    match decoder.download().await {
        Ok(Some(frame)) => render(frame),
        Ok(None)        => break,
        Err(e)          => return Err(e.into()),
    }
}

[DatCaptainHorse]

WHEW, okay I made sure to go through and pretty much refactor the PR to be more like the encoding-side.. thank heck for LLM help on this one 😅

Not to mention, the encode-API-like pipelining does give benefits, since I got no RTX 2060 anymore I temporarily rented a cloud server with L4 for testing and verifying, also found out that Intel Arc has Vulkan Video support, but behind sneaky ANV_DEBUG=video-decode,video-encode ENV var.

Results on decode speeds with pipelining API'fication:

GPU	320x240 display	320x240 decode	1080p display	1080p decode
AMD RX 9060 XT (RADV)	+19.0%	+6.3%	+57.1%	+14.8%
NVIDIA L4 (610.43.02)	+18.3%	+8.2%	+9.9%	+2.8%
Intel Arc A310 (ANV)	+90.7%	+185.5%	+42.6%	+20.6%

(Arc sees almost triple speeds, whew)

So basically, H.264-only since adding other codecs would bloat this way too much, but I've verified with all 3 major GPU brands and different hardware configs. Decode API is now more codec-agnostic as well to make future codec-support work less painful.

@hgaiser - it's a lot, but I tried my best to test various cases and hardware 👍

[hgaiser]

Those results look promising :o

Regarding the new changes, they look much better, and closer to what the encode pipeline currently does. I have my questions about the Decoder::split functionality though. I guess it came from the access_units(..) function in my example above? It seems with this API like you must have the entire buffer if you want to be able to split it correctly. Live decoding wouldn't work then, right?

I think we might need to split the decoder in a sink and a source, since a packet from the network could, in theory, lead to N frames (where N >= 0). I'm thinking something like:

// Split is optional, but necessary for multi threaded applications.
let (mut sink, mut source) = Decoder::new(context, DecodeConfig::h264())?.split();

// Producer (optionally a thread): feed only, blocks only on backpressure
for chunk in network_or_file {
    sink.decode(chunk, pts)?;            // owns framing; buffers partial pictures
}
sink.finish()?;                          // EOF: drains reorder buffer, unblocks consumer

// Consumer (optionally a thread) : pull frames as they're ready
while let Some(frame) = source.next_frame().await? {
    render(&frame);                      // zero-copy GPU image
    // drop(frame) -> storage returned
}

There are currently three ways to get decoded frames (download, copy_frame_to_planes, OutputOrder::Decode). I'd like only the zero-copy method: a next_frame() that returns the decoder's own GPU image. In the zero-copy path the frame occupies a DPB slot until it's dropped, so the user is responsible for reading it (via download) and dropping it ASAP; this is what provides backpressure. That behavior is documented. copy_frame_to_planes and OutputOrder::Decode become redundant.

I would like decoder.decode(..) to only return an error when there is actually an error. Currently it can also return NeedsKeyframe which is not an error per se. I see two options: 1) don't return a "frame" if no key frame has been received yet (maybe with a decoder.needs_keyframe() -> bool method) or 2) make Decoder::decode return an enum (with variants Frames, Idle and NeedKeyFrame).