Vendored ffmpeg: h264_vaapi aborts the process on libva < 2.21 — blocks the planned hardware encode ladder

[EtienneLescot]

Found while benchmarking against the software encoder for #572. Nothing is broken today — this is a landmine for the hardware-encode work, filed now so it is not rediscovered from a crash report.

What happens

The vendored build's VAAPI encoders abort the process — SIGABRT, not an AVERROR — the first time a frame is mapped:

implib-gen: libva.so.2: failed to resolve symbol 'vaMapBuffer2' via dlsym:
  /lib/x86_64-linux-gnu/libva.so.2: undefined symbol: vaMapBuffer2
ffmpeg: libva.so.2.init.c:290: _libva_so_2_tramp_resolve:
  Assertion `0 && "Assertion in generated code"' failed.

BtbN does not link libva; it generates implib stubs that dlopen by bare soname and resolve each symbol on first call. When a symbol is missing the generated trampoline calls abort(). There is no return code to check and nothing Rust can catch.

Reproduced with the vendored binary:

LD_LIBRARY_PATH=crates/thirdparty/ffmpeg-linux64-lgpl-shared/lib \
crates/thirdparty/ffmpeg-linux64-lgpl-shared/bin/ffmpeg \
  -vaapi_device /dev/dri/renderD128 -f rawvideo -pix_fmt yuv420p -s 1920x1080 -r 60 \
  -i in.yuv -frames:v 60 -vf "format=nv12,hwupload" -c:v h264_vaapi -b:v 8M out.mp4

The hardware is fine. The same encode through the system ffmpeg 6.1 (which links libva normally) works and honours the bitrate — 2 Mbps requested → 2.012 delivered, 8 → 2.802.

Why it is harmless right now

VideoEncoder::try_open (crates/compositor/src/pipeline_linux.rs) sets AV_PIX_FMT_YUV420P and never allocates a hw_frames_ctx, so ff_vaapi_encode_init refuses before any libva symbol is resolved:

[h264_vaapi] Specified pixel format yuv420p is not supported by the h264_vaapi encoder.
avcodec_open2(h264_vaapi) = -22

Clean EINVAL, process alive. So OPENSCREEN_EXPORT_ENCODER=h264_vaapi today just falls through to aucun encodeur video utilisable — annoying, not dangerous. The abort only becomes reachable once someone wires a hardware frames context, which is exactly what the hardware ladder on the roadmap would do.

Why it will bite

The affected base is not exotic. vaMapBuffer2 is absent from libva 2.20.0, which is Ubuntu 24.04 LTS (libva2 2.20.0-2ubuntu0.2) — the most common Linux desktop base there is. A ladder that probes h264_vaapi first would take the whole Electron process down mid-export on those machines, with no log and no catchable error.

Two consequences worth noting now:

1. A successful avcodec_open2 will not be proof of usability for VAAPI. The device opens fine; the abort comes later. Any probe has to survive a real frame, or check dlsym for vaMapBuffer2 before going near the encoder.
2. scripts/fetch-ffmpeg.mjs declares WANTED_ENCODERS.linux = ["h264_nvenc", "h264_vaapi"], so the vendor-time gate asserts the presence of an encoder that aborts on Ubuntu 24.04. Presence in -encoders is not usability — the same lesson as trap 5 in technical-documentation/architecture/export-pipeline.md.

Possible mitigation, untested

Ship a libva.so.2 ≥ the release that introduced vaMapBuffer2 in crates/thirdparty/ffmpeg-linux64-lgpl-shared/lib. The mechanism is plausible — the dlopen is by bare soname, that directory is already on LD_LIBRARY_PATH, and libva is MIT so there is no conflict with the LGPL vendoring gate. I could not test it (no newer libva available offline), and I could not confirm which libva release added the symbol — only that 2.20.0 lacks it and whatever BtbN built against has it. A symbol probe is safer than a version check either way.

Measured on AMD Ryzen 5 7520U / Radeon 610M, Ubuntu 24.04, Mesa 25.2 radeonsi, /dev/dri/renderD128 present, against BtbN n8.1.2-34-g9b6c8969e0-20260731.

Related: #572, #575.