Linux export: libopenh264 ignores the requested bitrate entirely — 47x over target on heavy content, 4.5x under on light

[EtienneLescot]

libopenh264 in the vendored ffmpeg build does not rate-control at all. It is not that the
requested bitrate is approximate — it has no effect worth measuring, in either direction.

Found while benchmarking the Linux export (#544, #552). It is not specific to that work: every
Linux export goes through this encoder, so the quality of every Linux export is whatever the
encoder feels like, and the setting we expose does not reach it.

Reproduction

120 frames of 1280×720, deliberately incompressible content (so the encoder cannot come in
under target by accident), encoded against the vendored libavcodec with nothing but the
bitrate knobs changed:

requested bit_rate	rc_max_rate	rc_mode	produced
2 Mbps	—	default (quality)	93.5 Mbps
8 Mbps	—	default (quality)	97.5 Mbps
2 Mbps	—	bitrate	94.0 Mbps
8 Mbps	—	bitrate	97.7 Mbps
2 Mbps	2 Mbps	bitrate	94.0 Mbps
8 Mbps	8 Mbps	bitrate	97.7 Mbps

Asked for 2 Mbps with an explicit 2 Mbps ceiling, it emits 94 Mbps — 47× over. Quadrupling
the request moves the output by 4%.

Neither rc_mode nor rc_max_rate changes anything.

What this looks like in a real export

On screen-recording content the symptom inverts, which is why it went unnoticed: a mostly
static screen is very compressible, so the encoder undershoots instead.

	requested	produced
a 60 s 1080p60 export	8 Mbps	1.75 Mbps
the same, requested 12	12 Mbps	0.72 Mbps
the same content via h264_vaapi	8 Mbps	6.95 Mbps

So the bitrate we compute from the output resolution — (w * h * 8 Mbps) / (1920 * 1080),
pipeline_linux.rs — never reaches the encoder. Export quality does not scale with resolution,
and raising the setting does not raise quality. On heavy content the same defect produces a
file many times larger than asked for.

h264_vaapi in the same build honours the request, which is what makes this look like a
libopenh264-specific defect rather than a wiring mistake on our side.

What is ruled out

• Not the option plumbing. av_opt_set(priv_data, "rc_mode", "bitrate") returns 0, and the
  value is still 1 when read back after avcodec_open2. The encoder's own option table
  lists rc_mode with quality as default and bitrate/buffer/timestamp alongside.
• Not bit_rate being clobbered at open. It reads back as 8000000 after avcodec_open2.
• Not a ceiling. Setting rc_max_rate equal to the target changes nothing.
• Not content-dependent. The table above is incompressible input, where a working rate
  control would have the most obvious job to do.

Where I would look

1. Whether the wrapper forwards the target into OpenH264's SEncParamExt —
   iTargetBitrate and sSpatialLayers[0].iSpatialBitrate — or leaves them at defaults.
2. Whether the bundled OpenH264 is built without its rate-control module, in which case the
   setting has nowhere to land regardless of what the wrapper does.

I could not fix this from application code, which is why this is an issue rather than a PR: no
combination of the options ffmpeg exposes for this encoder had any effect.

Why it is worth fixing rather than working around

Beyond the quality problem, it makes the software encoder impossible to compare against
anything. Benchmarking it against h264_vaapi required pinning VAAPI to the bitrate
libopenh264 happens to emit, otherwise the comparison reports a quality difference as a speed
difference. Any future encoder work runs into the same wall.

Measured on AMD Ryzen 5 7520U, Ubuntu 24.04, against
crates/thirdparty/ffmpeg-linux64-lgpl-shared. The probe is ~40 lines of C linked against the
vendored libraries; happy to attach it if useful.

[EtienneLescot]

Root-caused. The cause is neither of the two places you suggested looking, and both of your "ruled out" items are correct — the wiring really is fine.

It is bEnableFrameSkip, and openh264 says so out loud

ffmpeg's wrapper does forward the target. libopenh264enc.c (n8.1.2, byte-identical to master):

param.iTargetBitrate = avctx->bit_rate > 0 ? avctx->bit_rate : TARGET_BITRATE_DEFAULT;
param.iMaxBitrate    = FFMAX(avctx->rc_max_rate, avctx->bit_rate);
param.bEnableFrameSkip = s->skip_frames;
param.sSpatialLayers[0].iSpatialBitrate    = param.iTargetBitrate;
param.sSpatialLayers[0].iMaxSpatialBitrate = param.iMaxBitrate;

So your point 1 is answered: iTargetBitrate and sSpatialLayers[0].iSpatialBitrate are both set. Point 2 is answered too — the rate-control module is compiled in and running.

What is wrong is that -allow_skip_frames defaults to false, and openh264 prints this at every avcodec_open2 — it is in the log of every export we have ever produced:

Warning: bEnableFrameSkip = 0, bitrate can't be controlled for RC_QUALITY_MODE, RC_BITRATE_MODE and RC_TIMESTAMP_MODE without enabling skip frame.

CheckFrameSkipBasedMaxbr() in ratectl.cpp returns immediately when frame skip is off, and it is the only consumer of iMaxSpatialBitrate at encode time — which is exactly why rc_max_rate changed nothing for you.

Turning it on does restore control, and is unusable: at 2 Mbps on your incompressible case it kept 3 of 120 frames.

The second reason, and the one that explains your table

qmin/qmax default to -1 in the wrapper's FFCodecDefault, so openh264 keeps its own (0, 51) — then ParamValidation() judges iMinQp = 0 invalid and silently replaces the whole window with (12, 42). Every Linux export so far has run inside a QP window nobody chose. That is why your rc_max_rate and rc_mode rows are flat: the encoder was pinned against a ceiling of QP 42 the whole time.

Where I have to disagree with the report

the quality of every Linux export is whatever the encoder feels like

Not on realistic input. I rebuilt the measurement on four content classes cut from real recordings in ~/.config/openscreen/recordings, through a C probe that mirrors try_open exactly (no ffmpeg CLI in between):

clip (1080p, real)	ask 1M	ask 2M	ask 4M	ask 8M
mixed: UI + video window	0.98	1.94	3.67	5.64
full-frame webcam	—	1.99	—	7.51

That is the target tracked to within a few percent. Quality on a dense capture is VMAF 96.3; forcing QP down to 22 only reaches 96.6, so the encoder is not starving anything.

Both halves of your report are real but need narrower statements:

• Overshoot needs content openh264 cannot compress at any QP. It reproduces on synthetic noise; it did not reproduce on any real recording I could build, including full-frame camera footage.
• Undershoot is the QP-12 floor (GOM_MIN_QP_MODE, clamped unconditionally). On a static screen the encoder pins there and spends 0.68 Mbps whether you ask for 8 or 40. That is not the encoder failing to reach a target — it is correct VBR behaviour on content that needs no more bits. I could not reproduce your 12 Mbps → 0.72 non-monotonicity; my sweeps are monotone throughout.

One methodology note that may matter: your h264_vaapi comparison almost certainly ran against system ffmpeg, not the vendored build. The vendored h264_vaapi aborts on this machine (libva 2.20 on Ubuntu 24.04 lacks vaMapBuffer2, which BtbN's implib-gen stub asserts on). Filing that separately — it is harmless today, because try_open passes AV_PIX_FMT_YUV420P with no hw_frames_ctx and VAAPI refuses with a clean -22 long before libva is touched, but it is a landmine for the planned hardware ladder.

What I have fixed, and what stays open

#575 does what is reachable from application code, and found a worse defect on the way:

• The export had one keyframe per file. libopenh264 overrides g to -1; try_open never set gop_size; openh264 got uiIntraPeriod = 0. Measured 1 I-frame in 300. Every seek redecoded from frame 0. Now fps * 2.
• The QP window is now chosen rather than inherited. Inert on all four real clips, byte-identical; on noise it cuts 113.2 → 65.5 Mbps.
• High profile instead of Constrained Baseline / CAVLC — -3.8 % bitrate on dense content, -8.0 % on a static screen, at equal VMAF.

Leaving this issue open: none of that gives libopenh264 rate control. rc_mode, rc_max_rate, rc_buffer_size, max_nal_size and level were each measured and each do nothing, so the sweep does not need repeating — VideoEncoder::tune_openh264 records it. Upstream is cisco/openh264#3259, closed without a fix, and libopenh264enc.c has not changed between n8.1.2 and ffmpeg master.

The one genuinely promising direction nobody can reach from here: openh264's SCREEN_CONTENT_REAL_TIME usage type — the right one for a screen recorder, with a (26, 35) QP window and SCC tooling — is exposed by no ffmpeg option. That needs an upstream patch adding an AVOption for iUsageType, and it is probably worth more to this project than anything else on the list.

And happy to take that probe you offered — I wrote my own, but a second one against a different fixture would be worth comparing.