Remove WebCodecs tests with PRECONDITION_FAILED

[nt1m]

Test List

• ✅ full-cycle-test.https.any.html?vp9_p2
• ✅ full-cycle-test.https.any.worker.html?vp9_p2
• reconfiguring-encoder.https.any.html?h264_annexb
• reconfiguring-encoder.https.any.html?h264_avc
• ✅ reconfiguring-encoder.https.any.html?vp9_p2
• reconfiguring-encoder.https.any.worker.html?h264_annexb
• reconfiguring-encoder.https.any.worker.html?h264_avc
• ✅ reconfiguring-encoder.https.any.worker.html?vp9_p2
• temporal-svc-encoding.https.any.html?h264
• temporal-svc-encoding.https.any.worker.html?h264

Rationale

Support for different video codecs is out of scope for WebCodecs.

[nt1m]

@nairnandu @jgraham Can you find someone on the Blink / Gecko side to review? Thanks!

[jgraham]

@ChunMinChang does this make sense from Gecko's point of view? (I think if the spec doesn't mandate specific codec support we can't really have tests that depend on specific codecs in Interop without significant extra buy-in/work).

[ChunMinChang]

WebCodecs doesn't ask the user-agent to support specific codecs. Instead, it has an API that allows users to check if one codec is supported. An idiom way for WPT to run a codec test is to call isConfigSupported first to determine if the test should continue.

Removing av1 or not won't violate the spec, but what's the reason to remove av1? Can't using isConfigSupported first solve this issue?

[nairnandu]

@dalecurtis could you review this proposal for Chromium?

[nt1m]

I would be ok with updating tests to pass if isConfigSupported is false. Though I don't think codec support reflects the quality of the WebCodecs implementations.

[dalecurtis]

We wrote the tests to check isConfigSupported with assert_implements_optional, but may have missed some. As @ChunMinChang notes, tests should bail if they detect the UA doesn't support the codec.

[nt1m]

Seems like those tests indeed use assert_implements_optional, unfortunately, that reports a status of PRECONDITION_FAILED, and for scoring purposes, that counts the test as failing.

I guess this is just #319 then.

[nt1m]

OK I submitted a PR to fix assert_implements_optional being counted as failing: web-platform-tests/results-analysis#178

cc @DanielRyanSmith

[chrishtr]

@nt1m can this issue be closed now given your previous comment?

[nt1m]

AV1 tests are now passing, but we still have a small subset of tests that have PRECONDITION_FAILED for specific codecs.

[nt1m]

@chrishtr @jgraham Here's a list of tests we'd like to remove:

• full-cycle-test.https.any.html?vp9_p2
• full-cycle-test.https.any.worker.html?vp9_p2
• reconfiguring-encoder.https.any.html?h264_annexb
• reconfiguring-encoder.https.any.html?h264_avc
• reconfiguring-encoder.https.any.html?vp9_p2
• reconfiguring-encoder.https.any.worker.html?h264_annexb
• reconfiguring-encoder.https.any.worker.html?h264_avc
• reconfiguring-encoder.https.any.worker.html?vp9_p2
• temporal-svc-encoding.https.any.html?h264
• temporal-svc-encoding.https.any.worker.html?h264

Basically anything that is green and 0 of 1 here: https://wpt.fyi/results/webcodecs?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned&view=interop&q=label%3Ainterop-2023-webcodecs , since it is counted as failing for the purposes of scoring, despite the tests explicitly having optional precondition checks for codec support. The alternative would be PRECONDITION_FAILED tests to not be counted towards the number of total tests when scoring, though it doesn't seem like it is easy to do technically (nor is there consensus on changing scoring in general).

Can we get consensus to remove these specific tests? Thanks!

[dalecurtis]

vp9_2 makes sense, but why the h264 ones? I thought Safari supported that?

[nt1m]

@dalecurtis @youennf mentioned the temporal bits of h264 might not be supported. In any case, the VideoEncoder.isConfigSupported function reflects that in Safari if you check the test results, which is spec compliant behavior if something isn't supported.