[GStreamer][MediaCapabilities] missing validation of HdrMetadataType and TransferFunction during decodingInfo method

[pgorszkowski-igalia]

decodingInfo method: https://www.w3.org/TR/media-capabilities/#decodinginfo-method, in case of VideoConfiguration (https://www.w3.org/TR/media-capabilities/#videoconfiguration), should also validate HDRMetadataType (https://www.w3.org/TR/media-capabilities/#enumdef-hdrmetadatatype) and TransferFunction (https://www.w3.org/TR/media-capabilities/#transferfunction).

Available video decoders can provide HDRMetadataType and TransferFunction which they support.

[pgorszkowski-igalia]

@emutavchi : is it, what for you need additional API to set HDR types and EOTF?

[pgorszkowski-igalia]

Based on the release notes from the gstreamer: https://gstreamer.freedesktop.org/releases/1.18/

Video library

    High Dynamic Range (HDR) video information representation and signalling enhancements:

        New APIs for HDR video information representation and signalling:
            GstVideoMasteringDisplayInfo: display color volume info as per SMPTE ST 2086
            GstVideoContentLightLevel: content light level specified in CEA-861.3, Appendix A.
            plus functions to serialise/deserialise and add them to or parse them from caps
            gst_video_color_{matrix,primaries,transfer}_{to,from}_iso(): new utilility functions for conversion from/to ISO/IEC 23001-8
            add ARIB STD-B67 transfer chracteristic function
            add SMPTE ST 2084 support and BT 2100 colorimetry
            define bt2020-10 transfer characteristics for clarity: bt707, bt2020-10, and bt2020-12 transfer characteristics are functionally identical but have their own unique values in the specification.

        h264parse, h265parse: Parse mastering display info and content light level from SEIs.
        matroskademux: parse HDR metadata
        matroskamux: Write MasteringMetadata and Max{CLL,FALL}. Enable muxing with HDR meta data if upstream provided it
        avviddec: Extract HDR information if any and map bt2020-10, PQ and HLG transfer functions

we can assume that HDR types and EOTF things are supported by gstreamer >= 1.18.

[emutavchi]

@pgorszkowski-igalia we need a way to configure browser to accept HDR mime types MediaSource.isTypeSupported, for example to pass tests like:

https://ytlr-cert.appspot.com/latest/main.html?&test_type=formatsupport-test&tests=10,11,12#1739287094977

[pgorszkowski-igalia]

@pgorszkowski-igalia we need a way to configure browser to accept HDR mime types MediaSource.isTypeSupported, for example to pass tests like:

https://ytlr-cert.appspot.com/latest/main.html?&test_type=formatsupport-test&tests=10,11,12#1739287094977

Thanks @emutavchi . The failing test: "10. isTypeSupported EOTF Support" validates MIME types which also contain information about EOTF:
"video/webm; codecs="vp09.02.51.10.01.09.16.09.00"; width=1280; height=720; framerate=30; eotf=smpte2084"
"video/webm; codecs="vp09.02.51.10.01.09.18.09.00"; width=1280; height=720; framerate=30; eotf=arib-std-b67"
"video/webm; codecs="vp09.02.51.10.01.09.18.09.00"; width=1280; height=720; framerate=30; eotf=strobevision"

"eotf=smpte2084" and "eotf=arib-std-b67" are expected to be valid, while "eotf=strobevision" is expected to be invalid and the test expects that for invalid EOTF in MIME type, "isTypeSupported" should fail.

MIME type specification does not specify what EOTF values are valid and also in https://w3c.github.io/media-source/#istypesupported-method there is nothing about "optional parameters" and EOTF valid values.

I will try to find some more information about that but it seems that these tests are specific for some custom implementation (at least it is my assumption).
NOTE: Chrome and FF also fail with that.

[pgorszkowski-igalia]

EOTF parameter parser is implemented in Cobalt:
https://github.com/youtube/cobalt/blob/main/starboard/shared/starboard/media/parsed_mime_info.cc#L37

It bases on "YouTube TV HTML5 Technical Requirements" defined in https://developers.google.com/youtube/devices/living-room/files/pdf-guides/YouTube_TV_HTML5_Technical_Requirements_2018.pdf

[pgorszkowski-igalia]

I have added in upstream (WebKit/WebKit#40539) the similar implementation as is in Cobalt. With this implementation the tests from https://ytlr-cert.appspot.com/latest/main.html?&test_type=formatsupport-test&tests=10,11,12#1739287094977 pass.

[pgorszkowski-igalia]

And downstream PR: #1463

[pgorszkowski-igalia]

@emutavchi : #1463 is merged so now the tests from https://ytlr-cert.appspot.com/latest/main.html?&test_type=formatsupport-test&tests=10,11,12#1739287094977 pass.

[pgorszkowski-igalia]

@emutavchi : can we close this ticket?

[emutavchi]

Hi @pgorszkowski-igalia, sorry for the delayed reply. pr 1463 seems to always return that required eotf are supported. Which is not always true. We'd like to have an option to control that. Return not supported if connected display doesn't support HDR at all (i.e. ae22d69) or a particular transfer function is not supported.