Support more video and audio codecs

[danrossi]

It's not a bug as no error is produced. However when configuring an AV1 codec av01.0.04M.08 it encodes but the muxed video is corrupted with no video track displayed. I presume it needs a complicated mux support for av1 codec.

In ffprobe I get

[h264 @ 0x6284ed72ecc0] missing picture in access unit with size 469
[extract_extradata @ 0x6284ed86bb00] No start code is found.
just do it-trimmed-1752580521687.mp4: Invalid data found when processing input

https://electroteque.org/dev/webav/av1.html

[hughfenghen]

Thanks for your feedback, but I've been too busy lately;
If you're willing to fix that, I can help.

[danrossi]

The hvc1.1.6.L186.B0 isn't muxing properly either but encoding ok. I am not an expert in muxing I saw it required custom flags to support vp9 which I am about to test with Android to get 1080p encoding working. Should that not carry over from the decoder config. Is this something mp4box needs to support ?

https://electroteque.org/dev/webav/h265.html

[danrossi]

I can't see the source of the mp4box used. But I see there is no av1 in it. I see there is hvc1 but h265 isn't muxing correctly and seems a problem.

https://www.npmjs.com/package/@webav/mp4box.js?activeTab=code

There is av1 in the upstream but no reference to a av1decoderConfig I think. The file structure isn't the same ? I can't see in the upstream where there is fast starting as it's affecting Firefox and VLC playback if faststart isn't set waiting for metadata to load.

https://github.com/gpac/mp4box.js/tree/next/src/boxes

I see the upstream has Opus information missing in the other one ?

https://github.com/gpac/mp4box.js/blob/0ceab26c6687f84f1a2ff360a6ea753d649ac5f8/src/boxes/sampleentries/sampleentry.ts#L392

There is also this project https://github.com/Vanilagy/mp4-muxer

The upstream mp4box is es6 but the other one is CJS it seems. Is it a big job to move to the main one ? It might fix the bugs with avc1 and aac muxing I am experiencing.

[hughfenghen]

@webav/mp4box.js is an official fork of mp4box, which I am currently maintaining. Its source code is here.

If you encounter problems with outputting hevc or av1, you can check the commit related to supporting vp9

ccd0ec8
hughfenghen/mp4box.js@41a66e4

[danrossi]

Their updated branch already supports those codecs including Opus. Can it not be using their branch or merge their changes perhaps ?

This fork clearly is having muxing issues with avc1 and aac /opus. As it wont play in Firefox or VLC. It needs fast start also ?

[hughfenghen]

It's great to see that the official mp4box.js project has resumed iterations; I will consider submitting changes from my forked version to the official repository in the future.

[danrossi]

The fork is using their old CJS code not Es6 so impossible to merge changes and too impossible to refactor. What is changed in the forked branch ? I can see they don't offer decoderconfig for those other codecs still so requires a modification still once the branch is in sync with their changes ?

https://github.com/gpac/mp4box.js/blob/main/src/boxes/av1C.ts
https://github.com/gpac/mp4box.js/blob/main/src/isofile.ts#L2683

[hughfenghen]

What is changed in the forked branch ?

hughfenghen/mp4box.js@41a66e4
hughfenghen/mp4box.js@1dcbf87
hughfenghen/mp4box.js@39bfa3b
hughfenghen/mp4box.js@20878ea

[danrossi]

I had a look at other projects who use the other mp4muxer. The avc1/aac played back in VLC as an indication. They limited configuring codecs so won't work in Android at 1080 due to some bug in chrome requiring VP9 instead for 1080. They have an opus wasm fallback for Safari however. So the project isn't useful still.

Unfortunately the path of least resistance without refactoring es6 back to cjs is looking at reforking the updated upstream branch then making the changes back into it. IT does seem you have to patch VP9/AV1 decoderconfig options yourself still like you did for VP9. There is a bit in mp4box about Opus that may assist muxing Opus properly. You may not need to mangle the description for audio and just give it the decoderconfig directly.

[danrossi]

Looking at their updated code Im pretty sure they map the type to the correct box. so the decoderconfig is a legacy thing so modification there isnt needed. There was an error handling it the same way as avc1. have to debug further how to properly give it a description.

[danrossi]

The refactoring required is alot. There is no esds on Opus sample entries so not sure what to use for descriptions. They fixed esds so there is no need to mangling. Im not sure how to set descriptions yet, without addin missing decoderconfig properties. It might be simpler to use the other muxer or faster to just used libav wasm as the muxer. mp4box isn't properly documented how it works and the type hinting makes it worse.

[danrossi]

It's taken a while to figure out the api change and what is required for opus. aac you need to generate an esds box which you send to the description. For Opus you send two boxes and it parses two boxes when you load a video with opus.

There is zero documentation what to give the second opus bitrate box let alone how to type hint both sample entries. Most the work is type hint changes. You create the box by parsing from the decoder config. For VP9 and AV1 it can be parsed externally without having to modify the upstream source I reckon. You send the vp9/av1 box to the description.

So to support both audio codecs you send a list of boxes to description_boxes

[danrossi]

You can see the potential audio box creation here. There is no docs how to do this and no examples.

https://github.com/danrossi/WebAV/blob/opus-codec/packages/internal-utils/src/recodemux.ts#L491

[hughfenghen]

Thank you for spending so much time analyzing this issue.
Supporting demuxing for multiple video formats is definitely appealing to me,
but exporting videos in various container or codec formats isn’t my main goal.

You might want to try using livav.js to muxing the video.
If you're able to share what you achieve with it, I’d be very grateful!