BasisU: Use KTX2 format and add import options to configure encoder

[beicause]

Implements and closes godotengine/godot-proposals#12165
Should fix #105012.

Different from proposal: the project settings have been changed to import options to better adjust individual texture compression.

This PR switchs the basis format to ktx2 format and adds import options that allow to enable zstd supercompression and Rate-Distortion Optimizated (RDO) to significantly reduce disk size of basisu textures.

Also adds the same settings for PortableCompressedTexture2D.

[fire]

Needs testing, but agree with the feature proposal.

[beicause]

1024x1024 png(alpha unused):

method	disk size	encoding time
origin png	25.0 KiB (25,552)	-
fastest	1.0 MiB (1,048,828)	19 ms
fastest, zstd	9.9 KiB (10,179)	20 ms
fastest, zstd, rdo, rdo_quality_scalar=1.0	9.8 KiB (10,074)	36 ms
medium, zstd, rdo, dict_size=4096 (default basisu params)	11.7 KiB (11,937)	95 ms

4096x4096 jpg(alpha unused)

method	disk size	encoding time
origin jpg	9.2 MiB (9,637,440)	-
lossless	21.8 MiB (22,839,568)	3200 ms
lossy q=0.7	2.4 MiB (2,550,806)	1900 ms
fastest	16.0 MiB (16,777,468)	780 ms
fastest, zstd	15.4 MiB (16,117,890)	900 ms
fastest, zstd, rdo, rdo_quality_scalar=1.0	15.0 MiB (15,707,026)	10000 ms
fastest, zstd, rdo, rdo_quality_scalar=2.0	13.9 MiB (14,600,149)	10000 ms
fastest, zstd, rdo, rdo_quality_scalar=3.0	12.8 MiB (13,411,973)	9400 ms
fastest, zstd, rdo, rdo_quality_scalar=5.0	11.0 MiB (11,523,290)	7500 ms
fastest, zstd, rdo, rdo_quality_scalar=10.0	9.1 MiB (9,491,140)	5200 ms
fastest, zstd, rdo, rdo_quality_scalar=50.0(lowest quality)	8.5 MiB (8,874,016)	3900 ms
fastest, zstd, rdo, rdo_quality_scalar=3.0	12.8 MiB (13,411,973)	9100 ms
fastest, zstd, rdo, rdo_quality_scalar=3.0, dict_size=4096	12.0 MiB (12,539,692)	27500 ms
medium, zstd, rdo, dict_size=4096 (default uastc_level,dict_size params)	15.3 MiB (16,027,703)	53000 ms

basisu v1.50.0 command line

time basisu  -uastc aerial_rocks_02_diff_4k.jpg # RDO is disabled by default
________________________________________________________
Executed in   12.93 secs    fish           external
   usr time  186.04 secs    0.00 micros  186.04 secs 
   sys time    0.61 secs  970.00 micros    0.61 secs

aerial_rocks_02_diff_4k.ktx2 15.7 MiB(16,499,621)

time basisu  -uastc -uastc_rdo_l 1 ./aerial_rocks_02_diff_4k.jpg # RDO is enabled with quality_scalar 1.0
________________________________________________________
Executed in   53.91 secs    fish           external
   usr time  348.33 secs    0.06 millis  348.33 secs
   sys time    0.82 secs    1.07 millis    0.81 secs

aerial_rocks_02_diff_4k.ktx2 15.2 MiB (15,980,796)

[beicause]

I'm considering move these settings to texture import options that allow setting different compression params for different texture. Some extreme settings are not suitable for all textures.

[michaelharmonart]

I think having per texture import settings makes sense. That's the first place I'd think to go to adjust quality for any other texture format in godot

[fire]

I think we should get together in a videoconference and review the design of the options.

[beicause]

basisu v1.60 #103968 seems to be a great improvement and something is changed.

The m_quality_level is renamed to m_etc1s_quality_level which explicitly indicates it doesn't affect uastc. So this param should be removed.

Need retesting in v1.60.

[beicause]

In basisu v1.60 I also get ~4x imporvement (5500 ms -> 1400 ms) for https://polyhaven.com/a/cambridge 4096x2048, no mipmaps. But for the images in #105080 (comment), it doesn't, instead it's slightly slower. The compressed size is exactly the same in v1.50 and v1.60. Compression speed may vary depending on the texture.

[michaelharmonart]

ran into some mystical (probably linktime) problems with basisU on the web recently
#104497

Any chance you've tested the new basisU version on a release build/export template for the web

Guess I'm just hoping it'll magically fix it 😂 (probably not, but hey it's worth a shot)

[beicause]

Unfortunately, this PR with basisu v1.60 doesn't fix #104497 when enabling lto=thin
The export template in ci artifacts with lto=no works, I haven't tested lto=full
I notice there are some warnings related to basis_univeral_unpacker when using lto=thin:

wasm-ld: warning: function signature mismatch: _Z28basis_universal_unpacker_ptrPKhi
>>> defined as () -> void in bin/obj/modules/libmodule_basis_universal.web.template_release.wasm32.a(image_compress_basisu.web.template_release.wasm32.o)
>>> defined as (i32, i32, i32) -> void in lto.tmp

wasm-ld: warning: function signature mismatch: _ZNK19AudioStreamPlayback19get_sample_playbackEv
>>> defined as () -> void in bin/obj/servers/libservers.web.template_release.wasm32.a(scu_servers_audio.gen.web.template_release.wasm32.o)
>>> defined as (i32, i32) -> void in lto.tmp

wasm-ld: warning: function signature mismatch: _ZNK9Texture2D18create_placeholderEv
>>> defined as () -> void in bin/obj/scene/libscene.web.template_release.wasm32.a(scu_scene_resources_6.gen.web.template_release.wasm32.o)
>>> defined as (i32, i32) -> void in lto.tmp

wasm-ld: warning: function signature mismatch: _Z24basis_universal_unpackerRK6VectorIhE
>>> defined as () -> void in bin/obj/modules/libmodule_basis_universal.web.template_release.wasm32.a(image_compress_basisu.web.template_release.wasm32.o)
>>> defined as (i32, i32) -> void in lto.tmp

wasm-ld: warning: function signature mismatch: _ZNK11AudioStream15generate_sampleEv
>>> defined as () -> void in bin/obj/servers/libservers.web.template_release.wasm32.a(scu_servers_audio.gen.web.template_release.wasm32.o)
>>> defined as (i32, i32) -> void in lto.tmp

wasm-ld: warning: function signature mismatch: _ZNK9Texture3D18create_placeholderEv
>>> defined as () -> void in bin/obj/scene/libscene.web.template_release.wasm32.a(scu_scene_resources_6.gen.web.template_release.wasm32.o)
>>> defined as (i32, i32) -> void in lto.tmp

[beicause]

I placed the basisu parameters in a struct to improve readability.

And I'd like to enable zstd by default to actually make basisu disk size smaller as described in docs

[Mickeon]

There may be some confusion regarding how encoding speed is documented.

[BlueCube3310]

Tested on Windows 10 desktop and web, it works as expected:

Desktop	Web

Each texture, including the lightmap and shadowmask, is compressed as BasisU. The exported .pck is 9mb with supercompression, and 17mb without. Better results can likely be achieved by tweaking the settings

[Calinou]

In terms of UX, this adds a lot of import options when Basis Universal is chosen as a compression mode. It makes the already lengthy texture import options even longer.

I'm thinking these should be project settings instead (like #104571 does). Are there situations where you want specific textures only to import faster, but still use VRAM compression (instead of Lossless or VRAM Uncompressed while you're iterating on an asset)?

Maybe the UASTC level could be configurable per-texture, but that's probably the extent where I'd allow per-texture configuration.

[beicause]

The effects of supercompression with different textures vary greatly as my previously test shows. Generally, you would want to use rdo and more aggressive compression for larger textures, but this is not suitable for all textures since it increase encoding times.

In my opinion, adding these options is not a issue for UX, because they are grouped together and placed at the end, and they are only visible in the basisu mode.

What I'm a bit concerned about is whether these options are stable. Not sure if these will change with future updates of basisu. Maybe we should only provide some presets regarding quality and size.

By the way, ours ResourceFormatKTX doesn't transcode basis universal textures. If we support loading basisu KTX2 files, users can use the basisu cli tool as an alternative to customize compression parameters. I will take a look.

Edit: ktx2 loader supports transcoding basisu textures, but not native srgb textures. basisu tool uses srgb by default. Just needs basisu -linear -uastc

[beicause]

Maybe the UASTC level could be configurable per-texture, but that's probably the extent where I'd allow per-texture configuration.

UASTC level mainly controls the quality and has little impact on the size (Higher quality even slightly increases the file size).

I might prefer to only remain the "uastc level" and "rdo quality loss" in import options( and automatically disable rdo when the rdo quality loss is approximately 0), as the other options have a relatively small impact and the default values are fine.

[BlueCube3310]

Maybe the UASTC level could be configurable per-texture, but that's probably the extent where I'd allow per-texture configuration.

UASTC level mainly controls the quality and has little impact on the size (Higher quality even slightly increases the file size).

I might prefer to only remain the "uastc level" and "rdo quality loss" in import options( and automatically disable rdo when the rdo quality loss is approximately 0), as the other options have a relatively small impact and the default values are fine.

I agree with this, the level of aggressiveness with which RDO affects different textures highly varies depending on their type. For example, normal maps are highly sensitive to RDO, while artifacts on diffuse or ORM are much less noticeable. RDO also has the biggest influence on the resulting file size, so it makes sense for it to be exposed per-texture

[BlueCube3310]

Could the Basis Universal settings be moved next to the Compress category? I think it would be more intuitive, since these options are related:

[BlueCube3310]

Tested on Android, it works as expected. Textures are properly compressed and transcoded to ETC2/ASTC formats

[BlueCube3310]

Looks great!

[Repiteo]

Thanks!