Add wide color gamut support

[allenwp]

Describe the project you are working on

Godot engine

Describe the problem or limitation you are having in your project

Output from Godot is limited to the Rec. 709 (sRGB) gamut, even with HDR output.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

To support a wide color gamut, instead of Godot’s current Rec. 709 (sRGB) gamut, I propose that Godot provides an option to change the color primaries that are used throughout the engine and editor. This means that Godot will continue to not provide any chromatic adaptation transformations and assume a D65 white point, which minimizes the required transformations. (The D65 white point is used by Rec. 709, sRGB, Display P3, P3-D65, and Rec. 2020.) By allowing the developer to change the color primaries that Godot uses, it will enable a wide gamut working color space that is used for all simulations, calculations, and interfaces, in addition to wide gamut HDR output.

This proposal is independent of color management and is primarily intended to extend the capabilities of HDR output, which provides standardized methods for transmitting color values in wide color gamuts. I suspect that this implementation of a wide gamut working color space will help with implementation of color management in the future, if that is desired.

By only changing the color primaries used by the engine, minimal modifications to existing engine behaviour will be needed: The two main locations that will need changes are the import process and final output (blit). The behaviour and code of existing intermediate buffers will remain almost identical.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Color Primaries

I propose providing the developer with the choice between Rec. 709 (sRGB), P3, and Rec. 2020 color primaries. These three color primaries allow coverage of all Rec. 709, sRGB, Display P3, P3-D65, and Rec. 2020 transmission color spaces.

The P3 primaries could be omitted from the implementation, but I believe there is significant value and simplicity to using a working color gamut that can be correctly rendered on the developer’s monitor; the Rec. 2020 color gamut cannot currently be rendered on most consumer displays, but P3 can.

Another alternative is the ACEScg color primaries, which produce a gamut that is slightly larger than the Rec. 2020 gamut, but ACEScg content (textures, for example) are expected to be viewed with a different white point than the common three transmission color spaces: sRGB, Display P3, and Rec. 2020. I suspect that ACES provides a standard method of chromatic adaptation transformation from its white point to the D65 white point, but the extra computational cost and code complexity of considering different white points can be avoided entirely by omitting the ACES color gamut and primaries.

Availability

Rec. 709 (sRGB), P3, and Rec. 2020 primaries should be available for all rendering drivers and methods because I believe there is value to letting the developer easily switch rendering methods or drivers for certain exports of their project. Only if there is a strong technical reason that it is impossible to perform a color transformation before final (blit) output should we limit the rendering drivers or methods that allow for these different color primaries.

Project Settings

The three color primary options should be presented to the developer in the Project Settings as:

Color Primaries

• Rec. 709 (sRGB)
• P3
• Rec. 2020

It is very important these are presented as “Color Primaries” rather than using the “color space” terminology (which often includes a description of a linear/nonlinear transfer function sometimes data format).

Import

Currently, Godot allows negative values in imported textures. This approach is common with the scRGB (extended sRGB) colour space, but these negative values may produce unintentional and inaccurate results in lighting simulations because negative values are non-physical. For this reason, all colors for all assets should be first transformed to the working color primaries and then clipped to be non-negative before being used in the engine. This must also be the case for resources that are loaded at runtime. See the "Transformations" section below for the required matrices. Importantly, these transformations must be applied to linear RGB values. The steps must be equivalent to:

1. Decode to linear floating point values
2. Transform from the asset's color primaries to the working color primaries using the transformation matrices below
3. Clip to be only non-negative values
4. Process as normal to produce the resulting .ctex file or equivalent resource

Because the Color Primaries project setting affects import behaviour, changing this setting will require some resources to be automatically re-imported.

This hard-clipping of negative values may be visible and look bad when Rec. 2020 color primaries are used with Rec. 709 (sRGB) working primaries, for example. This issue already exists in Godot and I believe it is best addressed by using external tools to save assets with primaries that do not exceed the gamut of the working color primaries used in Godot.

Optional Feature: Color Primaries Import Parameter

In addition to automatically detecting the colour primaries of the imported asset and transforming it to the working color primaries, the developer may override how an asset is interpreted. The new import parameter will have these options:

Color primaries:

• Automatic (default)
• Rec. 709 (sRGB)
• P3
• Rec. 2020

Using this method, the developer may author an image that is saved as Rec. 709 (sRGB) primaries, but interpreted as P3 or Rec. 2020 primaries, for example. This “interpret color primaries as” feature is an optional feature that could be implemented in a separate PR. The use case for this feature is to add easier matching of in-engine color values when using art tools that do not support saving to files with non-sRGB color primaries.

Engine and Scripting

Currently, all Color is assumed to represent values that use Rec. 709 (sRGB) primaries. When the color primaries are changed in the project settings, Color will represent values with these color primaries instead. This is entirely independent of data format or transfer function, so it is expected that Color will still use the nonlinear sRGB transfer function even when P3 or Rec. 2020 color primaries are used.

linear_to_srgb and srgb_to_linear functions’ documentation should be updated to make it clear that these functions only affect the linear vs. nonlinear transfer function of the colour and do not affect the colour primaries or gamut. New functions should be added to convert between the new color primaries using these transformations:

Transformations

I recommend using the color transformations generated by Dash, which is a part of the well-regarded colour-science project. The standards documents often only provide transformation matrices for converting to and from XYZ, so using generated transformations allows for better precision and less error when converting to and from the three gamuts of this proposal.

The specific color spaces that should be used with this tool are:
ITU-R BT.709
P3-D65
ITU-R BT.2020

Chromatic Adaptation Transform: None

Here are the matrices, ready to be copied into the code:

GLES-ready matrices:

Rec. 709 (sRGB) -> P3
const mat3 rec709_to_p3 = mat3(
	0.822461968714362, 0.033194198850962, 0.017082630721120,
	0.177538031285638, 0.966805801149038, 0.072397440663963,
	0.000000000000000, 0.000000000000000, 0.910519928614917);
Rec. 709 (sRGB) -> Rec. 2020
const mat3 rec709_to_rec2020 = mat3(
	0.627403895934699, 0.069097289358232, 0.016391438875150,
	0.329283038377884, 0.919540395075458, 0.088013307877226,
	0.043313065687417, 0.011362315566309, 0.895595253247624);

P3 -> Rec. 709 (sRGB)
const mat3 p3_to_rec709 = mat3(
	1.224940176280560, -0.042056954709688, -0.019637554590334,
	-0.224940176280560, 1.042056954709689, -0.078636045550632,
	0.000000000000000, 0.000000000000000, 1.098273600140966);
P3 -> Rec. 2020
const mat3 p3_to_rec2020 = mat3(
	0.753833034361721, 0.045743848965358, -0.001210340354518,
	0.198597369052617, 0.941777219811694, 0.017601717301090,
	0.047569596585662, 0.012478931222948, 0.983608623053428);

Rec. 2020 -> Rec. 709 (sRGB)
const mat3 rec2020_to_rec709 = mat3(
	1.660491002108435, -0.124550474521591, -0.018150763354905,
	-0.587641138788550, 1.132899897125961, -0.100578898008007,
	-0.072849863319885, -0.008349422604369, 1.118729661362913);
Rec. 2020 -> P3
const mat3 rec2020_to_p3 = mat3(
	1.343578252584332, -0.065297452789120, 0.002821787261701,
	-0.282179670526136, 1.075787915848574, -0.019598494524494,
	-0.061398582058196, -0.010490463059455, 1.016776707262793);

Original copies from source (Transposed), if needed:

Rec. 709 (sRGB) -> P3
[[ 0.822461968714362  0.177538031285638  0.000000000000000]
 [ 0.033194198850962  0.966805801149038  0.000000000000000]
 [ 0.017082630721120  0.072397440663963  0.910519928614917]]
Rec. 709 (sRGB) -> Rec. 2020
[[ 0.627403895934699  0.329283038377884  0.043313065687417]
 [ 0.069097289358232  0.919540395075458  0.011362315566309]
 [ 0.016391438875150  0.088013307877226  0.895595253247624]]

P3 -> Rec. 709 (sRGB)
[[ 1.224940176280560 -0.224940176280560  0.000000000000000]
 [-0.042056954709688  1.042056954709689  0.000000000000000]
 [-0.019637554590334 -0.078636045550632  1.098273600140966]]
P3 -> Rec. 2020
[[ 0.753833034361721  0.198597369052617  0.047569596585662]
 [ 0.045743848965358  0.941777219811694  0.012478931222948]
 [-0.001210340354518  0.017601717301090  0.983608623053428]]

Rec. 2020 -> Rec. 709 (sRGB)
[[ 1.660491002108435 -0.587641138788550 -0.072849863319885]
 [-0.124550474521591  1.132899897125961 -0.008349422604369]
 [-0.018150763354905 -0.100578898008007  1.118729661362913]]
Rec. 2020 -> P3
[[ 1.343578252584332 -0.282179670526136 -0.061398582058196]
 [-0.065297452789120  1.075787915848574 -0.010490463059455]
 [ 0.002821787261701 -0.019598494524494  1.016776707262793]]

As an aside, the "sRGB" color space of the linked tool should be avoided for these internal calculations because it specifically uses the lower precision matrices from the spec, rather than basing the transformation off of the primaries. (I learned this from one of the members of the colour-science project.) By using high precision matrices that are based on the primaries, it produces minimal error while converting between colour spaces.

Okhsl and Okhsv

Okhsl and Okhsv are designed to work with Rec. 709 (sRGB) color primaries. This means that any time Okhsl or Okhsv are used with P3 or Rec. 2020 primaries, additional color transformations will be required. This may result in loss of color information and this is the intended and correct behaviour for Okhsl and Okhsv.

This will also affect the color picker. Care must be taken to ensure that color information is not lost when simply switching back and forth from the OKHSL picker mode.

Tonemapping

The ACES and AgX tonemappers use custom working color spaces. The current matrices for these spaces assume Rec. 709 (sRGB) color primaries as inputs. To address this, the following steps should be used to produce intended behaviour:

1. Perform clipping to non-negative values (as is done currently)
2. Transform to Rec. 709 (sRGB) primaries (leave the negative values and do not clip them)
3. Apply tonemapper working space transformation
4. Apply tonemapping curve
5. Apply tonemapper output transformation
6. Transform back to working primaries

To improve performance, the conversion from working primaries to/from Rec. 709 (sRGB) primaries can be convolved (matrix multiplied into) the tonemapper's working space and output transformations. When this is done, the final matrix from input to working space must have no negative values, as the tonemapping curves for these two tonemappers do not function with negative values. I haven't verified that this will be the case, but I believe both of these working spaces have a larger gamut than Rec. 2020 and P3.

(I may update this proposal with the new combined matrices for these tonemappers in the future...)

AgX

The AgX configuration designed for Blender and used by Godot has been mainly tested with Rec. 709 (sRGB) primaries. Thorough testing with Rec. 2020 primaries will be needed to ensure that the resulting image will look correct on future displays that are able to reproduce the full Rec. 2020 gamut.

Output

I propose that the Color Primaries project setting not only controls the working color gamut of the engine, but also dictates the gamut of colors that may be written to the display buffer.

To implement this, all negative values should be clipped before transforming to the final swap chain’s color space if this buffer’s color space does not implicitly do so. Here are some examples of how this would work:

Color Primaries Project Setting	Display Buffer Colorspace	Action
Rec. 709 (sRGB)	sRGB 8 bit	None (implicit clipping already happens)
Rec. 709 (sRGB)	P3 or Rec. 2020	Clip negative Rec. 709 values
P3	sRGB or P3 8 bit	None (implicit clipping already happens)
P3	Rec. 2020	Clip negative P3 values
Rec. 2020	sRGB or P3 8 bit	None (implicit clipping already happens)
Rec. 2020	Rec. 2020 10 bit	None (implicit clipping already happens)
Rec. 2020	Rec. 2020 16 bit float	Clip negative Rec. 2020 values
any	scRGB (extended sRGB) 16 bit float	Clip negative values

This approach has the advantage of ensuring that unintentional negative values introduced by negative lights or other effects will not have an unexpected consequences on the colors produced by different display modes, unless the user has explicitly chosen a wider color gamut to produce colors that may be beyond what they are able to test on their display.

This hard-clipping of negative values may be visible and look bad, especially when Rec. 2020 color primaries are used with a Rec. 709 (sRGB) display buffer. Continuity smoothing may optionally be applied, but I believe this would be best handled by a separate PR.

Naming

Rec. 709 vs. sRGB

To reduce confusion regarding what is meant by writing "sRGB" in Godot, I propose a convention to use Rec. 709 when referring to color primaries and use sRGB when referring to the sRGB nonlinear (piecewise) transfer function.

I described a few reasons for this in a comment, but in summary: Godot has historically used the term "sRGB" to specifically refer to the "sRGB nonlinear (piecewise) transfer function"; sRGB/Rec. 709 primaries have always been simply implied. Additionally, the sRGB standard adopted Rec. ITU-R BT.709 color primaries to allow for simple display of Rec. 709 content, so sRGB content has always been understood to simply use the Rec. 709 primaries.

Rec. vs BT. abbreviation

Both Rec. 709 and BT. 709 can be used as an abbreviation of Rec. ITU-R BT.709. Using the BT.709 abbreviation has the advantage of being unambiguous regardless of whether the ITU introduces other Recommendations under different categories that share the 709 ID. As an example, writing Rec. 656 is ambiguous between ITU-T Rec. Q.656 and ITU-R Rec. BT.656, which are both in effect. Conversely, writing BT.656 is unambiguous, as it clearly refers to ITU-R Rec. BT.656.

Unfortunately, it seems to be a convention for art and game tools to use the "Rec." abbreviation, so I propose using the (IMO inferior) "Rec." abbreviation to match existing tools.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, changing color primaries requires modification of the output (blit) shader at minimum.

Is there a reason why this should be core and not an add-on in the asset library?

This feature is best implemented as a tight integration into the the engine, especially relating to import and final output.

[ArchercatNEO]

The Wayland protocol (theoretically, needs compositor support) allows applications to submit frames in colorspaces with arbitrary primaries. If we were to think in the widest possible scope what would be the workload/performance implications of allowing the use of arbitrary primaries rather than the primaries of a select set of known colorspaces? With my limited experience it seems like we would require the ability to generate the conversion matrices at runtime (rather than having them baked in from an external generator) but don't know what other requirements we could have. It seems like it could be more scalable as instead of having (n^2)/2 conversion matrices we would only need to store the primaries.

[allenwp]

The Wayland protocol (theoretically, needs compositor support) allows applications to submit frames in colorspaces with arbitrary primaries. If we were to think in the widest possible scope what would be the workload/performance implications of allowing the use of arbitrary primaries rather than the primaries of a select set of known colorspaces? With my limited experience it seems like we would require the ability to generate the conversion matrices at runtime (rather than having them baked in from an external generator) but don't know what other requirements we could have. It seems like it could be more scalable as instead of having (n^2)/2 conversion matrices we would only need to store the primaries.

Oh, neat, thanks for sharing this!

First, in simple terms, the BT.2020 gamut is designed to be as wide as possible while still having only three colour primaries that are visible colours and a D65 white point. It was designed to be future-proofed as the ultimate RGB colour gamut of all future RGB displays. In order to get a wider gamut than BT.2020 using only three primaries, you need to have primaries that are no longer visible colours. This is what other wider colour gamuts do.

So if you wanted to support a wider colour gamut, maybe you could use the ACES2065-1 colour primaries which are designed to encompass all visible colours with only three primaries, but I'm actually not sure about what the lighting simulation implications are of using non-visible colours as primaries. It's probably fine, but I simply don't know. No future three primary (RGB) display will ever be able to reproduce these sorts of ultra-wide gamuts (more than three primaries would be needed to get closer to covering the gamut).

With the theory out of the way, I think there is a notable benefit to usability and simplicity in presenting the two or three colour primaries that I proposed as colour primary options. 99.9+% of users would not want to choose other colour primaries than these three. Although it wouldn't be a huuuuuge amount more complexity to support arbitrary colour primaries, it would be slightly more complex. And I simply don't know what problem we'd actually be solving if we added support for arbitrary primaries.

Additionally, it is already extremely difficult for developers to test colour gamuts that are larger than P3. I'd say from that perspective, it's actually somewhat risky for a developer to use the BT.2020 colour primaries in their project -- But sRGB, P3, and BT.2020 are all transmission primaries used in consumer displays, so this is still reasonable to provide all three options.

But let's say we only provided the user with the three colour primary options and under the hood we generated the matrices dynamically: This is definitely a more complex approach that has more surface area for bugs. Again, even though the option is there for us to do this, I don't see what real-world problem this would solve. Hardcoding six transformation matrices (plus the extra 12 tonemapping matrices) is an incredibly simple solution that is relatively easy to test and ensure correct behaviour compared to generating these matrices from arbitrary colour primaries. I also expect more complexity would be needed to optimize the shaders for arbitrary transformations.

[ArchercatNEO]

I think there is a notable benefit to usability and simplicity in presenting the two or three colour primaries that I proposed as colour primary options

I fully agree with this, even if arbitrary primaries were to be implemented we should still allow users to select from pre-defined primaries (sRGB, BT.2020, P3) in an enum-style selector. This is also what wayland does by providing an enum-based primaries_named API and a float-based primaries API.

And I simply don't know what problem we'd actually be solving if we added support for arbitrary primaries.
...are all transmission primaries used in consumer displays,

I believe the purpose for arbitrary primaries in the wayland protocol ultimately is just flexibility and future proofing godot likely won't need. I believe part of the motivation was that display primaries are not always "precisely" the primaries of the colorspace so using these "slightly-off" primaries could be more accurate. I should note that the wayland protocol allows us to obtain optimal raw primaries for the window as well as an enum value for what primary set it most closely matches. However the difference is likely very minor and as long as these 3 primary sets are the only commonly used display primaries it is unlikely we would ever need to add support for more.

However, if someday some new primary set becomes common in consumer displays we would need to add another primary set and add transformations to and from that new set which could someday become harder to maintain while with arbitrary primary support it could be natively supported with no upstream changes.

That being said, there is the edge case where assets have embedded primaries which are not part of any of the set supported by godot. The simpler solution is to use an external program to fix the assets and transform them into something godot could use then just move on but with arbitrary primary support this could be more seamless.

I don't see what real-world problem this would solve.

Dynamically generating matrices without support for arbitrary primaries definitely doesn't make sense. The biggest issue with hardcoded matrices when we only support 3 primary sets is scalability. Maybe someday we wish to add more primary sets and need to add more matrices. However 6 + 12 matrices is definitely very manageable for now.

Ultimately the suggestion is a mix of curiosity and seeing if utilizing the full wayland protocol is feasible. If arbitrary primaries are too complex and the possibility of adding additional primary sets more seamlessly isn't worth too much then there's not much lost as there will always be the enum-based API which we can make use of.

[allenwp]

So I've been thinking a bit about the naming of the sRGB primaries... there is notable confusion that arises when using the term "sRGB": are you referring to the color primaries or the nonlinear transfer function?

The sRGB primaries (and white point) were chosen specifically to match those of the existing BT.709 standard to allow for easy reproduction of BT.709 content over sRGB. Additionally, Godot has historically used the term "sRGB" to specifically refer to the "sRGB (piecewise) nonlinear transfer function"; sRGB/BT.709 primaries have always been simply implied.

So all of this has me thinking: maybe it makes sense to introduce a convention in Godot to call the primaries "BT.709" and the transfer function "sRGB". This approach will help mitigate confusion when using these terms in both user-facing APIs and internal discussions.

I've noticed that Direct 3D also uses this approach with its naming the primaries: DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709 (P709 means BT.709 primaries and G22 means Gamma 2.2, which in this case is actually nonlinear piecewise sRGB transfer function.)

In practice, this would mean that the Project Setting and Import parameters described in this proposal would be written as:

BT.709 (sRGB)

Because users will be interested in knowing what primaries are used by sRGB, it will be valuable to add (sRGB) to give this hint to users.

Thoughts?

[ArchercatNEO]

The Wayland protocol names BT.709/sRGB primaries as simply srgb in the enum values and clarifies in the description that this is equivalent to BT.709. If we were to take a similar approach it would be something like naming the enum value PRIMARIES_SRGB and then in documentation saying this is also BT.709. Since the dropdown wouldn't be in code and could have a different name the version we would use is sRGB (BT.709). The transfer function is named gamma22.

If there is already a precedent for the transfer function being named sRGB in public APIs I would recommend the primaries being named BT.709 and clarified in docs it is also the sRGB primaries, otherwise I would follow the wayland convention of sRGB primaries / gamma22 transfer function.

Regardless it should be clear in docs and the dropdown that BT.709 and srgb primaries are the same and making sure that the name for primaries and transfer function are different (ie either bt709/srgb or srgb/gamma22).

[allenwp]

BT.709 (sRGB)

After a discussion on RocketChat, iuymatiao pointed out that it is a widespread convention to use the Rec. 709 instead of BT.709 to abbreviate Rec. ITU-R BT.709 in tools like game engines and art/video production tools. I'll update the PR to use this abbreviation instead.

[allenwp]

@Quadtree I noticed that you were asking about wide colour gamut support in the HDR output PR. If you're still interested, feel free to take a look at this proposal.