Primitive Color Type for .NET Core

[kendrahavens]

Primitive Color Type for .NET Core

The previous discussion of primitives showed that color discussion should expand into a separate issue.

The primitive discussion was started because many customers wanted to move to the .NET Core, but were blocked by the lack of System.Drawing primitives there. There was much discussion and internal debate over defining Point, Rect, and Size and much of that was fueled by scenarios that you guys cited in the previous primitives issue. (Thank you!) Here we are looking for additional scenarios to discuss the advantages of creating a new Color type for the cross-platform .NET Core Graphics API. Feel free to repost your two cents from the previous primitives issue. Thanks for your input!

Topics of Interest: (This is where providing scenarios would be awesome)

1. Naming (ColorRgba, sRGB)
2. Different Color Formats (int8 bits, float 16 bits, float 32 bits, alpha or not) Color4F, Color3F, Color4F
3. Color Spaces (conversion issues)
4. Gamma vs Linear
5. Extension Methods (vector operations: lerp, scaling, conversions: from RGB to HSB and reverse, from vectors)
6. Semantic parity with CSS
7. WinRT Compatibility

Limit Scope: KnownColors will not be brought over as they don’t always translate well into the cross-platform world depending on the hardware so let’s focus our discussion on defining the Color object for now.

[omariom]

How is the primitive going it be used? Which graphics technologies will expose or consume it?

[kendrahavens]

That's actually what we are asking you. Many people have used different definitions of color to accomplish different things. We would like to know what worked and what didn't in different scenarios. We would like to bring over the legacy primitive color type from System.Drawing, but we also want to be aware of scenarios that may require a different structure.

[dsaf]

This is the Color type used by two popular well-designed time-tested game frameworks - Microsoft's own XNA and it's new multi-platform incarnation - MonoGame:

https://msdn.microsoft.com/en-us/library/microsoft.xna.framework.color.aspx
https://github.com/mono/MonoGame/blob/develop/MonoGame.Framework/Color.cs

They have other great primitive types as well, e.g. Rectangle, Vector2/3/4, Point which are used for both 2D and 3D scenarios.

In addition to that, from my limited experience:

1. DarkGray should be darker than Gray: http://stackoverflow.com/questions/3811973/why-is-darkgray-lighter-than-gray

2. I use a custom ColorF (float R, G, B and A) to implement smooth color animations.

3. Any built-in SIMD/NEON support for everything is always appreciated :).

[dsaf]

I guess it makes sense to link the previous discussion: https://github.com/dotnet/corefx/issues/1563

[JimBobSquarePants]

Here's the guts of a Color struct I've been working on. It's pretty handy.

https://github.com/JimBobSquarePants/ImageSharp/blob/master/src/ImageSharp/Colors/Color.cs

[alexperovich]

@kendrahavens is there something here that isn't satisfied by the System.Drawing.Color type that is now in core?

[mellinoe]

System.Drawing.Color is really just a legacy type, and shouldn't be used for new code targeting .NET, in my opinion. It's only there for compatibility.

[alexperovich]

Okay, we need an api proposal to review.

[JimBobSquarePants]

There's more to a single color struct that meets the eye. Whatever would be designed would have to interop well with existing libraries and be performant. TBH I'm not sure it is something that should exist on its own outwith a graphics library.

[kendrahavens]

@alexperovich Well, our whole graphics discussion was torn between designing something new and amazing, but that can also support the legacy type. We decided to start supporting the legacy type first and if designs came up for new and amazing things we could think about them later.

[CatGirlsAreLife]

Well if we need a new color struct, should the usage of the old one in places like System.Windows.Forms properties in some of it’s controls/classes and functions in like System.Drawing.Image/Bitmap/Graphics also need to use this. Although in a way that does not break backwards compatibility. Like maybe have the new derive from the old?

[aaronfranke]

Color math should ideally be done with float to support high precision. 10-bpc and 12-bpc displays are quite common, and HDR color math often needs >= 16 bits per channel. float aka Single precision floats have 23 bits of significant binary digits which is plenty for color math.

Floats also avoid the problem of clipping, as colors above 1 still work fine. Even if they clip when displayed, if the brightness is reduced for the whole image, the brightness differences in bright areas is preserved when using floats. Floats are also more friendly to the user. It's easier to understand that "0.5" is 50% strength than "127". Or was it "128"? I hope you get my point.

Therefore, for all of the above reasons, I propose that a type simply called Color uses float values, of which there are four: R, G, B, and A. Alpha could simply be ignored by users who don't need it, or set to a special value such as float.NaN to disable it. There could also be H, S, and V properties.

Of course, some people may wish to define a color with byte, since it would use less memory. One option is to simply allow "importing/exporting" a Color into an int, and have properties called R8, G8, B8, and A8 for reading/writing byte. Example implementation (MIT licensed)

Another option for the above is that there could be a separate Color8 struct which uses byte directly.

[seanmcd-msft]

Having a standard primitive 3 or 4 byte RGB value type for dealing with simple peripherals such LEDs and LCD panels would be helpful. It should be independent of System.Drawing or other specific UI technology.

[aaronfranke]

If this were to be added, where would it be added to? System.Numerics? As someone who would like to have new standard Color types, I would like to work on this, but only if we agreed on how to do it.

[seanmcd-msft]

that makes sense to me.

[Joe4evr]

💭 Having only glanced at this conversation, I feel like it'd be simply too much work, even conceptually, to unify all the scenarios, lest we end up with:

So it might be better to consider some things separate from others. In particular, I have it boiled down to 3:

• The "main"/"full fidelity" primitive: For the sake of argument, let's call this one ColorF. It'd be 4 floats, and pretty much what @aaronfranke describes. Guidance could/would be that if someone was planning on making a higher-level abstraction for color operations, this is the type they'd want to target the most, as it has the largest range.
• ColorRgba/RgbaColor: This one would fulfill @seanmcd-msft's wish for the simple, independent representation. Since it'd be exactly 4 bytes in size, it's also well-suited for various optimizations.
• ColorHsv/HsvColor: It probably really would be better if the HSV representation of a given color was considered separate instead of tacked on to the main type.

Conversions would be available between any of these in either direction, with the documented caveat that they'll be lossy, except probably when going from either of the bottom two to the first one.

[saucecontrol]

That XKCD is right on.

A completely agnostic ColorF with 4 floats would be useful for optimized/SIMD scenarios, in the same way that System.Numerics.Vector4 already is. A type specifically designed for color could have some useful utility methods, but many/most of those probably end up being specific to a certain interpretation of the color data, making them inappropriate for a one-size-fits-all color structure.

RgbaColor would be broken from the start in the same way System.Drawing.Color is. RGB interpretation is device-specific by nature. This is why System.Windows.Media.Color added ColorContext in its alternate color struct, which allows the color to specify which RGB it refers to. WPF's version of Color also tries to solve the issue of linear vs gamma-companded value interpretation by supporting either/both a byte (companded) and float (linear) value for each color channel.

HsvColor would be useful to some devs, but there is not necessarily the clear/correct conversion path between HSV and RGB that some believe. HSV is essentially a hack that approximates device-independent color from RGB, but that still requires an understanding of which RGB. L.A.B. color is a better device-independent representation, but it requires complex (slow) calculations for conversion. And you need a mapping layer to get from device-independent to device-specific at some point, and the requirements around that vary by device. This is a commonly misunderstood problem, where people believe there is a simple formula that converts from RGB->CMYK or RGB->LAB. Such conversions cannot be performed accurately without device-specific mappings.

These issues become even more salient in light of the fact that display hardware is improving to the point that the sRGB gamut (which was defined to match the output of CRT displays) is no longer enough. Display P3 is already establishing itself as a new standard, and that means the old assumption that sRGB is the one true RGB is no longer safe.

[aaronfranke]

@Joe4evr It definitely can be useful to have multiple Color representations, though I would make a few tweaks to your suggestion.

• ColorF and ColorRgba would both still store colors in RGBA, it's just that the latter would use the 8-bit byte. For this reason, it would make vastly more sense to call it Color8 or ColorB.

• Since ColorF would be the more flexible / fully capable implementation, I suggest that it be called simply Color to mark it as the "recommended" color type in the name. However, it would still be fine to call it ColorF if that's desired.

I don't particularly care about a direct HSV representation. We can just provide FromHsv and ToHsv methods in Color, and users can calculate their own HSV values beforehand.

@saucecontrol One of the advantages of floating-point color is that you can work with values outside the normal sRGB range if desired. Normal values are from 0 to 1, but you can easily have a value of 1.5, though this can't be converted to a "traditional" Color8 and would need to be interpreted in a special way for other standards such as Display P3.

[tannergooding]

I would believe that the appropriate thing to do would be to make the color structs generic (e.g. ColorRgb<T>) as that is the only way to satisfy the byte vs float vs half vs uint vs etc discussion.

Likewise, you can't have a low overhead "general-purpose" Color struct (such as one that supports Rgb or Hsv or Rgba or Bgra/etc). You can really only have distinct types (ColorRgb, ColorHsv, etc) or share when the number of components is consistent at best (such as ColorRgb and ColorHsv each having three or ColorRgba and ColorBgra each having 4).

Anything else requires you to carry additional state somewhere which can make low-level/high-perf cases not as worthwhile (due to additional conversion/copying overhead or additional space taken in memory).

[tannergooding]

What you might ultimately end up with then is a Color3<T> and Color4<T> struct (potentially others, depending on use-case) with various extension methods or static helper classes for operating on particular spectrums (such as Hsv vs Rgb vs Rgba premultiplied vs etc)

[aaronfranke]

ColorRgb<T> would be a disaster for several reasons.

• As far as I know, there's no way to do math with generic types: Proposal: Ability to restrict Generic types csharplang#1345 (and see linked issues in the comments)

• float and half would be on a range of 0 to 1, while byte would be from 0 to 255, and ushort would be from 0 to 65535 (no idea what you have in mind with uint, we don't need nearly that many bits). You would basically need math that checks and works differently for all of these.

There's no valid use case IMO for a type without the alpha channel. If you don't want alpha, you can either ignore it, or we can reserve a special value like NaN to "disable" it. A type that just stores RGB may end up taking the same amount of space as RGBA anyway for alignment purposes. It would be a lot of bloat to include every possible type with and without alpha for very little gain.

What specifically can't be done with a general-purpose Color that uses float?

[tannergooding]

As far as I know, there's no way to do math with generic types

No single-line of code way. It would be supported the same way as System.Numerics.Vector<T> where there are "blessed set" of T (which can be expanded in the future as needed) and the implementation would utilize generic specialization to ensure each T does the right thing. The JIT will then eliminate the dead code paths, ultimately giving you efficient code that is the same as specialized non-generic types.

float and half would be on a range of 0 to 1, while byte would be from 0 to 255, and ushort would be from 0 to 65535 (no idea what you have in mind with uint, we don't need nearly that many bits). You would basically need math that checks and works differently for all of these.

Yup, generic specialization allows this, as mentioned above. DXGI alone defines about ~100-130ish color formats, all of which are used in varying contexts: https://docs.microsoft.com/en-us/windows/desktop/api/dxgiformat/ne-dxgiformat-dxgi_format. These include half-precision, single-precision, 8-bit, 16-bit, and 32-bit integer components from a glance. Other graphics libraries sometimes define additional ones. Having several general-purpose generic types allow you to satisfy/expose all of those formats with little overall overhead as compared to special-casing each one with a separate type.

There's no valid use case IMO for a type without the alpha channel.

There are a plethora of formats which have no support for any alpha channel. Many formats also allow or use pre-multiplied alpha components rather than requiring or allowing an alpha channel at all.

A type that just stores RGB may end up taking the same amount of space as RGBA anyway for alignment purposes. It would be a lot of bloat to include every possible type with and without alpha for very little gain.

That is the point of exposing a few select generic, but component sized Color types. There is no bloat, it is exactly the size you need, and works for the exact scenario you need without necessarily requiring conversion to/from the target type.

What specifically can't be done with a general-purpose Color that uses float?

It may take more space than is required and may not be possible to blit to/from the target format.
There are plenty of valid use cases

[charlesroddie]

What specifically can't be done with a general-purpose Color that uses float?

32 bit per channel is too much space. The idea of having ranges exceding 0. to 1. goes along with specific encodings and color spaces (e.g. scRGB). Doesn't make sense with more standard color profiles.

[JimBobSquarePants]

I don't think there is an advantage to adding this that exceeds the development costs.

[aaronfranke]

I made a PR that implements a new System.Numerics.Colors: dotnet/corefx#40733

The PR has been closed due to the API not being approved, so... consider this comment a proposal.

Here's direct links to my proposed Color.cs, Colors.cs, Color8.cs, Colors8.cs, and ColorUtils.cs.

The two types are Color, which uses four single-precision floats for the RGBA components, and Color8, which uses four bytes for the RGBA components. Personally I would be fine with only having a type that uses floats, but a type that uses byte was a very strongly requested feature here, so I think it's worth including. Each of these types can be converted to each other, constructed with each other, and have properties R8/Rf/etc to allow storing as one type and easily using the other in code.

Another commonly requested feature was a Color type that uses HSV. Against @Joe4evr's advice, I don't think this is worth adding as its own type, but Color and Color8 also have H/S/V properties for working with hue, saturation, and value, and there are ToHsv and FromHsv methods.

To make this, I used and/or referenced several existing Color types:

• Of course, System.Drawing.Color (see docs). Many things from System.Drawing.Color were not included or implemented differently because they did not seem relevant for a generic built-in Color type. For example, instead of GetHue etc methods, H etc properties were used.

• Godot's Color type (see non-C# docs), which uses 32-bit float RGBA, MIT license. Godot's strategy is to have one universal Color type that's used everywhere.

• Xenko's color types, MIT license. Xenko's strategy is to have tons of types and write tons of code for every use case, and there's a point of bloat and diminishing returns here. Xenko has Color as using four byte values and Color4 as using four float values, among several other types, but this isn't ideal and they have expressed that they're open to changing this.

A few important notes:

• All preset Color structs are in a separate static class Colors. This makes code vastly more readable because it separates preset colors from actual Color members, and I think it's a great improvement compared to System.Drawing.Color. All preset colors from System.Drawing.Color are included, as well as all preset colors from Godot ("MaroonX11" is Godot's "Maroon").

• Currently HSV in Color8 are float properties. It does make sense to have saturation and value as byte, but hue is usually expressed in terms of degrees, and I've never seen APIs that express it as a byte. I decided to use float everywhere for HSV for S/V consistency with hue, but we could still change these to byte.

• Traditionally, Microsoft APIs such as DirectX and System.Drawing use ARGB, but the rest of the world uses RGBA. These types use RGBA order everywhere, which also helps fit with the goal if "Semantic parity with CSS".

[aaronfranke]

Any update on this? .NET Core 3.1 has been out for awhile, and .NET 5 is intended to be released at the end of this year, so this seems like the appropriate time to add System.Numerics.Colors.

I have a branch here, and a closed PR with more information here. I encourage any .NET Core members to take this code and get it merged and conforming to whatever standards you have, or please let me know what I can do to help or what I need to change about it to get it merged.

[saucecontrol]

As was pointed out in your PR that was closed, the first step is a concrete API proposal. See this doc for the process and examples of how to proceed: https://github.com/dotnet/runtime/blob/master/docs/project/api-review-process.md

[tannergooding]

Closing this as I don't think it is super actionable. As indicated by dev's extremely familiar with the area (such as @JimBobSquarePants and @rickbrew) here and on related issues such as #32418, this space is extremely scenario specific.
Trying to create a general-purpose color type or a set of color types is likely a non-starter and isn't something I can see us adding into the core framework.

To essentially reiterate what was said here: #32418 (comment), I think the only actionable thing would be to add some "common algorithms" for specific scenarios where applicable.
For example, the types in System.Numerics are oriented around games and multimedia based applications. They are already providing core algorithms similar to the DirectX or OpenGL Math Libraries.

It might be reasonable to then expose some algorithms that operate on a Vector4 as if it were a 128-bit, 4x float, RGBA color format. Likewise, it might be reasonable to extend the existing System.Drawing.Color or the System.Windows.Media.Color types with additional functionality as appropriate.