Compressed texture support

[superdump]

Description

• Support DDS, KTX2, and Basis Universal compressed textures
• Support loading them from gltf

Solution

• [Merged by Bors] - KTX2/DDS/.basis compressed texture support #3884
• Generic compressed texture support
  • Load compressed texture data as-is by adding create_texture_with_data to RenderDevice and using it
  • When preparing the GpuImage, if it is a compressed format, use RenderDevice::create_texture_with_data with the specified format, which includes all mipmaps. If it is not compressed, copy mipmap level 0 only as before.
  • Explicitly specify whether a texture to be loaded from a buffer is sRGB or linear at load time. For glTF textures are marked as being linear or not and this is used to indicate sRGB or not.
  • Respect platform-supported compressed formats
    • Add CompressedImageFormats flag type with ASTC_LDR, BC, and ETC2 flags to match what is supported by wgpu::Features
    • Add a supported_compressed_formats member to ImageTextureLoader and GltfLoader and populate it based on WgpuOptions at loader-initialisation-time.
    • When loading an Image using Image::from_buffer(), pass the CompressedImageFormats stored in the loader to inform the load() about whether compressed formats are supported, and/or about formats to transcode to, in case of universal compressed formats such as UASTC/ETC1S.
• DDS
  • Load raw data from DDS files, using the ddsfile crate, into Image and set the correct image format. Note that this includes all mipmaps.
  • Map all DDS formats, both D3D and DXGI to wgpu texture formats
• KTX2
  • Load raw mip level data from KTX2 files, using the ktx2 crate, into Image and set the correct image format. Note that this includes all mipmaps.
  • Map all KTX2 formats to wgpu texture formats
  • KTX2 supercompression support
    • Parse the Data Format Descriptor and Sample Information sections of the KTX2 header to get all the necessary information needed to identify the format. PR here: Data format descriptor BVE-Reborn/ktx2#4
    • ZLIB using flate2 which in turn uses miniz_oxide which is a rust DEFLATE implementation
    • Zstandard using ruzstd which is a rust Zstandard decompressor
  • Transcoding from KTX2 UASTC to an appropriate format using the basis-universal crate which uses bindings to the C++ Basis Universal library
    • PR here: Add support for the low-level UASTC transcoder aclysma/basis-universal-rs#8
• Basis Universal
  • .basis file support for ETC1S/UASTC with transcoding to an appropriate format, also using the basis-universal crate
• Support for 2-component normal maps stored in the red and green channels, deriving the third component assuming the stored normal is a unit normal.

Next Steps

• WASM target support for both .basis files and transcoding UASTC from KTX2 files. The basis-universal library has been compiled to WASM. It is possible to call other JavaScript code from rust code compiled to WASM using C-style FFI so maybe it will work for WASM to WASM as well?
• KTX2 BasisLZ supercompression support. BasisLZ is quite non-trivial so I feel like it is reasonable to only support it via .basis files to start with.

[mrec]

We could have a fallback that decompresses and compresses to another format that is supported, but this all starts to move in the direction of what the asset pipeline should do however.

True, although I don't see that as a reason to rule it out as something supported at runtime. There are (very niche) edge cases where the build-time asset pipeline doesn't get a look-in, e.g. runtime-procedurally-generated content or user-supplied images.

[fintelia]

Has there been thought about assets that work on both desktop and mobile? The former typically only supports BC compression and while the latter supports ETC2 and ASTC, but almost never BC.

The trendy approach at this point is to store assets in Basis format, and then transcode them at runtime to whatever compressed format is required. KTX2 actually supports Basis, so perhaps this is planned?

[cwfitzgerald]

At this point I'm basically of the opinion that basis doesn't provide much over just doing 2/3 encodings for the various formats. Not only do you don't have the chance to use more interesting formats like bc6h, bc4/5, eac, and hdr astc, but there's a quality hit and it requires a cpu side decode step, which isn't slow, but isn't terribly fast either, especially compared to being able to pull GB/s out of compressed textures.

Every build only has to ship with one "kind" of assets, the exception being web, and you can still just pull the relevant platform assets based on what the device supports.

Because od this I'm very disillusioned by basis and the only kind of supercompression I might use is simple zstd, but that likely should happen at the asset bundle level anyway.

[superdump]

I started looking into adding ktx2 support using the ktx2 crate. I found an example model with ktx2 textures. They are unfortunately 'supercompressed' with zstd and the crate doesn't yet support supercompression. I may do support for non-supercompressed to start with and leave it at that. Supporting supercompressed looks to be non-trivial (it's not just getting some data and munging it through a zstd decompressor as far as I can see.)

[superdump]

I've pushed basic (not super-compressed) KTX2 support. I think support for supercompression should be a separate issue as it's a deeper rabbit hole.

I think next I'll refactor it a bit (too much stuff in image.rs) and then look into properly handling the 2-component normal maps.

[fintelia]

Supporting supercompressed looks to be non-trivial (it's not just getting some data and munging it through a zstd decompressor as far as I can see.)

Did you confirm that feeding the level data through a zstd decompressor doesn't work? A quick glance at the spec suggests that for Zstd and ZLIB compression that might just be all you need. (BasisLZ super compression on the other hand looks way more involved and almost certainly not worth worrying about now...)

Totally understand putting this off for later though!

[superdump]

I inspected the bytes of the levels and I think I fed them into zstd but I’m not totally sure. I’ll give it another go as for a while I was using the wrong data handle so it’s worth checking. I also looked at the spec and it looked like it is intended that each mip level is individually compressed by the supercompression technique. But there is also some supercompression global data that isn’t yet parsed in the ktx2 crate and I don’t know what is in that. I can take a look at the sample file I found in a Khronos repo too to see what’s in that. I’m thinking that it could be zstd metadata that needs to be used to configure the zstd decompressor before feeding it a mip level.

[superdump]

It seems the supercompression global data for Zstandard is empty (0-byte length) and indeed shovelling the individual mip levels through Zstandard decompression does work (in the sense that it decompresses without errors.) However, I think parsing of the data format description part of the KTX2 metadata is necessary to discover the format of the supercompressed data.

[superdump]

I've got basic parsing of the Data Format Description working, but the file I'm looking at has all UASTC format textures so I can't just load them directly. I'll either have to see if I can hook up a UASTC -> supported format transcoder or find a file which uses plain BC* or so. Maybe I could convert the dds texture version of bistro to ktx2... I expect there are tools around to do that.

[superdump]

I have just finished implementing a hopefully reasonably correct mapping of ktx2 data format descriptor to wgpu texture formats where supported. It may not be 100% correct in the sense that maybe it could map some overlapping format to a wgpu format that it shouldn't, however unlikely this may be.

I still need to add zlib decompression support, and then try to verify how well this works in practice by gathering a bunch of ktx2 files that contain raw or standard-compressed image data, not 'universal' compressed image data like ETC1S or UASTC. Then get a PR in shape against the ktx2 crate for the Data Format Descriptor parsing, and then make a PR out of this.

[superdump]

Here is the ktx2 crate PR for adding Data Format Descriptor parsing support: BVE-Reborn/ktx2#4

[superdump]

I've got transcoding from UASTC from KTX2 files to appropriate formats (I tried BC7 and BC4, the latter for fun) using the basis-universal crate. As I 'had' to use the basis-universal crate for transcoding, and supporting BasisLZ supercompression in KTX2 is non-trivial, I added support for .basis files as well, which brings support for ETC1S.

[superdump]

I also made the various formats optional and non-default so people can pick and choose what they want/need to build in.

[superdump]

Here's a PR for the exposing the so-called 'low-level' UASTC transcoder in basis-universal that I have used for transcoding UASTC from KTX2 to e.g. BC7: aclysma/basis-universal-rs#8

[superdump]

I've implemented respecting the supported compressed formats when loading images, as reported by wgpu. So if a GPU-supported format (BC/ASTC/ETC2) is not supported by the GPU, an error should communicate that. And if loading a universal format like ETC1S/UASTC, the supported formats are used to identify to what GPU-supported compressed format we should transcode.

I had previously written that we could implement a fallback to decompress the compressed format. I have implemented that for UASTC/ETC1S universal formats so they will fall back to RGBA32. I don't think this is trivial to implement for other cases.

[superdump]

The ktx2 PR was merged and it has been released on crates.io as 0.3.0.

The basis-universal PR was merged but I wanted to test it some more to give aclysma some more confidence before publishing it in a new crate version.

Along the way I was enlightened that M1 GPUs support ASTC / BC / EAC/ETC2 / PVRTC (so pretty much everything) also on macOS but that wgpu was only reporting support for BC. I filed an issue here but dug into it some more and ended up making a PR here.

Once basis-universal has a new crate version, I think the main thing remaining to fix is how to handle the various numbers of components in textures where more components could be expected. So normal maps having 2 components instead of 3 as the third can be inferred if unit length is assumed. Then perhaps the feature is mostly rounded-out and I'll create a PR for review.

[superdump]

A new version of basis-universal was released and it contains the exposure of the low-level UASTC transcoder.

I added support for 2-component normal map textures through a standard material flag.

[superdump]

I made a draft pull request: #3884