NVIDIA DLSS (Deep Learning Super Sampling)

[JMS55]

What problem does this solve or what need does it fill?

We already have TAA. DLSS takes temporal rendering a step further, and renders at a lower resolution and then upscales it over time. This can be a huge performance increase when the cost of shading a pixel is very expensive, such as in per-pixel ray/path tracing.

What solution would you like?

Integrate DLSS as an optional anti-aliasing method, completing our existing MSAA, FXAA, TAA options.

Current Progress

• I have a seemingly-working dlss_wgpu crate allowing usage of DLSS from wgpu https://github.com/JMS55/dlss_wgpu
  • Depends on my wgpu fork More as_hal() methods gfx-rs/wgpu#3376
• I have a branch of bevy adding support for DLSS https://github.com/JMS55/bevy/blob/dlss/crates/bevy_core_pipeline/src/dlss.rs

Problems

• Quality seems bad/wrong - there's heavy aliasing even at 4k quality mode. I'm not sure why.
• There's a crash when enabling one of the ultra quality/performance modes (I forget which).
• Currently using DLSS requires disabling pipelined rendering, due to non-thread safety of the DLSS SDK. I need to figure out how to get around this.
• I haven't setup support (in either dlss_wgpu or bevy) for DLSS A-F presets (different from the quality/performance modes).
• Only Vulkan is supported, and not yet DirectX12.

What alternative(s) have you considered?

DLSS is NVIDIA only, and closed-source. FSR (#5232) is an alternative option from AMD that works on more GPUs and is open-source. There's also XeSS from Intel. For macOS, the only option is MTLFXTemporalScaler.

I've looked into integrating FSR but ran into difficulties. I haven't bothered looking into XeSS or MTLFXTemporalScaler.

[alice-i-cecile]

I'd prefer to prioritize a cross-hardware compatible algorithm in Bevy itself: we should just make sure that a third-party DLSS crate can interop cleanly.

[JMS55]

I'd prefer to prioritize a cross-hardware compatible algorithm in Bevy itself: we should just make sure that a third-party DLSS crate can interop cleanly.

Me too, but I couldn't get FSR working. It would require deep wgpu changes around texture state tracking in order to work. That, or maybe a custom fork of FSR. I'm not sure what the solution would be. That's why I've shifted focus towards DLSS.

That said, I'd like to support all options in bevy, out of the box. In the future we could have a TemporalBundle that tries DLSS, FSR, XeSS, MTLFXTemporalScaler, TAA, choosing the best one or falling back if not supported on the current platform.

[duplexsystem]

I do not think adding integration of proprietary software in bevy (i.e. any proprietary software which a game must ship as opposed to just linking to it) out of the box (or even behind an cargo option) is the right call.

[PROMETHIA-27]

I agree with alice; I think we should do what is necessary to ensure DLSS is as easy as possible to integrate, but it should be strictly up to a third party to do so.

[IceSentry]

ensure DLSS is not as easy as possible to integrate

Do you mean as easy as possible? I don't see why we should make it not as easy as possible.

[PROMETHIA-27]

Yep, typo