Include Godot in Mesa trace-based testing

[anholt]

Describe the project you are working on

I'm a Mesa (linux open source graphics drivers) developer working on core infrastructure of the project, currently looking at expanding our trace-based testing. We currently have trace-based testing of radeonsi (current Radeon), panfrost (ARM Mali), freedreno (Qualcomm), llvmpipe (software rasterization) and virgl (Chrome OS linux environment on all platforms) drivers, and soon with the Intel graphics drivers for linux.

Describe the problem or limitation you are having in your project

Right now, Mesa development has excellent testing of the conformance suites, but limited testing of complicated rendering pipelines in real world apps. I would like to improve the automated testing of real apps to ensure that they continue to work as we develop Mesa, and we should prioritize supporting open source software when doing so.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

If someone from the godot project could get samples of rendering into our traces-db repository, then we can test each MR for regressions in rendering those samples, which should translate to godot engine experiencing more reliability on Linux graphics drivers.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Submit trimmed apitraces traces to traces-db of redistributable content covering the interesting components of godot engine's graphics rendering. There's one trace in there currently, hopefully the samples repository can provide more.

Once we merge new traces, then you or I can edit the trace expectations for each driver that can replay it to list the traces -- you initially fill the checksum with a dummy "0" value or whatever, then run a CI pipeline and check the rendering on the dashboard or in the failed job's artifacts to see if it looks good on that driver and get the proper checksum to put in the file.

If this enhancement will not be used often, can it be worked around with a few lines of script?

The alternative to trace-based testing is building and running the tests, but that means custom per-app infrastructure for running and capturing, which doesn't scale.

Is there a reason why this should be core and not an add-on in the asset library?

Someone from godot engine working on this will be much more informed as to what are useful samples to be capturing than a Mesa developer, so hopefully someone from your side can help.

[Calinou]

Note that the current Godot master branch only has a Vulkan renderer. It doesn't have an OpenGL renderer yet – it's being worked on. Therefore, if you need OpenGL traces, it would be better to work with the 3.x branch or a tagged Godot 3.x release.

[lfelipe]

For reference, I believe some traces were already added there as part of https://gitlab.freedesktop.org/gfx-ci/tracie/traces-db/-/merge_requests/43/commits
@Calinou or other Godot folks: are there other games/examples that would likely be good to try to get in there?

[lfelipe]

In a somewhat related topic, there were a few shaders from Godot that were imported into Mesa's shader-db as well: https://github.com/freedesktop/mesa-shader-db/commit/f9a727be9071a087c06f08a10542063a8cf90e9e#diff-8074348a993e74b1030dd45887ae24118afbd274293a1e21cbcc644156a7a5fc
Didn't see yet any fossils for Vulkan, perhaps that might be something worthwhile looking at?

[anholt]

Getting vulkan fossils would also be great.