Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add automated tests using Meta XR Simulator #150

Merged
merged 1 commit into from
Jun 14, 2024

Conversation

dsnopek
Copy link
Collaborator

@dsnopek dsnopek commented May 15, 2024

The goal of the PR is to add the structure for automated tests running on GitHub Actions using Meta XR Simulator.

With the XR Simulator, it's possible to record inputs and periodically take screenshots, which both (the inputs and the images) get saved to a .vrs file. Then we can have the XR Simulator play back those inputs, taking screenshots at the same moments, recording into a new .vrs file. Then using pyvrs and a Python script provided with the XR Simulator, we can compare the screenshots, generate diffs, and decide whether they match or not based on a threshold of similarity.

Ultimately, I'd like to have a sample project and a .vrs file for testing each feature that can be tested via the XR Simulator (which includes passthrough along with the Scene API and Spatial Anchors, using synthetic environments, which is pretty cool).

However, the goal of this PR is just to figure out the structure, including one example .vrs file.

This is marked as a draft, because there's still a few things to figure out:

  • Running the tests with a working Vulkan driver. I tried using SwiftShader as a CPU-based Vulkan implementation (the code is still commented out in the PR), but while this worked for Godot, the XR Simulator refuses to start an OpenXR session because there is not a "suitable" graphics device (I forget the exact error message). So, right now I'm running this with a self-hosted GitHub runner on one of my Windows machines. If we're going to stick with a self-hosted runner with a real GPU, I'd like to get it running in the cloud somewhere first.
  • I'm using the build_profile PR. This makes build times much faster, but I'm not sure it's ready to be merged - see Use build_profile to improve build times #149 - if this PR ends up being ready first, I'll rebase that PR out of this one.
  • I'm using a custom Windows build of Godot master. Once Godot 4.3-beta1 is out, we should be able to use that, but using Godot 4.3-dev6 crashes at the moment.

@dsnopek dsnopek added the enhancement New feature or request label May 15, 2024
@dsnopek dsnopek marked this pull request as draft May 15, 2024 22:44
@BastiaanOlij
Copy link
Member

Neat, I think the only thing we need to be careful off is that we don't put ourselves in a jam when upstream changes in Godot cause false positives in testing and we find ourselves unable to merge features because someone changed something in the rendering engine and missed the impact on XR (happens sadly all to often).

@dsnopek
Copy link
Collaborator Author

dsnopek commented May 16, 2024

Neat, I think the only thing we need to be careful off is that we don't put ourselves in a jam when upstream changes in Godot cause false positives in testing and we find ourselves unable to merge features because someone changed something in the rendering engine and missed the impact on XR (happens sadly all to often).

Since we can play them back and automatically record the new outcome, I think we could fairly easily make a script that just re-records all the .vrs files when there are rendering changes. We should probably pin a specific Godot version, and then when we update that, we can run that script if necessary.

@dsnopek
Copy link
Collaborator Author

dsnopek commented May 16, 2024

I spent a little more time trying to make SwiftShader work, but still haven't managed it. So I don't lose track of it, this is the error that appears in the console when Godot tries to start an OpenXR session when using SwiftShader:

[XrSim][00000.146735][E][arvr\projects\openxr_simulator\src\sim_xrapilayer_rendering_vulkan.cpp:419] Failed to find a suitable GPU

We then get these messages from Godot on the following lines:

OpenXR: Failed to obtain vulkan physical device [ XR_ERROR_RUNTIME_FAILURE ]
ERROR: Condition "!device_retrieved" is true. Returning: ERR_CANT_CREATE
   at: _initialize_devices (drivers/vulkan/rendering_context_driver_vulkan.cpp:475)
ERROR: Condition "err != OK" is true. Returning: err
   at: initialize (drivers/vulkan/rendering_context_driver_vulkan.cpp:591)
ERROR: Unable to create DisplayServer, all display drivers failed.

However, Godot itself (even the editor) seems to run fine with SwiftShader, albeit very slowly. :-) It's the XR Simulator that seems to have a problem with SwiftShader.

@dsnopek dsnopek added this to the 3.0.0 milestone Jun 12, 2024
@dsnopek dsnopek marked this pull request as ready for review June 12, 2024 14:53
@dsnopek
Copy link
Collaborator Author

dsnopek commented Jun 12, 2024

I've updated the PR to point at Godot 4.2-beta1, and removed the dependency on PR #149.

So, as soon as this passes tests, it'll finally be ready for review and merging!

@dsnopek dsnopek changed the title [DRAFT] Add automated tests using Meta XR Simulator Add automated tests using Meta XR Simulator Jun 12, 2024
@dsnopek dsnopek force-pushed the xr-sim-tests branch 2 times, most recently from 3391e76 to 5d63c83 Compare June 12, 2024 19:07
@dsnopek
Copy link
Collaborator Author

dsnopek commented Jun 13, 2024

This one is passing CI and ready for review now!

@fire
Copy link

fire commented Jun 14, 2024

We discussed how the vulkan code in xr simulator didn't like swiftshader and how we could check it and maybe create a minimal reproduction project (pull request) for someone to look at.

@m4gr3d m4gr3d merged commit 87dff2b into GodotVR:master Jun 14, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants