Tiles: Memory adjusted screen space error

[Avnerus]

Hi,
Opening this PR for feedback, even though this is still a work in progress.
Based on the CesiumJS implementation, I added basic functionality for memory adjusted SSE. The screen space error is adjusted based on the specified maximum GPU memory boundary. I think this feature is crucial for getting tilesets to work on mobile, especially iPhone, because Safari crashes immediately when too much GPU memory is allocated.

I made the feature optional with the memoryAdjustedScreenSpaceError flag (though in CesiumJS it's always on).
Some caveats:

1. The estimation of GPU memory intake for glTFs is not accurate. It should estimate the total memory sum of the geometry, textures, and cesium tables and not just the total bytes of the payload. Hints here and here for implementing this later. Still, even with the current estimate, the implementation is already very useful in restricting the memory usage.
2. The SSE adjustment works by creating "headroom" for memory overflow before increasing the SSE. For example, the maximum memory is 32MB with 1MB overflow. The SSE will increase when the memory usage goes over 33MB, and decrease when it goes back to 32MB after more tiles are unloaded. This is to prevent a ping pong situation where the SSE indefinitely increases and decreases. This overflow amount should be adjustable because it depends on tile sizes, so I added the memoryCacheOverflow option. For example, I noticed that Cesium uses a 1MB overflow for Google 3D Tiles, and it works well here too. But when I tried it on a point cloud it didn't work well. I noticed there is also an open issue in Cesium about this behavior.

I also added the screen space error value to the statistics, since it's very useful to see how it's adjusted.
Finally, note that I removed the maximumScreenSpaceError option from tileset-traverser and instead I use the value from the tileset. I didn't find any use for that option, but might be I missed something from I3S or other.

Thanks and let me know your comments!

[ibgreen]

Thanks for putting up this PR. I suppose we should also update docs and add at least one test.

I am starting to see a pattern where contributors copy in feature X or feature Y from CesiumJS.

• Not sure how easy it would be to create, but for my review and planning purposes it would be helpful to have a list of all missing features so we can track against something, right now I am not sure how much of this to expect...

[Avnerus]

Thanks for the review! Pushed an update and posted some comments.

[Avnerus]

@ibgreen, I think now the only open question for this PR is the matter here. Could you comment on what you think is the best course of action?

[ibgreen]

This is OK to merge. That said:

• Shouldn't we add something to the docs?
• Worth mentioning this as a bullet in the v4.1 release notes (docs/whats-new.md)?
• Also, showing the current screenspaceerror in the stats object should be a minimal change and would be nice so we can track how it changes.

[Avnerus]

Thanks!

Also, showing the current screenspaceerror in the stats object should be a minimal change and would be nice so we can track how it changes.

That one is already in the PR (MAXIMUM_SSE stat).

Regarding a docs update. I could run through that when I make the PR for the typescript cleanup.

[Avnerus]

@ibgreen: two comments about this PR.

1. About the estimation of GPU memory: I realized that this should be implemented on the glTF/model level. Since I am not using loaders.gl's GLTFLoader, but the one provided with three.js, see here and here how I implemented the counts.
   Is there a maintainer for GLTFLoader than can implement those? One only needs to put the sum into tile.content.gpuMemoryUsageInBytes and loader.gl's tile loader will pick it up. As expected, the actual memory counts are much higher than the current estimates (about 10 times).

2. Is there going to be a new alpha soon? I was hoping to have this feature available in three-loader-3dtiles.

Thanks!

[ibgreen]

About the estimation of GPU memory: I realized that this should be implemented on the glTF/model level.

• We don't do that at the moment (However, we do track GPU memory allocation in luma.gl, but that mainly helps us track how much memory grows when we load).
• Adding memory calculations to the GLTFLoader would indeed be a nice feature. I am the primary maintainer but I don't have time, but I'd be happy to review a PR.
• Even if the calculations are incomplete, having something may be better than nothing, and we can improve later.

Is there going to be a new alpha soon?

• We are planning to release 4.1 before the end of the month. Perhaps @dsavinov-actionengine and team could cut an alpha before that?

[dsavinov-actionengine]

@ibgreen @Avnerus
Got an alpha released today:

v4.1.0-alpha.10

• feat(tile-converter): add convertion status dump file (feat(tile-converter): add convertion status dump file #2858)
• chore: Fix publint issues, step 1 (chore: Fix publint issues, step 1 #2862)
• docs: Improve flatgeobuf docs (docs: Improve flatgeobuf docs #2859)
• concatenateArrayBuffers fix (fix(loader-utils): concatenateArrayBuffers bottleneck fix #2856)
• feat(tiles): Memory adjusted screen space error (Tiles: Memory adjusted screen space error #2851)
• chore(deps): bump follow-redirects from 1.15.2 to 1.15.4 in /website (chore(deps): bump follow-redirects from 1.15.2 to 1.15.4 in /website #2852)

Your PR is in there @Avnerus 👍

[Avnerus]

Awesome! Thank you @dsavinov-actionengine .

@ibgreen ,

Adding memory calculations to the GLTFLoader would indeed be a nice feature. I am the primary maintainer but I don't have time, but I'd be happy to review a PR.

I won't be able to work on that for now, but if you'd like I could open an issue, referencing my three.js implementation.