Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gltfpack: Substantiall reduce memory footprint #147

Merged
merged 9 commits into from
May 19, 2020
Merged

gltfpack: Substantiall reduce memory footprint #147

merged 9 commits into from
May 19, 2020

Commits on May 19, 2020

  1. gltfpack: Allow using full 4GB heaps in Wasm builds 

    This will let us process larger scenes with gltfpack.js
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    457c5ae View commit details
    Browse the repository at this point in the history

  2. gltfpack: Avoid data copies during glTF mesh parsing 

    We now reserve the space for mesh and stream data and parse the data
directly into the target vectors. This avoids the cost of the copy and
reduces the peak memory consumption for mesh-heavy scenes; on Thai
Buddha model this reduces peak RSS from 1390 MB to 1198 MB.
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    ecaa585 View commit details
    Browse the repository at this point in the history

  3. gltfpack: Use zero-copy animation parsing 

    We now reserve the space for animation data before parsing it similarly
to mesh data.

Also clean up mesh data parsing a bit to follow a similar push_back +
back structure for streams.
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    f726c1f View commit details
    Browse the repository at this point in the history

  4. gltfpack: Free unused buffers after glTF parsing 

    After we parse mesh and animation data, we can usually free the buffer
data. The exceptions are when the buffers are referenced by images or
skins, since we don't copy these out.

We also can't free the GLB buffers yet because this requires freeing
JSON data which requires fixing extras[] access.

This reduces max RSS when processing Thai Buddha scene from 1198 MB to
893 MB.
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    38f5f46 View commit details
    Browse the repository at this point in the history

  5. gltfpack: Don't generate fallback buffer unless required 

    Instead of always generating the fallback buffer, we only do this when
the fallback is actually requested. This saves us some time to copy the
uncompressed data, and means we don't need to create uncompressed
fallback in memory just to throw it away.

This reduces max RSS when processing Thai Buddha from 893 MB to 822 MB.
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    c0293bd View commit details
    Browse the repository at this point in the history

  6. gltfpack: Move extras writing out of writeNode 

    This will make it slightly easier for us to implement extras evacuation.
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    575b557 View commit details
    Browse the repository at this point in the history

  7. gltfpack: Evacuate extras out of JSON blob 

    Instead of reading the extras data directly out of data->json, we copy
the data we need into a std::string during parsing.

This allows us to reclaim the memory used by the JSON data early,
something that will happen in a separate change.

Ideally we wouldn't need to do this since cgltf could do all of this
itself, but this is a conservative change that doesn't require upstream
work.
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    91011f4 View commit details
    Browse the repository at this point in the history

  8. gltfpack: Free .gltf / .glb data early 

    Since we now extract extras into a separate string, we can free .gltf
file contents immediately after parsing it. If we're parsing a .glb
file, we need to hold onto the buffer data until we parse it fully, but
once we do - if we don't have any skin/image data inside - we can free
it as well.

This reduces the max RSS for stadium.glb from 100 MB to 75 MB.
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    7b85fed View commit details
    Browse the repository at this point in the history

  9. gltfpack: Minor cleanup 

    Checking data::bin is more direct and cleaner. Also we can use `s`
directly instead of streams.back.
    @zeux
    zeux committed May 19, 2020
    Configuration menu
    Copy the full SHA
    bd8dfc2 View commit details
    Browse the repository at this point in the history