gltfpack: Error running through large gltf file

[oscilate]

Running gltfpack from npm.
gltfpack -i sourcefile.gltf -o output\destination.gltf

Error:

exception thrown: RuntimeError: memory access out of bounds,RuntimeError: memory access out of bounds
    at wasm-function[30]:0xaed
    at wasm-function[52]:0x1acd
    at wasm-function[845]:0x238d1
    at wasm-function[847]:0x23d08
    at wasm-function[850]:0x24121
    at wasm-function[410]:0xd47a
    at wasm-function[786]:0x20e32
    at Module._main (\Roaming\npm\node_modules\gltfpack\bin\gltfpack.js:2:84300)
    at callMain (\Roaming\npm\node_modules\gltfpack\bin\gltfpack.js:2:85315)
    at doRun (\Roaming\npm\node_modules\gltfpack\bin\gltfpack.js:2:85893)

Input model is a large gltf file (440 MB gltf + 502MB bin).

[zeux]

For files this large I would recommend using native gltfpack builds (you can download them on release page, see https://github.com/zeux/meshoptimizer/releases/tag/v0.14).

Wasm is limited in the heap space; the current limit is 2 GB. I think I can compile it to raise the limit to 4 GB, but I'm not sure it's going to be enough to fit a file this big. Native builds on the other hand support however much memory your computer has.

Any chance you can share the file, possibly privately via e-mail? The largest file I've tested so far was a 200 MB .glb which is 5 times smaller; would be good to be able to test files larger than that and possibly optimize memory consumption somewhere.

[zeux]

Something else that would help would be to run the attached debug build of gltfpack with the following node option:

node.exe --wasm_max_mem_pages=65535 ./gltfpack.js -i lucy.glb -o lucypack.glb -cc

I suspect it will still run out of memory since 4 GB might not be enough, but it should be closer and the callstack with the failure will help me prioritize memory optimizations.

gltfpack-debug.zip

[oscilate]

node.exe --wasm_max_mem_pages=65535 ..\gltfpack1\gltfpack.js -i .\input.gltf -o .\output.gltf -cc -v

exception thrown: RangeError: WebAssembly.Memory.grow(): Maximum memory size exceeded,RangeError: WebAssembly.Memory.grow(): Maximum memory size exceeded
    at emscripten_realloc_buffer (gltfpack.js:3907:14)
    at _emscripten_resize_heap (gltfpack.js:3926:21)
    at sbrk (wasm-function[135]:0x4a4f)
    at dlmalloc (wasm-function[105]:0x343c)
    at operator new(unsigned long) (wasm-function[60]:0x1d66)
    at std::__2::allocator<Attr>::allocate(unsigned long, void const*) (wasm-function[403]:0xa3cd)
    at std::__2::vector<Attr, std::__2::allocator<Attr> >::__vallocate(unsigned long) (wasm-function[262]:0x744d)
    at std::__2::vector<Attr, std::__2::allocator<Attr> >::vector(std::__2::vector<Attr, std::__2::allocator<Attr> > const&) (wasm-function[376]:0x9ed7)
    at Stream::Stream(Stream const&) (wasm-function[936]:0x26abd)
    at void std::__2::allocator_traits<std::__2::allocator<Stream> >::construct<Stream, Stream&>(std::__2::allocator<Stream>&, Stream*, Stream&) (wasm-function[235]:0x6a4c)

Something else that would help would be to run the attached debug build of gltfpack with the following node option:

node.exe --wasm_max_mem_pages=65535 ./gltfpack.js -i lucy.glb -o lucypack.glb -cc

I suspect it will still run out of memory since 4 GB might not be enough, but it should be closer and the callstack with the failure will help me prioritize memory optimizations.

gltfpack-debug.zip

[zeux]

This should be substantially improved by #142; I have one more change I'm going to make in the next little while which would reduce the peak memory consumption a bit more by unloading parts of the input file early.