Runtime allocation profiling

[koute]

It'd be nice to add some sort of memory allocation/deallocation profiling support for allocations made within the runtime (through the FreeingBumpHeapAllocator), and maybe write a script so that the data can be easily exported into something like my Bytehound for analysis.

(This could probably be worked around by profiling the native runtime, but that's not ideal since 1) the native runtime is going to go away anyway, 2) it'll be extra noisy so it'd be harder to analyze and 3) it might not exactly match with what happens in an actual WASM runtime considering e.g. the native environment will be 64-bit and WASM is 32-bit, etc.)

[bkchr]

I also already thought about something similar, but maybe for the start just some kind of prometheus metric. Collect the different allocation sizes, the maximum memory usage etc. I think based on that we should already be able to create some nice dashboard.

[koute]

Hmm... well, yeah, there are essentially two different use cases here: one is monitoring, and this would be something that I think could work well as Prometheus metrics and could even be turned on by default, the other one is actually digging into the details and seeing exactly where, what and why is being allocated, and that would need more detailed allocation logs which would only be enabled on demand.

[bkchr]

Yeah that makes sense. However, I think first having at least some broader view would be a good start.

[kianenigma]

I am up for having monitoring as well.

The only way that I can get this metric accurately now is to use the execute-block command of try-runtime, which works but is slow and not a continuous process.

Ideally, we should have a prometheus metric that tells you how much memory a block execution consumed per block. You can currently roughly achieve this by enabling wasm-heap=debug. But this emits a log for every instance of the allocator not just the block import.

The hard and rather noisy part is that we need need to somehow return these allocation amounts, and then pipe it back to prometheus if the call is actually relevant (i.e execute_block).

Looking around the code, it seems like changing the interface of WasmInstance's fn call to return the memory usage would be a good start, probably feature gated.

[koute]

Looking around the code, it seems like changing the interface of WasmInstance's fn call to return the memory usage would be a good start, probably feature gated.

Yeah, we intend to refactor the executor interface anyway in the near future so I'll be keeping this issue in mind when working on refactoring it.