Memory management issues

[Elliot-Wadge]

Hey, sorry I'm new to uproot and I'm struggling with how to properly extract what I need from root files and also manage the memory being allocated. Here's what I've been trying to make work, however the ram usage will just increase with every iteration until my computer crashes.

import uproot
import gc
import psutil

with uproot.open("example.root") as root_file:
    for key in root_file.keys():
        gg_counts = root_file[key].values()
        x = root_file[key].axis('x').centers()
        # do something useful
        del x
        del gg_counts
        collected = gc.collect()
        print('RAM memory % used:', psutil.virtual_memory()[2])

Here I've tried deleting and garbage collecting but I'm guessing that the root_file is keeping a reference to the loaded arrays so this doesn't work. I was hoping someone could tell me the proper way to do this.

Thanks!

[Elliot-Wadge]

I did find one work around which is just reversing the order of the for loop and the with statement but is there a more proper way of doing this?

[MoAly98]

I have recently been working a lot on using uproot to handle large inputs that don't fit into memory, so I may be able to share my insights from many discussions with the experts, but I would need a bit more information :)

I may be missing something really straight-forward, but can you clarify a abit more what you are aiming to do? Are you expecting to extract all the data from the ROOT files? how large are ROOT files and what's the content (are they histograms or events organised in trees and branches?). I am also unsure how flipping the order of the for loop and with statement helps? Do you know the keys you want in advance? because if not then you will have to open the file anway to get them, right?

Sorry if I'm missing something trivial here :)

[Elliot-Wadge]

great! Okay so

1. No I don't need to extract all the data from the ROOT file at one time, but I do need to access most of the data at some point, but it doesn't have to all be in memory at the same time
2. the ROOT file is about 1Gb and I'm accessing 2D histograms from it which are about 30-50mb (in python), I make a few projections, fit a few peaks and then I'm done with the histogram and it should be removed from memory
3. flipping the for and the with statement helps because when you close the root file at the end of the with statement it is no longer referencing the histogram so python's built in garbage collector will now delete the histogram from memory freeing it. However if the with statement is outside the for loop the file is never closed through the 160 histograms opening and so the loaded histograms are never dereferenced and so the garbage collector doesn't collect it (I think) they just build up until their all in memory and it crashes.
4. Yes I do know what keys I want before hand, the code snippet there was just an example of what would be a worst case I guess. There's about 160 TH2D histograms that I need to access and perform some fits on.

Hope that makes it clearer, let me know if doesn't, thanks

[MoAly98]

Yes that's clear :)

I am not a fan of swapping the loops -- opening and closing the file ~200 times does not sound like the most efficient way to do this and it will not scale well (but I may be wrong).

From my understanding, this looks like a possible use-case for uproot.lazy -- you can just open the file lazily and then through a loop over the keys you can apply your functions to perform the fits and all. This would mean that only one histogram is loaded at a time. You can probably do this more efficiently by processing chunks of histograms in one go rather than 1-by-1. I don't think you would need to use a garabage collector at all when doing that -- python memory handling is often good enough (at least I rarely ever need to gc myself). Can you see any limitation from using lazy loading?

You might also benefit from using coffea processors -- I am not a power user of coffea yet, but I believe you will not need to worry about the memory management when running your code inside the processor. You can also benefit from the coffea executors to parallelize your workflow and operate on multiple histograms at the same time.

That's just my 2 cents, and I am sure a more elaborate answer or suggestion can come from the experts and we can both learn something new :)

[Elliot-Wadge]

Yes, that's exactly the reason I didn't want to swap the loops as well haha. Okay great I'll have a look at the documentation for that, that sounds perfect. Yes the garbage collector was more of a last resort attempt to force the memory to be free but it didn't work anyway, the python garbage collecting should be fine.

Thanks!

[Elliot-Wadge]

Since uproot.lazy is deprecated I found that setting object_cache=None in uproot.open() solved my problems and was the simplest solution