library='np' returns numpy array of STLVectors

[bojohnson5]

When reading a file with doubly-nested arrays with library='np' I get a numpy array of STLVectors when I was hoping to get numpy arrays all the way down. It's very similar to this issue that was closed last year when using library='pd' here. Wanted to check here if this is something I should open an issue for before doing so.

>>> for batch in cenns.iterate(['event/channels/channels.waveform'], step_size='50 kB', library='np'):
...     print(type(batch['event/channels/channels.waveform']))
...     print(batch['event/channels/channels.waveform'])
...     break
...
<class 'numpy.ndarray'>
[array([<STLVector [3644.0, 3644.0, 3644.0, ..., 3645.0, 3645.0, 3644.0] at 0x7fb0d269b910>,
        <STLVector [3631.0, 3628.0, 3629.0, ..., 3629.0, 3629.0, 3629.0] at 0x7fb0d2699f30>,
        <STLVector [1169.0, 1165.0, 1166.0, ..., 1168.0, 1167.0, 1168.0] at 0x7fb0d269a1d0>,
        <STLVector [1118.0, 1118.0, 1117.0, ..., 1119.0, 1117.0, 1118.0] at 0x7fb0d269a170>],
       dtype=object)

>>> for batch in cenns.iterate(['event/channels/channels.waveform'], step_size='50 kB'):
...     print(type(batch['event/channels/channels.waveform']))
...     print(batch['event/channels/channels.waveform'])
...     break
...
<class 'awkward.highlevel.Array'>
[[[3.64e+03, 3.64e+03, 3.64e+03, ..., 3.64e+03, 3.64e+03, 3.64e+03], ...]]

[jpivarski]

This is the intended behavior; a NumPy array of NumPy arrays is really just a NumPy array of arbitrary Python objects, and the uproot.containers.STLVector class represents std::vector, to distinguish it from, say, the uproot.containers.ROOTRVec class for ROOT::VecOps::RVec.

But it would be nice if our STLVector class had an __array__ method to convert itself into a NumPy array in np.asarray. Right now, the underlying array is in self._values, which is private. Are you interested in adding a pull request for

class STLVector:
    ...

    def __array__(self, *args, **kwargs):
        return numpy.asarray(self._vector, *args, **kwargs)

on the array-like types? (STLMap is not really array-like, but several others are.)

(Wrapping an already-NumPy self._vector in np.asarray doesn't copy it by default, but it would make any non-NumPy data into NumPy, and it also offers an opportunity to pass along any arguments.)

[bojohnson5]

Yeah let me open a pull request and add that to the SLTVector container and others that make sense.

[bojohnson5]

I've gone ahead and forked the code and made those changes. This probably comes down to my limited understanding on the library internals but I'm still seeing numpy arrays of STLVectors. Maybe @jpivarski you could help me know what behavior I should expect to see? Thanks.

[jpivarski]

Adding an __array__ method to STLVector just makes functions that want NumPy arrays be happy with getting an STLVector instead. For instance, if you want to pass every element to np.whatever, you can pass the STLVector instances instead of np.ndarray instances and it will convert on the fly. They'll still be STLVector instances because that's intentional in Uproot.

Given a NumPy array of STLVector instances with the __array__ method,

for i in range(len(array)):
    array[i] = np.asarray(array[i])

would be more efficient than it is now—the np.asarray calls would get the array data direction and not copy it (by default). Currently, a loop like the one above would work, but it would be inefficient because it would copy the data by iterating over it (because STLVector already implements __len__ and __iter__).

[bojohnson5]

Oh I follow now. That's pretty neat. I'll need to check my code and see how it handles the changes I've made and then I can submit a pull request with those changes. Thanks!