Rearrnaging dimensions of full dataset

[anonymousForPeer]

Hi,

Many calculation processes take long because the dimensions are arranged in an unfavourable way. So my idea is to rearrange it.

Here is what I have:

Dimensions:               (latitude: 1138, longitude: 1634, time: 513)
Coordinates:
  * time                        (time) datetime64[ns] 1970-12-01 ... 2098-12-01
  * longitude                (longitude) float64 -10.98 -10.95 ... 40.95
  * latitude                   (latitude) float64 71.48 71.45 ... 35.08 35.05
Data variables:
    myVariable             (time, latitude, longitude) float64 ...

and this is what I need:

Dimensions:               (time:513, latitude: 1138, longitude: 1634)
Coordinates:
  * time                        (time) datetime64[ns] 1970-12-01 ... 2098-12-01
  * longitude                (longitude) float64 -10.98 -10.95 ... 40.95
  * latitude                   (latitude) float64 71.48 71.45 ... 35.08 35.05
Data variables:
    myVariable             (time, latitude, longitude) float64 ... 

I tried ds.transpose but it is just transposing the data variables but not the actual file dimension order.
Also I tried to create a new dataset with setting the dimensions and coords accordingly, but this takes as long as calculating directly on these data.

I very much appreciate any hints.

Thanks
Anne

[benbovy]

Hi Anne,

I think that the Dimensions section of the Dataset repr just shows all the dataset dimensions in the alphabetical order, not in the order in which those dimensions are stored in memory or on disk (this may vary from one variable to another).

Also, if ds.transpose loads the data in memory (unless the data is chunked), I don't think it reorders the data in memory (it's a view of the original array), so if you want to speed-up your computation you might want to do something like ds.transpose(...).copy(deep=True).

[anonymousForPeer]

Hi @benbovy,

apparently in xarray version 0.18.2 the display is shown in alphabetical order but in version 0.19.0 the order is shown in the actual order. In my case (time:513, latitude: 1138, longitude: 1634). So no rearrangement needed.
The storage pressure/computational speed varies depending on what dimension you put your index on. Using an index on the innermost dimension reduces the pressure significantly compared to an index on the outermost dimension as it iterates through the dimensions from inner to outer.

Cheers
Anne

[keewis]

the order is shown in the actual order

that's not entirely correct: it's in "insert order" which allows having a repr with differently typed names. The difference is a bit more visible once you have more than one variable with different dimensions.

In any case, the order of dimensions for a particular variable is displayed in the data variables / coordinates section:

    myVariable             (time, latitude, longitude) float64 ...

[benbovy]

The storage pressure/computational speed varies depending on what dimension you put your index on. Using an index on the innermost dimension reduces the pressure significantly compared to an index on the outermost dimension as it iterates through the dimensions from inner to outer.

I'm not sure to understand what you mean by "index" here. In Xarray, indexes may be defined through coordinates and it seems that in your example you have indexes (coordinates) for each of the time, longitude and latitude dimensions of your variable myVariable. It might help to have a simple code snippet to give us an idea of your computation.

[anonymousForPeer]

I read in a fixed slice of time and longitude (middle and innermost dimension) to reduce the storage pressure using:
xarray.open_dataset(item).isel(longitude=slice(0,30),time=slice(0,1))