Change of the output dimensions order

[koldunovn]

There is a plan for changing dimensions of the output to [time, depth, vertices/elements] by @hegish . I suggest to make it also part of refactoring efforts - so that we package a lot of changes all together into FESOM2.2 or something. Having it in current FESOM2.1 master would kill compatibility with post processing tools. I would combine also with #172

[patrickscholz]

Whats the advantage of this ?

[koldunovn]

@hegish might give you a better explanation, but my understanding is that it is faster to dump files to the disk. Plus we are consistent with the reso of the world :)

[trackow]

I think @hegish showed that this is much faster to dump to disk across machines than any netcdf chunking setting can possibly achieve. There were issues on Juwels where @JanStreffing had found a solution, and personally I had problems on the cray at ECMWF where 3D output for a relatively small grid (0.25°) took on the order of tens of minutes.

[JanStreffing]

See also #103

[hegish]

Yes, the data we use to write 3D FESOM output is per level. The previous ordering in FESOM 2 NetCDF is per column. This means a jump between writes depending on the number of levels (i.e. an avoidable seek operation). On some systems (like ECMWF or JUWELS) this leads to a slow down by a factor of about the number of levels, i.e. 50 times slower. The same for reading. With the reordering we can read and write a whole level in one go, which is much more efficient.
This would also make the 3D data layout identical to the FESOM CMIP6 data.

[koldunovn]

Closed by #193

[trackow]

Nice :)