relocate data loading in standardize-gcm

[emileten]

• partially closes large EC-Earth data fails at CMIP6 cleaning due to OOM error downscaleCMIP6#574
• tests added / passed
• docs reflect changes
• entry in CHANGELOG.md

@brews there is more detailed background information in ClimateImpactLab/downscaleCMIP6#574, but here I summarize it :

This PR directly modifies a previous one that had breaking changes : #151
Back then, the change was to load the data in memory in dodola.services.clean_cmip6, to fix chunking problems in the case where in the underlying dodola.core.standardize_gcm, calendar == 360_day.

We observe in ClimateImpactLab/downscaleCMIP6#574 that this PR broke runs with data that's too large for our nodes and the downstream computations.

My suggested fix here is to simply move that data loading operation to where it's needed. It turns out that models with high resolution data do not have a 360 days calendar. It's only luck, but I find this quick, cheap fix appealing. And loading operations are arguably part of dodola.core, since all the code inside of it relies on xarray datasets.

[brews]

Thanks for the quick fix, @emileten.

I have one change/confirmation before this merges. It's noted with an inline-comment.

I'm curious where exactly we have the memory explosion with EC-Earth3 input... we may not have time to profile that properly. I don't know if you've already looked into this?

Once this merges we want to be sure to manually check that dodola:dev works properly with 360 calendar GCM input as well as with input from EC-Earth3.

[emileten]

Thanks for the quick fix, @emileten.

I have one change/confirmation before this merges. It's noted with an inline-comment.

I'm curious where exactly we have the memory explosion with EC-Earth3 input... we may not have time to profile that properly. I don't know if you've already looked into this?

Once this merges we want to be sure to manually check that dodola:dev works properly with 360 calendar GCM input as well as with input from EC-Earth3.

@brews Thanks for reviewing !

Yes I looked at this, it's a quick manual profiling, but those are the two lines that trigger this memory blow up :

ds_cleaned["pr"] = ds_cleaned["pr"] * mmday_conversion

ds_converted = xclim_remove_leapdays(ds_cleaned)

Which is not surprising, if the data is loaded in memory.

[emileten]

I checked with dodola:dev that this PR

• fixes our problems, in this successful workflow, that's repeating a standardize step from that one that had failed.

• introduces backward compatible changes, in this successful workflow, that's repeating a standardize step from that one, that had succeeded.