Slight but problematic differences in coordinates of model.predict outputs compared to X_t

[polpel]

I have encountered an insidious bug in the outputs of model.predict. It seems that the coordinates of the output mean and std Datasets can be slightly different from those of the original Dataset passed as X_t defining the target grid. In my case, I consistently get three values of longitude with a difference of 2e-6 from those of the X_t dataset.

The problem is that apparently this is significant enough that when I compute operations between the two datasets (e.g. err_ds = mean_ds - truth_ds) those coordinates are not seen as aligned and are therefore dropped altogether, resulting in err_ds missing some columns. The bug does not raise any errors so it can be hard to notice the unexpected behaviour.

I've looked into it a bit and the problem originates due to the normalise-unnormalise steps, where going from lat/lon to x1/x2 and back results in these slightly different lat/lon than the originals. I guess it's a numerical approximation issue so not sure if it is fixable directly...
A workaround might be to use Dataset.assign_coords with the original X_t's coordinates at the end of model.predict, instead of using data_processor.unnormalise (if appropriate i.e. when resolution_factor == 1).

[tom-andersson]

Thank you for identifying and raising this, @polpel, I will look into this ASAP. It's possible that using float64s for all the normalisation will avoid these numerical issues - I will check if using single precision values anywhere is the source of the error.

A quick question: were you computing DataProcessor normalisation parameters within the same session where this bug occurred, or did you save the norm_params to JSON and then load them?

[polpel]

I am computing the norm_params in the same session, but I checked and the same occurs if I load them from JSON.

[tom-andersson]

Closed by #25