Regulatory calculations do not work with latest pandas version

[rschwant]

The regulatory calculations error with the latest pandas version. The error is

KeyError: 'time_local'

in this code:
File /scratch2/BMC/rcm1/rhs/miniconda3/envs/mm_dev_all/lib/python3.9/site-packages/melodies_monet/plots/surfplots.py:74, in calc_8hr_rolling_max_v1(df, col, window)
72 df_rolling_max = df_rolling.groupby("siteid").resample("D", on="time_local").max(min_count=18).reset_index(drop=True).dropna()
73 df = df.reset_index(drop=True)
---> 74 return df.merge(df_rolling_max, on=["siteid", "time_local"])

@mlirenzhenmayi I think this looks similar to the issue you brought up awhile ago and maybe we just didn't notice the problem because Hera only just allowed the latest version of Pandas on it and maybe you were testing this on a different system like Meiyu? @jianheACM, do you have time next week to help fix this problem, since you know this regulatory code the best? Meng's suggestion awhile back may work well, but we should probably test that it doesn't change the answer and works for all options. Thanks!

[rschwant]

I forgot to mention. A quick fix until this is resolved is to downgrade your pandas version. I tested it and it works well. So while your conda environment is active type:

conda install 'pandas <1.5'

[zmoon]

Seems related to this stuff pandas-dev/pandas#47079. That is, it seems that the previous behavior of retaining on ('time_local') when applying .max() was considered a bug, which has now been fixed.

Maybe good fix would be to add a new temporary local time column which can be safely dropped after using, like suggested here

-    df_rolling_max = df_rolling.groupby("siteid").resample("D", on="time_local").max(min_count=18).reset_index(drop=True).dropna()
+    df_rolling["key"] = df["time_local"]
+    df_rolling_max = df_rolling.groupby("siteid").resample("D", on="key").max(min_count=18).reset_index(drop=True).dropna()

That way df_rolling_max still has 'time_local'.

[jianheACM]

@zmoon @rschwant I just pushed my changes to develp_reg2.
The changes are for the pandas issue mentioned here and also for the colormap issue with new matplotlib.
I will submit a PR for your review. Thanks,

[rschwant]

The fix for this has been merged to develop, I will close this once we merge develop to main.
#154

[rschwant]

This fix is now in the main branch.