Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement cloud metrics into PMP #741

Closed
lee1043 opened this issue May 20, 2021 · 1 comment · Fixed by #851
Closed

Implement cloud metrics into PMP #741

lee1043 opened this issue May 20, 2021 · 1 comment · Fixed by #851
Assignees
Milestone

Comments

@lee1043
Copy link
Contributor

lee1043 commented May 20, 2021

Implement @mzelinka's cloud metric.

Message from @mzelinka:

"... Note that the paper documenting this just went under review, so it is possible that reviewers will demand that we change our metric. But it is a simple RMSE, so there is nothing really complicated. (I will present the science of this at the meeting on Thursday.)

The primary code is main.py. Please ignore all my awful coding and inability to optimize things. You will note that the code begins by creating xmls that point to the end-user’s local netcdf files. I’m sure the PMP handles this a bit differently. And probably this could just be replaced by xarray.open_mfdataset("path/files/*.nc", combine='by_coords'), skipping over the whole xml-generation step. I haven’t yet taking the plunge into xarray, so I only know this from the xCDAT doc that Jiwoo and others have put together (thanks for that!)."

@lee1043 lee1043 added this to the 2.0 milestone May 20, 2021
@lee1043 lee1043 self-assigned this May 20, 2021
@lee1043 lee1043 modified the milestones: 2.0, 3.0 Aug 22, 2022
@lee1043 lee1043 linked a pull request Aug 22, 2022 that will close this issue
@lee1043
Copy link
Contributor Author

lee1043 commented Feb 8, 2023

Comment from @mzelinka (11/3/2022):

There are a few things that could be written out to the JSON file. You have two of them already, the ECS and the cloud feedback RMSE. Additionally the total cloud feedback (the last element of “assessed”) and the climatological cloud error metric (referred to as “E_NET”) should be included. These are the fields plotted in these figures (GFDL is highlighted as an example):

https://github.com/mzelinka/assessed-cloud-fbks/blob/main/figures/WCRP_totcldfbks2_v_E_NET_amip-p4K.pdf

https://github.com/mzelinka/assessed-cloud-fbks/blob/main/figures/WCRP_assessed_RMSE_v_cldfbk2_amip-p4K.pdf

I think what this means on your end is instead of returning just RMSE6[m] and ECS6[m] from dataviz.make_all_figs(), you should additionally return E_NET6[m] and assessed6[m,-1].

So I might suggest returning E_NET6[m], RMSE6[m], assessed6[m,-1], ECS6[m] from dataviz.make_all_figs()

And then in cloud_feedback_driver.py you would have:
climo_cld_rmse, cld_fbk_rmse, tot_cld_fbk, ecs = dataviz.make_all_figs()

If we wanted to also write out all the individual assessed cloud feedbacks, that could also be done. That would help people rationalize the RMSE score that they got. These data reside in the “assessed” variable. I don’t really see a good reason to not do this, as ultimately it is just another set of scalars that can be reported in the JSON (but please let me know if there is a reason). In this case, rather than just returning the last element of assessed6, you’d return the entire vector: assessed6[m,:].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant