Dynamic kwargs generation as part of the recipe #7
Conversation
|
🎉 New recipe runs created for the following recipes at sha
|
|
Ughh sorry for the formatting changes here @cisaacstern. I guess my black was updated? |
|
I think we can absorb #6 into this one here? Or should we move this ahead and then add new iids as part of a new PR? Might be cleaner. |
|
/run recipe-test recipe_run_id=166 |
|
/run recipe-test recipe_run_id=167 |
|
✨ A test of your recipe I'll notify you with a comment on this thread when this test is complete. (This could be a little while...) In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/166 |
|
✨ A test of your recipe I'll notify you with a comment on this thread when this test is complete. (This could be a little while...) In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/167 |
| return output_chunksizes[0] | ||
|
|
||
|
|
||
| def dynamic_kwarg_generation( |
There was a problem hiding this comment.
Wow @jbusecke heroic work here :)
|
Pangeo Forge Cloud told me that our test of your recipe To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/166 If you haven't yet tried pruning and running your recipe locally, I suggest trying that now. Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps! |
|
Pangeo Forge Cloud told me that our test of your recipe To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/167 If you haven't yet tried pruning and running your recipe locally, I suggest trying that now. Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps! |
|
🎉 New recipe runs created for the following recipes at sha
|
|
🎉 New recipe runs created for the following recipes at sha
|
|
🎉 New recipe runs created for the following recipes at sha
|
|
🎉 New recipe runs created for the following recipes at sha
|
|
Failures like we saw on #7 (comment) look like they could be related to not running the latest versions of |
|
/run recipe-test recipe_run_id=174 |
|
/run recipe-test recipe_run_id=175 |
|
✨ A test of your recipe I'll notify you with a comment on this thread when this test is complete. (This could be a little while...) In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/175 |
|
✨ A test of your recipe I'll notify you with a comment on this thread when this test is complete. (This could be a little while...) In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/174 |
|
🚨 Caveat: I just noticed that the filesize we extract from the ESGF API does not agree with the size we get for cached files. import requests
url = "https://esgf-data.dkrz.de/esg-search/search"
params = {'type': 'File',
'format': 'application/solr+json',
'latest': 'true',
'limit': 500,
'mip_era': 'CMIP6',
'activity_id': 'CMIP',
'institution_id': 'CCCma',
'source_id': 'CanESM5',
'experiment_id': 'historical',
'variant_label': 'r1i1p1f1',
'table_id': 'Omon',
'variable_id': 'zos',
'grid_label': 'gn'}
resp = requests.get(url=url, params=params)
resp.json()['response']['docs'][0]['size']/1e6gives ~470MB But when I cache the actual file with this recipe from feedstock.recipe import recipes
import pathlib
import xarray as xr
recipe = recipes['CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429']
root = pathlib.Path(recipe.storage_config.cache.root_path)
for f in os.listdir(root):
print(xr.open_dataset(root.joinpath(f)).nbytes/1e6)I get a bit over 800MB. I suspect that the API is giving some sort of compressed filesize? Maybe @agstephens can confirm/deny this as somebody who has more experience with the API? Basically unless we can get an estimate of the filesize as it appears when opened the inferred chunksizes will not adhere to our parameters! In the short run, I think it is acceptable here, and we should move on to scale to more datasets. Just wanted to document this here. Ultimately pangeo-forge/pangeo-forge-recipes#355 would solve this issue, since we would not have to rely on the ESGF API at all, and could infer these kwargs from the cached files, which should be more accurate. |
|
🥳 Hooray! The test execution of your recipe Here is a static representation of the dataset built by this recipe: You can also open this dataset by running the following Python code import fsspec
import xarray as xr
dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-174/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
dsin this ChecklistPlease copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.
|
|
The test of variable I will manually cancel this test now, since it's been stuck here for many hours. This seems to be a different issue than those reported in pangeo-forge/pangeo-forge-recipes#358. As a short-term workaround, I wonder if there's a way to tune the |
I would think we could, but I need to understand what causes the locking, better. Once I know what to tune for, I can definitely try to address this. |
|
I wonder if there is a way to specifiy the subsets in a more robust way. If I could say e.g. |
|
🎉 New recipe runs created for the following recipes at sha
|
|
I just pushed a simplified logic for the subsetting. I previously tried to find divisors that would result in subsets that are evenly dividable into the
|
|
/run recipe-test recipe_run_id=177 |
|
and I will also do 176, just to be sure that still works |
|
/run recipe-test recipe_run_id=176 |
|
✨ A test of your recipe I'll notify you with a comment on this thread when this test is complete. (This could be a little while...) In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/177 |
|
✨ A test of your recipe I'll notify you with a comment on this thread when this test is complete. (This could be a little while...) In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/176 |
|
From the Pangeo Forge Cloud backend logs: which looks promising! |
|
Pangeo Forge Cloud told me that our test of your recipe To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/177 If you haven't yet tried pruning and running your recipe locally, I suggest trying that now. Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps! |
|
🥳 Hooray! The test execution of your recipe Here is a static representation of the dataset built by this recipe: You can also open this dataset by running the following Python code import fsspec
import xarray as xr
dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-176/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
dsin this ChecklistPlease copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.
|
|
🎉 New recipe runs created for the following recipes at sha
|
|
@jbusecke, the To move forward, I've removed |
This PR reflects work that @cisaacstern and I did together a few days ago + plus some additional work I finished today.
In general this PR enables us to dynamically generate values for the required input kwargs:
target_chunksandsubset_inputs, which vary widely across the datasets in the CMIP6 archive. This will hopefully enable us to scale this recipe to a larger number of instance ids, which would otherwise present a tremendous burden in terms of maintenance.The basic logic starts from a list of 'allowed' chunksizes, to ensure that we do not chunk too 'odd', and if possible maintain full years, or multiples of 5 years etc. Currently this is only added for monthly resolution (but we can add others later easily, which will be required for e.g. the daily data in #5 ).
We then pull information of the ESGF API for any given instance id to determine the number of timesteps (from the daterange given in the filename) and the size for each file. Consequently we try to maximize the chunksize to get as close as possible with the given constraints to a target chunksize of 150MB (but never larger!) and designate the chunksize value as
target_chunkkwarg.We apply a similar logic to determine into how many parts we need to divide the input to have an input size <500MB, and parse that into the
subset_inputskwarg.