Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update pooch downloader: use CESMDATAROOT variable when available #52

Merged
merged 12 commits into from
Dec 15, 2020

Conversation

andersy005
Copy link
Contributor

@andersy005 andersy005 commented May 22, 2020

Addresses permission issue when local storage path points to a location that we don't have permissions to write to.

Closes #51

@andersy005
Copy link
Contributor Author

Here's an example of this in action on casper:

In [1]: from pop_tools.grid import grid_defs

In [2]: example = grid_defs['POP_gx1v7']

In [3]: example
Out[3]:
{'lateral_dims': [384, 320],
 'vertical_dims': 60,
 'vert_grid_file': 'gx1v7_vert_grid',
 'horiz_grid_fname': 'inputdata/ocn/pop/gx1v7/grid/horiz_grid_20010402.ieeer8',
 'topography_fname': 'inputdata/ocn/pop/gx1v7/grid/topography_20161215.ieeei4',
 'region_mask_fname': 'inputdata/ocn/pop/gx1v7/grid/region_mask_20151008.ieeei4',
 'type': 'dipole',
 'region_mask_regions': {'Black Sea': -13,
  'Baltic Sea': -12,
  'Red Sea': -5,
  'Southern Ocean': 1,
  'Pacific Ocean': 2,
  'Indian Ocean': 3,
  'Persian Gulf': 4,
  'Atlantic Ocean': 6,
  'Mediterranean Sea': 7,
  'Lab. Sea & Baffin Bay': 8,
  'GIN Seas': 9,
  'Arctic Ocean': 10,
  'Hudson Bay': 11}}

In [4]: from pop_tools.grid import INPUTDATA

In [5]: from pop_tools.datasets import fetch

In [6]: path = fetch(INPUTDATA, fname=example['horiz_grid_fname'])

In [7]: path
Out[7]: '/glade/p/cesmdata/cseg/inputdata/ocn/pop/gx1v7/grid/horiz_grid_20010402.ieeer8'

Copy link
Collaborator

@matt-long matt-long left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andersy005, thanks for your prompt response to this issue.

One thing I find annoying is the statement when importing pop_tools, that the intputdata directory is not writable. Will this PR remove that message?

If inputdata is not writable, we effectively have to assume that all files are present. I see two potential design responses to this:

  1. Explicitly switch-off pooch and make fetch use direct access.
  2. Have the option of multiple cache directories in order of preference, if a file is not found in 1, use pooch to download it to 2.

What do you think?

pop_tools/datasets.py Outdated Show resolved Hide resolved
@andersy005 andersy005 changed the title Update pooch downloader Update pooch downloader: use CESMDATAROOT variable when available Dec 15, 2020
@andersy005 andersy005 merged commit bd1236c into NCAR:master Dec 15, 2020
@andersy005 andersy005 deleted the update-pooch-downloader branch December 15, 2020 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pop_tools trying to download files already local to cheyenne
2 participants