Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test for latest version resolver #874

Merged
merged 6 commits into from
Dec 11, 2020
Merged

Add test for latest version resolver #874

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fe162d8
Simplify latest version resolver
stefsmeets Nov 20, 2020
195ad2f
Add test for resolving latest version
stefsmeets Nov 23, 2020
c74ec17
Fix possibly undefined loop variable
stefsmeets Nov 23, 2020
58d3e2b
Revert "Simplify latest version resolver"
stefsmeets Dec 1, 2020
01b2529
Add reminder to avoid globbing on large clusters
stefsmeets Dec 1, 2020
c2177d8
Merge branch 'master' into simplify_latestversion_resolver
stefsmeets Dec 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion esmvalcore/_data_finder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,11 @@ def _apply_caps(original, lower, upper):


def _resolve_latestversion(dirname_template):
"""Resolve the 'latestversion' tag."""
"""Resolve the 'latestversion' tag.

This implementation avoid globbing on centralized clusters with very
large data root dirs (i.e. ESGF nodes like Jasmin/DKRZ).
"""
if '{latestversion}' not in dirname_template:
return dirname_template

Expand Down
21 changes: 21 additions & 0 deletions tests/integration/data_finder.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,27 @@ get_input_filelist:
- ta_Amon_HadGEM2-ES_historical_r1i1p1*.nc
found_files: []

- drs: BADC
variable:
<<: *variable
start_year: 1980
end_year: 2002
available_files:
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20110329/ta/ta_Amon_HadGEM2-ES_historical_r1i1p1_193412-195911.nc
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20110329/ta/ta_Amon_HadGEM2-ES_historical_r1i1p1_195912-198411.nc
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20110329/ta/ta_Amon_HadGEM2-ES_historical_r1i1p1_198412-200511.nc
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20120928/ta/ta_Amon_HadGEM2-ES_historical_r1i1p1_193412-195911.nc
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20120928/ta/ta_Amon_HadGEM2-ES_historical_r1i1p1_195912-198411.nc
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20120928/ta/ta_Amon_HadGEM2-ES_historical_r1i1p1_198412-200511.nc
dirs:
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20120928/ta
file_patterns:
- ta_Amon_HadGEM2-ES_historical_r1i1p1*.nc
found_files:
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20120928/ta/ta_Amon_HadGEM2-ES_historical_r1i1p1_195912-198411.nc
- MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r1i1p1/v20120928/ta/ta_Amon_HadGEM2-ES_historical_r1i1p1_198412-200511.nc


- drs: BADC
variable:
<<: *variable
Expand Down