-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add variable mapping of psm3 #1374
Conversation
It seems the However, should the psm zenith column map then be mapped to 'apparent_zenith' or 'apparent_solar_zenith'? The pvliv Variables and Symbols List has 'solar_zenith' as the standard name, which is also what zenith is mapped to in the get_cams() function. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please also add 2020
, tmy-2020
, tdy-2020
, and tgy-2020
to the docstring tables? :)
amap = {value: key.lower().replace(' ', '_') for (key, value) in | ||
PSM3_VARIABLE_MAP.items()} | ||
attributes = [a if a not in amap.keys() else amap[a] for a in attributes] | ||
attributes = list(set(attributes)) # remove duplicate values |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is interesting. Do we do "reverse mapping" in any other iotools functions? It is unfortunate that the PSM3 API's input parameter names are different from the output column names.
Maybe clearer for the second line, up to you: attributes = [amap.get(a, a) for a in attributes]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we keep this, it should probably be mentioned in the docstring
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I cannot immediately think of any other function where reverse mapping would make sense. Pretty nifty though! And I think it is in line with the spirit of the iotools, in that it conforms the data access interface with pvlib conventions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@AdamRJensen should we add a sentence to the attributes
docstring description for this? Something like (feel free to edit):
Alternatively, pvlib names may also be used (e.g.
'ghi'
rather than'GHI'
); see :const:PSM3_VARIABLE_MAP
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we're referencing the iotools variable maps via :const:
then we should think about giving them their own entries in the docs. Something like what we did for pvlib.temperature.TEMPERATURE_MODEL_PARAMETERS
maybe? Let's do that in a separate issue though.
Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
amap = {value: key.lower().replace(' ', '_') for (key, value) in | ||
PSM3_VARIABLE_MAP.items()} | ||
attributes = [a if a not in amap.keys() else amap[a] for a in attributes] | ||
attributes = list(set(attributes)) # remove duplicate values |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@AdamRJensen should we add a sentence to the attributes
docstring description for this? Something like (feel free to edit):
Alternatively, pvlib names may also be used (e.g.
'ghi'
rather than'GHI'
); see :const:PSM3_VARIABLE_MAP
.
amap = {value: key.lower().replace(' ', '_') for (key, value) in | ||
PSM3_VARIABLE_MAP.items()} | ||
attributes = [a if a not in amap.keys() else amap[a] for a in attributes] | ||
attributes = list(set(attributes)) # remove duplicate values |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we're referencing the iotools variable maps via :const:
then we should think about giving them their own entries in the docs. Something like what we did for pvlib.temperature.TEMPERATURE_MODEL_PARAMETERS
maybe? Let's do that in a separate issue though.
@AdamRJensen sorry for bombing this issue with irrelevant request, but would you be interested in showing your work on IO tools at SciPy conference? https://www.scipy2022.scipy.org/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @AdamRJensen! I am personally excited to not have to remember anymore that get_psm3
returns GHI
instead of ghi
:)
pvlib/iotools/psm3.py
Outdated
'Clearsky GHI': 'ghi_clear', | ||
'Clearsky DHI': 'dhi_clear', | ||
'Clearsky DNI': 'dni_clear', | ||
'Solar Zenith Angle': 'apparent_zenith', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, maybe we should hold off merging for a moment -- #717 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since a decision has been made, I have renamed apparent_zenith
to solar_zenith
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok with me to merge pending resolution of altitude/elevation comment below.
pvlib/iotools/psm3.py
Outdated
data = data.rename(columns=VARIABLE_MAP) | ||
metadata['latitude'] = metadata.pop('Latitude') | ||
metadata['longitude'] = metadata.pop('Longitude') | ||
metadata['elevation'] = metadata.pop('Elevation') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pvlib uses altitude
rather than elevation
, so I think this line should be metadata['altitude'] = metadata.pop('Elevation')
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting... I'll make a follow-up issue requesting that this term be added to the Variables and Symbols list.
As it is right now, it would be hard for users to figure out that map_variables
return altitude instead of elevation. Is there a good way to denote that? E.g., adding it to the VARIABLE_MAP
?
data, metadata = psm3.read_psm3(MANUAL_TEST_DATA, map_variables=True) | ||
columns_mapped = ['Year', 'Month', 'Day', 'Hour', 'Minute', 'dhi', 'dni', | ||
'ghi', 'dhi_clear', 'dni_clear', 'ghi_clear', | ||
'Cloud Type', 'Dew Point', 'apparent_zenith', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@AdamRJensen this test is failing locally for me because of an apparent_zenith
/solar_zenith
difference. Shouldn't this be solar_zenith
, and shouldn't this test have failed before we merged the PR? I'm very confused why this wasn't failing before.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes indeed. Sorry for the slow response. Thanks for the fix!
commit 5047b26 Author: Prajwal Borkar <48290911+PrajwalBorkar@users.noreply.github.com> Date: Tue May 17 19:14:53 2022 +0530 Updated get_cams protocol to https pvlib#1457 (pvlib#1458) * Updated get_cams protocol to https pvlib#1457 * Updated instances of http to https. pvlib#1457 * Updated documentation links to https * Added Contributor commit a0812b1 Author: roger-lcc <58332996+roger-lcc@users.noreply.github.com> Date: Wed May 4 20:01:42 2022 +0800 CI asv check (pvlib#1454) * CI asv check * added CI asv check * CI asv check * CI asv check * updated CI asv check * Update docs/sphinx/source/whatsnew/v0.9.2.rst updated v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 83e379a Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Apr 28 19:26:09 2022 -0400 Bump pandas to 0.25.0; test updates (pvlib#1448) * bump pandas min from 0.22.0 to 0.25.0 * fix buggy test__check_pandas_assert_kwargs don't use monkeypatch and mocker in the same test function. pytest-dev/pytest-mock#289 * fix psm3 test (apparent_zenith -> solar_zenith) * whatsnew * better UTC conversion in sun_rise_set_transit_ephem * helpful comments * more whatsnew * '3.0' -> '3' in read_crn test? * apply dtypes during parsing in read_crn * move dropna() post-processing into read_fwf call * fix read_crn for pandas<1.2.0 * Update pvlib/solarposition.py Co-authored-by: Will Holmgren <william.holmgren@gmail.com> * nix pytz * UTC -> utc * address simd arccos issue in tracking.singleaxis Co-authored-by: Will Holmgren <william.holmgren@gmail.com> commit 8d0f863 Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Tue Apr 12 22:55:58 2022 +0530 Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (pvlib#1440) * Advance numba from 0.36.1 to 0.40.0 * Advance numba from 0.36.1 to 0.40.0 * Updated whatsnew.rst commit 5cb695d Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Wed Apr 6 23:58:03 2022 +0530 Remove unnecessary **kwargs from spa_python and get_total_irradiance (pvlib#1437) * Update Solarposition.py Removed **kwargs from pvlib.solarposition.spa_python * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235) * Update docs/sphinx/source/whatsnew/v0.9.2.rst * Update docs/sphinx/source/whatsnew/v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 8460b36 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 15:31:25 2022 -0600 Finalize 0.9.1 (pvlib#1431) * fix heading levels in user_guide/bifacial.rst * whatsnew cleanup * fix readme html missing tag, maybe unicode zero-width spaces? * readme: link to universal zenodo doi * readme: update installation link for pvlib#1173 * whatsnew date * additional contributors * delete errant space commit edbf2a6 Author: RoyCoding8 <92641125+RoyCoding8@users.noreply.github.com> Date: Wed Mar 30 01:58:18 2022 +0530 Updated plot_singlediode.py (pvlib#1434) * Update plot_singlediode.py Changed the unit from C to degree C (°C) * Update plot_singlediode.py Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters. * Update v0.9.1.rst Added name to the contributors' list * Update v0.9.1.rst commit cf4a8ad Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 14:04:40 2022 -0600 Update sphinx to 4.5.0 (pvlib#1435) * bump sphinx and pydata-sphinx-theme versions * clean up sphinx conf.py * fix distutils strangeness, maybe * use freshly-released sphinx==4.5.0 commit 884a153 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Wed Mar 23 13:41:35 2022 -0600 Clarify delta_t docstring descriptions (pvlib#1429) * clarify delta_t docstrings * whatsnew commit c243183 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 12:01:57 2022 -0600 Deprecate pvlib.forecast (pvlib#1426) * deprecate pvlib.forecast classes * catch warnings in tests * add warning admonition to forecasts.rst * whatsnew * stickler * pin pytest < 7.1.0 * pin pytest in the right place this time * more warning suppression in tests * unpin pytest * Update docs/sphinx/source/whatsnew/v0.9.1.rst * copy warning to reference/forecasting.rst commit e3baa12 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 11:28:56 2022 -0600 Fix conditional dependency on dataclasses (pvlib#1422) * better conditional dependency on dataclasses * whatsnew commit 27cba7a Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Thu Mar 17 22:48:08 2022 +0530 Added asv benchmarking badge to the table of badges in the main README. (pvlib#1427) * Update Readme.md Added benchmarks asv badge to the badge section * Updated v.0.9.1.rst Added my name to the list of Contributers. commit 1893b20 Author: Adam R. Jensen <39184289+AdamRJensen@users.noreply.github.com> Date: Mon Mar 14 18:37:58 2022 +0100 Add variable mapping of psm3 (pvlib#1374) * Add variable mapping of psm3 * Add enhancement entry in whatsnew * Fix stickler * Map keys in metadata dict * Remove double spaces in docs * Fix stickler * Doc update Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> * Reformatting - changes by kanderso-nrel * Update docstring table with 2020 * Add deprecation warning test coverage * Rename to VARIABLE_MAP * Change apparent_zenith to solar_zenith Based on the decision in pvlib#1403 * Update attributes docstring * Change elevation to altitude when mapping variables * Update psm3 variable mapping test Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
* Add cams.get_cams_radiation function * Revert "Add cams.get_cams_radiation function" This reverts commit d7deb80. * Allow parsing of http files * Add test for https file * Squashed commit of the following: commit 5047b26 Author: Prajwal Borkar <48290911+PrajwalBorkar@users.noreply.github.com> Date: Tue May 17 19:14:53 2022 +0530 Updated get_cams protocol to https #1457 (#1458) * Updated get_cams protocol to https #1457 * Updated instances of http to https. #1457 * Updated documentation links to https * Added Contributor commit a0812b1 Author: roger-lcc <58332996+roger-lcc@users.noreply.github.com> Date: Wed May 4 20:01:42 2022 +0800 CI asv check (#1454) * CI asv check * added CI asv check * CI asv check * CI asv check * updated CI asv check * Update docs/sphinx/source/whatsnew/v0.9.2.rst updated v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 83e379a Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Apr 28 19:26:09 2022 -0400 Bump pandas to 0.25.0; test updates (#1448) * bump pandas min from 0.22.0 to 0.25.0 * fix buggy test__check_pandas_assert_kwargs don't use monkeypatch and mocker in the same test function. pytest-dev/pytest-mock#289 * fix psm3 test (apparent_zenith -> solar_zenith) * whatsnew * better UTC conversion in sun_rise_set_transit_ephem * helpful comments * more whatsnew * '3.0' -> '3' in read_crn test? * apply dtypes during parsing in read_crn * move dropna() post-processing into read_fwf call * fix read_crn for pandas<1.2.0 * Update pvlib/solarposition.py Co-authored-by: Will Holmgren <william.holmgren@gmail.com> * nix pytz * UTC -> utc * address simd arccos issue in tracking.singleaxis Co-authored-by: Will Holmgren <william.holmgren@gmail.com> commit 8d0f863 Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Tue Apr 12 22:55:58 2022 +0530 Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (#1440) * Advance numba from 0.36.1 to 0.40.0 * Advance numba from 0.36.1 to 0.40.0 * Updated whatsnew.rst commit 5cb695d Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Wed Apr 6 23:58:03 2022 +0530 Remove unnecessary **kwargs from spa_python and get_total_irradiance (#1437) * Update Solarposition.py Removed **kwargs from pvlib.solarposition.spa_python * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235) * Update docs/sphinx/source/whatsnew/v0.9.2.rst * Update docs/sphinx/source/whatsnew/v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 8460b36 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 15:31:25 2022 -0600 Finalize 0.9.1 (#1431) * fix heading levels in user_guide/bifacial.rst * whatsnew cleanup * fix readme html missing tag, maybe unicode zero-width spaces? * readme: link to universal zenodo doi * readme: update installation link for #1173 * whatsnew date * additional contributors * delete errant space commit edbf2a6 Author: RoyCoding8 <92641125+RoyCoding8@users.noreply.github.com> Date: Wed Mar 30 01:58:18 2022 +0530 Updated plot_singlediode.py (#1434) * Update plot_singlediode.py Changed the unit from C to degree C (°C) * Update plot_singlediode.py Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters. * Update v0.9.1.rst Added name to the contributors' list * Update v0.9.1.rst commit cf4a8ad Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 14:04:40 2022 -0600 Update sphinx to 4.5.0 (#1435) * bump sphinx and pydata-sphinx-theme versions * clean up sphinx conf.py * fix distutils strangeness, maybe * use freshly-released sphinx==4.5.0 commit 884a153 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Wed Mar 23 13:41:35 2022 -0600 Clarify delta_t docstring descriptions (#1429) * clarify delta_t docstrings * whatsnew commit c243183 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 12:01:57 2022 -0600 Deprecate pvlib.forecast (#1426) * deprecate pvlib.forecast classes * catch warnings in tests * add warning admonition to forecasts.rst * whatsnew * stickler * pin pytest < 7.1.0 * pin pytest in the right place this time * more warning suppression in tests * unpin pytest * Update docs/sphinx/source/whatsnew/v0.9.1.rst * copy warning to reference/forecasting.rst commit e3baa12 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 11:28:56 2022 -0600 Fix conditional dependency on dataclasses (#1422) * better conditional dependency on dataclasses * whatsnew commit 27cba7a Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Thu Mar 17 22:48:08 2022 +0530 Added asv benchmarking badge to the table of badges in the main README. (#1427) * Update Readme.md Added benchmarks asv badge to the badge section * Updated v.0.9.1.rst Added my name to the list of Contributers. commit 1893b20 Author: Adam R. Jensen <39184289+AdamRJensen@users.noreply.github.com> Date: Mon Mar 14 18:37:58 2022 +0100 Add variable mapping of psm3 (#1374) * Add variable mapping of psm3 * Add enhancement entry in whatsnew * Fix stickler * Map keys in metadata dict * Remove double spaces in docs * Fix stickler * Doc update Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> * Reformatting - changes by kanderso-nrel * Update docstring table with 2020 * Add deprecation warning test coverage * Rename to VARIABLE_MAP * Change apparent_zenith to solar_zenith Based on the decision in #1403 * Update attributes docstring * Change elevation to altitude when mapping variables * Update psm3 variable mapping test Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> * Revert "Squashed commit of the following:" This reverts commit b313c64. * Update whatsnew * Update read_surfrad documentation Co-authored-by: AdamRJensen <arajen@byg.dtu.dk>
[ ] Closes #xxxx[ ] Updates entries todocs/sphinx/source/api.rst
for API changes.docs/sphinx/source/whatsnew
for all changes. Includes link to the GitHub Issue with:issue:`num`
or this Pull Request with:pull:`num`
. Includes contributor name and/or GitHub username (link with:ghuser:`user`
).Add
map_variables
argument toget_psm3
andread_psm3
functions to be consistent with the general iotools function pattern.At the moment
map_variables
defaults to None (False) and a deprecation warning has been added. The PR is similar to what was done forget_pvgis_tmy
in #1268.Additionally, I have implemented it such that pvlib names can be used when specifying the
attributes
arguments (requested columns), using the variable map in reverse.