Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add variable mapping of psm3 #1374

Merged
merged 20 commits into from
Mar 14, 2022
Merged

Conversation

AdamRJensen
Copy link
Member

@AdamRJensen AdamRJensen commented Jan 8, 2022

  • [ ] Closes #xxxx
  • I am familiar with the contributing guidelines
  • Tests added
  • [ ] Updates entries to docs/sphinx/source/api.rst for API changes.
  • Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
  • New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
  • Pull request is nearly complete and ready for detailed review.
  • Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

Add map_variables argument to get_psm3 and read_psm3 functions to be consistent with the general iotools function pattern.

At the moment map_variables defaults to None (False) and a deprecation warning has been added. The PR is similar to what was done for get_pvgis_tmy in #1268.

Additionally, I have implemented it such that pvlib names can be used when specifying the attributes arguments (requested columns), using the variable map in reverse.

@AdamRJensen
Copy link
Member Author

It seems the zenith column returned by psm3 is "apparent_zenith", see comparison below:

image

However, should the psm zenith column map then be mapped to 'apparent_zenith' or 'apparent_solar_zenith'?

The pvliv Variables and Symbols List has 'solar_zenith' as the standard name, which is also what zenith is mapped to in the get_cams() function.

Copy link
Member

@kandersolar kandersolar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please also add 2020, tmy-2020, tdy-2020, and tgy-2020 to the docstring tables? :)

pvlib/iotools/psm3.py Outdated Show resolved Hide resolved
pvlib/iotools/psm3.py Outdated Show resolved Hide resolved
pvlib/iotools/psm3.py Show resolved Hide resolved
amap = {value: key.lower().replace(' ', '_') for (key, value) in
PSM3_VARIABLE_MAP.items()}
attributes = [a if a not in amap.keys() else amap[a] for a in attributes]
attributes = list(set(attributes)) # remove duplicate values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting. Do we do "reverse mapping" in any other iotools functions? It is unfortunate that the PSM3 API's input parameter names are different from the output column names.

Maybe clearer for the second line, up to you: attributes = [amap.get(a, a) for a in attributes]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we keep this, it should probably be mentioned in the docstring

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot immediately think of any other function where reverse mapping would make sense. Pretty nifty though! And I think it is in line with the spirit of the iotools, in that it conforms the data access interface with pvlib conventions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AdamRJensen should we add a sentence to the attributes docstring description for this? Something like (feel free to edit):

Alternatively, pvlib names may also be used (e.g. 'ghi' rather than 'GHI'); see :const:PSM3_VARIABLE_MAP.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're referencing the iotools variable maps via :const: then we should think about giving them their own entries in the docs. Something like what we did for pvlib.temperature.TEMPERATURE_MODEL_PARAMETERS maybe? Let's do that in a separate issue though.

pvlib/iotools/psm3.py Outdated Show resolved Hide resolved
pvlib/iotools/psm3.py Outdated Show resolved Hide resolved
amap = {value: key.lower().replace(' ', '_') for (key, value) in
PSM3_VARIABLE_MAP.items()}
attributes = [a if a not in amap.keys() else amap[a] for a in attributes]
attributes = list(set(attributes)) # remove duplicate values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AdamRJensen should we add a sentence to the attributes docstring description for this? Something like (feel free to edit):

Alternatively, pvlib names may also be used (e.g. 'ghi' rather than 'GHI'); see :const:PSM3_VARIABLE_MAP.

pvlib/iotools/psm3.py Outdated Show resolved Hide resolved
amap = {value: key.lower().replace(' ', '_') for (key, value) in
PSM3_VARIABLE_MAP.items()}
attributes = [a if a not in amap.keys() else amap[a] for a in attributes]
attributes = list(set(attributes)) # remove duplicate values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're referencing the iotools variable maps via :const: then we should think about giving them their own entries in the docs. Something like what we did for pvlib.temperature.TEMPERATURE_MODEL_PARAMETERS maybe? Let's do that in a separate issue though.

@mikofski
Copy link
Member

@AdamRJensen sorry for bombing this issue with irrelevant request, but would you be interested in showing your work on IO tools at SciPy conference? https://www.scipy2022.scipy.org/

Copy link
Member

@kandersolar kandersolar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @AdamRJensen! I am personally excited to not have to remember anymore that get_psm3 returns GHI instead of ghi :)

'Clearsky GHI': 'ghi_clear',
'Clearsky DHI': 'dhi_clear',
'Clearsky DNI': 'dni_clear',
'Solar Zenith Angle': 'apparent_zenith',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, maybe we should hold off merging for a moment -- #717 (comment)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since a decision has been made, I have renamed apparent_zenith to solar_zenith.

Copy link
Member

@wholmgren wholmgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok with me to merge pending resolution of altitude/elevation comment below.

data = data.rename(columns=VARIABLE_MAP)
metadata['latitude'] = metadata.pop('Latitude')
metadata['longitude'] = metadata.pop('Longitude')
metadata['elevation'] = metadata.pop('Elevation')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pvlib uses altitude rather than elevation, so I think this line should be metadata['altitude'] = metadata.pop('Elevation')

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting... I'll make a follow-up issue requesting that this term be added to the Variables and Symbols list.

As it is right now, it would be hard for users to figure out that map_variables return altitude instead of elevation. Is there a good way to denote that? E.g., adding it to the VARIABLE_MAP?

@cwhanse cwhanse mentioned this pull request Mar 14, 2022
14 tasks
@kandersolar kandersolar merged commit 1893b20 into pvlib:master Mar 14, 2022
@AdamRJensen AdamRJensen deleted the map_variables_psm3 branch March 14, 2022 17:39
data, metadata = psm3.read_psm3(MANUAL_TEST_DATA, map_variables=True)
columns_mapped = ['Year', 'Month', 'Day', 'Hour', 'Minute', 'dhi', 'dni',
'ghi', 'dhi_clear', 'dni_clear', 'ghi_clear',
'Cloud Type', 'Dew Point', 'apparent_zenith',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AdamRJensen this test is failing locally for me because of an apparent_zenith/solar_zenith difference. Shouldn't this be solar_zenith, and shouldn't this test have failed before we merged the PR? I'm very confused why this wasn't failing before.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes indeed. Sorry for the slow response. Thanks for the fix!

AdamRJensen added a commit to AdamRJensen/pvlib-python that referenced this pull request May 19, 2022
commit 5047b26
Author: Prajwal Borkar <48290911+PrajwalBorkar@users.noreply.github.com>
Date:   Tue May 17 19:14:53 2022 +0530

    Updated get_cams protocol to https pvlib#1457 (pvlib#1458)

    * Updated get_cams protocol to https pvlib#1457

    * Updated instances of http to https. pvlib#1457

    * Updated documentation links to https

    * Added Contributor

commit a0812b1
Author: roger-lcc <58332996+roger-lcc@users.noreply.github.com>
Date:   Wed May 4 20:01:42 2022 +0800

    CI asv check (pvlib#1454)

    * CI asv check

    * added CI asv check

    * CI asv check

    * CI asv check

    * updated CI asv check

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    updated v0.9.2.rst

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

commit 83e379a
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Apr 28 19:26:09 2022 -0400

    Bump pandas to 0.25.0; test updates (pvlib#1448)

    * bump pandas min from 0.22.0 to 0.25.0

    * fix buggy test__check_pandas_assert_kwargs

    don't use monkeypatch and mocker in the same test function.  pytest-dev/pytest-mock#289

    * fix psm3 test (apparent_zenith -> solar_zenith)

    * whatsnew

    * better UTC conversion in sun_rise_set_transit_ephem

    * helpful comments

    * more whatsnew

    * '3.0' -> '3' in read_crn test?

    * apply dtypes during parsing in read_crn

    * move dropna() post-processing into read_fwf call

    * fix read_crn for pandas<1.2.0

    * Update pvlib/solarposition.py

    Co-authored-by: Will Holmgren <william.holmgren@gmail.com>

    * nix pytz

    * UTC -> utc

    * address simd arccos issue in tracking.singleaxis

    Co-authored-by: Will Holmgren <william.holmgren@gmail.com>

commit 8d0f863
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Tue Apr 12 22:55:58 2022 +0530

    Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (pvlib#1440)

    * Advance numba from 0.36.1 to 0.40.0

    * Advance numba from 0.36.1 to 0.40.0

    * Updated whatsnew.rst

commit 5cb695d
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Wed Apr 6 23:58:03 2022 +0530

    Remove unnecessary **kwargs from spa_python and get_total_irradiance (pvlib#1437)

    * Update Solarposition.py

    Removed **kwargs from pvlib.solarposition.spa_python

    * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py

    Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235)

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

commit 8460b36
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Tue Mar 29 15:31:25 2022 -0600

    Finalize 0.9.1 (pvlib#1431)

    * fix heading levels in user_guide/bifacial.rst

    * whatsnew cleanup

    * fix readme html

    missing tag, maybe unicode zero-width spaces?

    * readme: link to universal zenodo doi

    * readme: update installation link for pvlib#1173

    * whatsnew date

    * additional contributors

    * delete errant space

commit edbf2a6
Author: RoyCoding8 <92641125+RoyCoding8@users.noreply.github.com>
Date:   Wed Mar 30 01:58:18 2022 +0530

    Updated plot_singlediode.py (pvlib#1434)

    * Update plot_singlediode.py

    Changed the unit from C to degree C (°C)

    * Update plot_singlediode.py

    Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters.

    * Update v0.9.1.rst

    Added name to the contributors' list

    * Update v0.9.1.rst

commit cf4a8ad
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Tue Mar 29 14:04:40 2022 -0600

    Update sphinx to 4.5.0 (pvlib#1435)

    * bump sphinx and pydata-sphinx-theme versions

    * clean up sphinx conf.py

    * fix distutils strangeness, maybe

    * use freshly-released sphinx==4.5.0

commit 884a153
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Wed Mar 23 13:41:35 2022 -0600

    Clarify delta_t docstring descriptions (pvlib#1429)

    * clarify delta_t docstrings

    * whatsnew

commit c243183
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Mar 17 12:01:57 2022 -0600

    Deprecate pvlib.forecast (pvlib#1426)

    * deprecate pvlib.forecast classes

    * catch warnings in tests

    * add warning admonition to forecasts.rst

    * whatsnew

    * stickler

    * pin pytest < 7.1.0

    * pin pytest in the right place this time

    * more warning suppression in tests

    * unpin pytest

    * Update docs/sphinx/source/whatsnew/v0.9.1.rst

    * copy warning to reference/forecasting.rst

commit e3baa12
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Mar 17 11:28:56 2022 -0600

    Fix conditional dependency on dataclasses (pvlib#1422)

    * better conditional dependency on dataclasses

    * whatsnew

commit 27cba7a
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Thu Mar 17 22:48:08 2022 +0530

    Added asv benchmarking badge to the table of badges in the main README. (pvlib#1427)

    * Update Readme.md

    Added benchmarks asv badge to the badge section

    * Updated v.0.9.1.rst

    Added my name to the list of Contributers.

commit 1893b20
Author: Adam R. Jensen <39184289+AdamRJensen@users.noreply.github.com>
Date:   Mon Mar 14 18:37:58 2022 +0100

    Add variable mapping of psm3 (pvlib#1374)

    * Add variable mapping of psm3

    * Add enhancement entry in whatsnew

    * Fix stickler

    * Map keys in metadata dict

    * Remove double spaces in docs

    * Fix stickler

    * Doc update

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

    * Reformatting - changes by kanderso-nrel

    * Update docstring table with 2020

    * Add deprecation warning test coverage

    * Rename to VARIABLE_MAP

    * Change apparent_zenith to solar_zenith

    Based on the decision in pvlib#1403

    * Update attributes docstring

    * Change elevation to altitude when mapping variables

    * Update psm3 variable mapping test

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
kandersolar pushed a commit that referenced this pull request Jul 19, 2022
* Add cams.get_cams_radiation function

* Revert "Add cams.get_cams_radiation function"

This reverts commit d7deb80.

* Allow parsing of http files

* Add test for https file

* Squashed commit of the following:

commit 5047b26
Author: Prajwal Borkar <48290911+PrajwalBorkar@users.noreply.github.com>
Date:   Tue May 17 19:14:53 2022 +0530

    Updated get_cams protocol to https #1457 (#1458)

    * Updated get_cams protocol to https #1457

    * Updated instances of http to https. #1457

    * Updated documentation links to https

    * Added Contributor

commit a0812b1
Author: roger-lcc <58332996+roger-lcc@users.noreply.github.com>
Date:   Wed May 4 20:01:42 2022 +0800

    CI asv check (#1454)

    * CI asv check

    * added CI asv check

    * CI asv check

    * CI asv check

    * updated CI asv check

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    updated v0.9.2.rst

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

commit 83e379a
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Apr 28 19:26:09 2022 -0400

    Bump pandas to 0.25.0; test updates (#1448)

    * bump pandas min from 0.22.0 to 0.25.0

    * fix buggy test__check_pandas_assert_kwargs

    don't use monkeypatch and mocker in the same test function.  pytest-dev/pytest-mock#289

    * fix psm3 test (apparent_zenith -> solar_zenith)

    * whatsnew

    * better UTC conversion in sun_rise_set_transit_ephem

    * helpful comments

    * more whatsnew

    * '3.0' -> '3' in read_crn test?

    * apply dtypes during parsing in read_crn

    * move dropna() post-processing into read_fwf call

    * fix read_crn for pandas<1.2.0

    * Update pvlib/solarposition.py

    Co-authored-by: Will Holmgren <william.holmgren@gmail.com>

    * nix pytz

    * UTC -> utc

    * address simd arccos issue in tracking.singleaxis

    Co-authored-by: Will Holmgren <william.holmgren@gmail.com>

commit 8d0f863
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Tue Apr 12 22:55:58 2022 +0530

    Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (#1440)

    * Advance numba from 0.36.1 to 0.40.0

    * Advance numba from 0.36.1 to 0.40.0

    * Updated whatsnew.rst

commit 5cb695d
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Wed Apr 6 23:58:03 2022 +0530

    Remove unnecessary **kwargs from spa_python and get_total_irradiance (#1437)

    * Update Solarposition.py

    Removed **kwargs from pvlib.solarposition.spa_python

    * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py

    Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235)

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

commit 8460b36
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Tue Mar 29 15:31:25 2022 -0600

    Finalize 0.9.1 (#1431)

    * fix heading levels in user_guide/bifacial.rst

    * whatsnew cleanup

    * fix readme html

    missing tag, maybe unicode zero-width spaces?

    * readme: link to universal zenodo doi

    * readme: update installation link for #1173

    * whatsnew date

    * additional contributors

    * delete errant space

commit edbf2a6
Author: RoyCoding8 <92641125+RoyCoding8@users.noreply.github.com>
Date:   Wed Mar 30 01:58:18 2022 +0530

    Updated plot_singlediode.py (#1434)

    * Update plot_singlediode.py

    Changed the unit from C to degree C (°C)

    * Update plot_singlediode.py

    Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters.

    * Update v0.9.1.rst

    Added name to the contributors' list

    * Update v0.9.1.rst

commit cf4a8ad
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Tue Mar 29 14:04:40 2022 -0600

    Update sphinx to 4.5.0 (#1435)

    * bump sphinx and pydata-sphinx-theme versions

    * clean up sphinx conf.py

    * fix distutils strangeness, maybe

    * use freshly-released sphinx==4.5.0

commit 884a153
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Wed Mar 23 13:41:35 2022 -0600

    Clarify delta_t docstring descriptions (#1429)

    * clarify delta_t docstrings

    * whatsnew

commit c243183
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Mar 17 12:01:57 2022 -0600

    Deprecate pvlib.forecast (#1426)

    * deprecate pvlib.forecast classes

    * catch warnings in tests

    * add warning admonition to forecasts.rst

    * whatsnew

    * stickler

    * pin pytest < 7.1.0

    * pin pytest in the right place this time

    * more warning suppression in tests

    * unpin pytest

    * Update docs/sphinx/source/whatsnew/v0.9.1.rst

    * copy warning to reference/forecasting.rst

commit e3baa12
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Mar 17 11:28:56 2022 -0600

    Fix conditional dependency on dataclasses (#1422)

    * better conditional dependency on dataclasses

    * whatsnew

commit 27cba7a
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Thu Mar 17 22:48:08 2022 +0530

    Added asv benchmarking badge to the table of badges in the main README. (#1427)

    * Update Readme.md

    Added benchmarks asv badge to the badge section

    * Updated v.0.9.1.rst

    Added my name to the list of Contributers.

commit 1893b20
Author: Adam R. Jensen <39184289+AdamRJensen@users.noreply.github.com>
Date:   Mon Mar 14 18:37:58 2022 +0100

    Add variable mapping of psm3 (#1374)

    * Add variable mapping of psm3

    * Add enhancement entry in whatsnew

    * Fix stickler

    * Map keys in metadata dict

    * Remove double spaces in docs

    * Fix stickler

    * Doc update

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

    * Reformatting - changes by kanderso-nrel

    * Update docstring table with 2020

    * Add deprecation warning test coverage

    * Rename to VARIABLE_MAP

    * Change apparent_zenith to solar_zenith

    Based on the decision in #1403

    * Update attributes docstring

    * Change elevation to altitude when mapping variables

    * Update psm3 variable mapping test

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

* Revert "Squashed commit of the following:"

This reverts commit b313c64.

* Update whatsnew

* Update read_surfrad documentation

Co-authored-by: AdamRJensen <arajen@byg.dtu.dk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants