Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature 1319 no pickle #1699

Merged
merged 43 commits into from
Mar 15, 2021
Merged

Feature 1319 no pickle #1699

merged 43 commits into from
Mar 15, 2021

Conversation

dwfncar
Copy link
Contributor

@dwfncar dwfncar commented Mar 7, 2021

Pull Request Testing

The branch now how temporary pickle files replaced by NetCDF for the dataplane and ASCII for the ascii2nc python handler.
MPR pickle replacement will be encapsulated in a separate issue.
Note: This version is a temporary solution that will be superseded by the Python3_Module refactorization when complete.

  • Describe testing already performed for these changes:

  • Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

    Run python unit tests.

  • Do these changes include sufficient documentation and testing updates? [Yes or No]
    No.

  • Will this PR result in changes to the test suite? [Yes or No]

    If yes, describe the new output and/or changes to the existing output:

    No.

Pull Request Checklist

See the METplus Workflow for details.

  • Complete the PR definition above.
  • Ensure the PR title matches the feature or bugfix branch name.
  • Define the PR metadata, as permissions allow.
    Select: Reviewer(s), Project(s), and Milestone
  • After submitting the PR, select Linked Issues with the original issue number.
  • After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
  • Close the linked issue and delete your feature or bugfix branch from GitHub.

@JohnHalleyGotway JohnHalleyGotway added this to the MET 10.0.0 milestone Mar 8, 2021
Copy link
Collaborator

@JohnHalleyGotway JohnHalleyGotway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

David, I tried running unit_python.xml using these changes and almost got through all of them. However, I'm getting a runtime error on my Mac laptop. Perhaps it's version-specific?

Here's the 2nd to last command from those tests:

export MET_PYTHON_EXE='/usr/local/python3/bin/python3'
/Volumes/d1/projects/MET/MET_development/MET-python_pickle/met/share/met/../../bin/plot_data_plane \
      PYTHON_NUMPY \
      /Volumes/d1/projects/MET/MET_unit_test/MET_test_output/python/letter_pickle.ps \
      'name = "/Volumes/d1/projects/MET/MET_development/MET-python_pickle/met/share/met/python/read_ascii_numpy.py /Volumes/d1/projects/MET/MET_unit_test/MET_test_input/python/letter.txt LETTER";' \
      -plot_range 0.0 255.0 \
      -title "Python enabled plot_data_plane" \
      -v 1
unset MET_PYTHON_EXE

Here's the error I get:

ERROR  : 
ERROR  : tmp_nc_dataplane() -> command "/opt/local/bin/python3 /Volumes/d1/projects/MET/MET_development/MET-python_pickle/met/share/met/wrappers/write_tmp_dataplane.py /tmp/tmp_met_nc_36779_0 /Volumes/d1/projects/MET/MET_development/MET-python_pickle/met/share/met/python/read_ascii_numpy /Volumes/d1/projects/MET/MET_unit_test/MET_test_input/python/letter.txt LETTER" failed ... status = 256
ERROR  : 

And re-running that single python command manually, I get:

/opt/local/bin/python3 /Volumes/d1/projects/MET/MET_development/MET-python_pickle/met/share/met/wrappers/write_tmp_dataplane.py /tmp/tmp_met_nc_36779_0 /Volumes/d1/projects/MET/MET_development/MET-python_pickle/met/share/met/python/read_ascii_numpy /Volumes/d1/projects/MET/MET_unit_test/MET_test_input/python/letter.txt LETTER

Traceback (most recent call last):
  File "/Volumes/d1/projects/MET/MET_development/MET-python_pickle/met/share/met/wrappers/write_tmp_dataplane.py", line 59, in <module>
    setattr(ds, attr, attr_val)
  File "netCDF4/_netCDF4.pyx", line 2957, in netCDF4._netCDF4.Dataset.__setattr__
AttributeError: 'name' is one of the reserved attributes ('_grpid', '_grp', '_varid', 'groups', 'dimensions', 'variables', 'dtype', 'data_model', 'disk_format', '_nunlimdim', 'path', 'parent', 'ndim', 'mask', 'scale', 'cmptypes', 'vltypes', 'enumtypes', '_isprimitive', 'file_format', '_isvlen', '_isenum', '_iscompound', '_cmptype', '_vltype', '_enumtype', 'name', '__orthogoral_indexing__', 'keepweakref', '_has_lsd', '_buffer', 'chartostring', '_use_get_vars', '_ncstring_attrs__'), cannot rebind. Use setncattr instead.

Looks like a complaint about "name" being a reserved word. Please let me know if there's something else you'd like me to test.

@dwfncar
Copy link
Contributor Author

dwfncar commented Mar 12, 2021

Change _name in tmp netcdf file to name_str.
This is done in write_tmp_netcdf.py and read_tmp_netcdf.py wrappers.
Be sure to check the installed versions of these scripts to be sure they are treating the name attribute as special.

@dwfncar dwfncar merged commit 18fb169 into develop Mar 15, 2021
@JohnHalleyGotway JohnHalleyGotway mentioned this pull request Mar 31, 2021
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants