Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ValueError: Unable to create dataset (error during user callback) for python3 #49

Closed
jeenriquez opened this issue Aug 29, 2018 · 12 comments
Closed

ValueError: Unable to create dataset (error during user callback) for python3 #49

jeenriquez opened this issue Aug 29, 2018 · 12 comments
Assignees
@telegraphic @jeenriquez
Labels
bug

Comments

@jeenriquez
Copy link
Contributor

test_fil2h5py crashes, by using the latest version of h5py. Shown in this pip freeze:

astropy==2.0.4
attrs==17.4.0
backports.functools-lru-cache==1.5
bitshuffle==0.3.4
blimpy==1.3.4
cycler==0.10.0
Cython==0.28.1
funcsigs==1.0.2
h5py==2.8.0
matplotlib==2.1.2
numpy==1.14.2
pandas==0.22.0
pluggy==0.6.0
py==1.5.2
pyparsing==2.2.0
pytest==3.4.1
python-dateutil==2.6.1
pytz==2018.3
scipy==1.0.0
six==1.11.0
subprocess32==3.2.7
turbo-seti==0.7.3

this is the error message:

(eepy) eenriquez@blh0:~/software/bl-soft/blimpy/tests$ python test_fil2h5.py 
HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 139787481073408:
  #000: ../../../src/H5Pocpl.c line 1102 in H5Pget_filter_by_id2(): can't find object for ID
    major: Object atom
    minor: Unable to find atom information (already closed?)
  #001: ../../../src/H5Pint.c line 3381 in H5P_object_verify(): property list is not a member of the class
    major: Property lists
    minor: Unable to register new atom
  #002: ../../../src/H5Pint.c line 3331 in H5P_isa_class(): not a property list
    major: Invalid arguments to routine
    minor: Inappropriate type
Traceback (most recent call last):
  File "test_fil2h5.py", line 26, in <module>
    test_fil2h5_conversion()
  File "test_fil2h5.py", line 16, in test_fil2h5_conversion
    bl.fil2h5.make_h5_file('Voyager_data/Voyager1.single_coarse.fine_res.fil', new_filename = 'test.h5')
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/blimpy-1.3.4-py2.7.egg/blimpy/fil2h5.py", line 44, in make_h5_file
    fil_file.write_to_hdf5(new_filename)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/blimpy-1.3.4-py2.7.egg/blimpy/waterfall.py", line 302, in write_to_hdf5
    self.__write_to_hdf5_light(filename_out)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/blimpy-1.3.4-py2.7.egg/blimpy/waterfall.py", line 431, in __write_to_hdf5_light
    compression_opts=bs_compression_opts)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/h5py/_hl/group.py", line 116, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 140, in make_new_dset
    dset_id = h5d.create(parent.id, None, tid, sid, dcpl=dcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 79, in h5py.h5d.create
ValueError: Unable to create dataset (error during user callback)

I get test_fil2h5py to work, by using this h5py==2.5.0 instead.

Plus fixing this error by directly editing /opt/pyve/eepy/local/lib/python2.7/site-packages/h5py/_hl/dataset.py as suggested below. Oddly, this hackable issue I only have in the GBT cluster, but not in my laptop.

(eepy) eenriquez@blh0:~/software/bl-soft/blimpy/tests$ python test_fil2h5.py 
blimpy.waterfall INFO     Conversion time: 0.18sec
blimpy.file_wrapper WARNING  Selection size of 0.06 GB, exceeding our size limit 0.00 GB. Instance created, header loaded, but data not loaded, please try another (t,v) selection.
blimpy.waterfall INFO     Detecting high frequency resolution data.
Traceback (most recent call last):
  File "test_fil2h5.py", line 26, in <module>
    test_fil2h5_conversion()
  File "test_fil2h5.py", line 19, in test_fil2h5_conversion
    bl.fil2h5.make_h5_file('Voyager_data/Voyager1.single_coarse.fine_res.fil', new_filename = 'test_large.h5', max_load = 0.001)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/blimpy-1.3.4-py2.7.egg/blimpy/fil2h5.py", line 44, in make_h5_file
    fil_file.write_to_hdf5(new_filename)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/blimpy-1.3.4-py2.7.egg/blimpy/waterfall.py", line 300, in write_to_hdf5
    self.__write_to_hdf5_heavy(filename_out)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/blimpy-1.3.4-py2.7.egg/blimpy/waterfall.py", line 339, in __write_to_hdf5_heavy
    dtype=self.data.dtype)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/h5py/_hl/group.py", line 103, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
  File "/opt/pyve/eepy/local/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 70, in make_new_dset
    if isinstance(chunks, tuple) and (-numpy.array([ i>=j for i,j in zip(tmp_shape,chunks) if i is not None])).any():
TypeError: The numpy boolean negative, the `-` operator, is not supported, use the `~` operator or the logical_not function instead.

This is related to #27
@gijzelaerr
Copy link
Member

i seem to be only able to reproduce this bug on OSX with Python3 and h5py==2.8.0

@gijzelaerr
Copy link
Member

If I use Python2 and KERN blimpy installs and the test suite runs:

https://travis-ci.org/UCBerkeleySETI/blimpy/jobs/422323318

https://github.com/UCBerkeleySETI/blimpy/blob/8817a86d4dbd02220ee755058f88ba3b43fd6a74/docker/py2_kern.docker

Also, a manual install of bitshuffle and h5py seem to work.

The error I see for the python3 builds is a different error from the one you copy paste:

https://travis-ci.org/UCBerkeleySETI/blimpy/jobs/422323343

and seems to be related to the other py3 problems we had where floats are used for indexing, which are not automatically converted (anymore).

@gijzelaerr
Copy link
Member

I think this might be related to how bitshuffle is compiled and to which python this is linked. I'm not completely sure yet what the issue is, but it might be the mixing of py2 and py3 related...

@gijzelaerr gijzelaerr changed the title current way to install blimpy. ValueError: Unable to create dataset (error during user callback) for python3 Sep 3, 2018
@gijzelaerr
Copy link
Member

even more interesting, the py2 test run completes even without setting HDF5_PLUGIN_PATH! Py3 fails if the plugin path is unset withValueError: Unknown compression filter number: 32008

@gijzelaerr
Copy link
Member

I think this is a mix of issues, but basically, it boils down to that it works now when:

  • the KERN packages are used
  • If not, make sure you manually install h5py using: $ pip install --no-binary=h5py h5py

@jeenriquez
Copy link
Contributor Author

This is great! I'll add it to the readme for future reference.

jeenriquez added a commit that referenced this issue Sep 7, 2018
@jeenriquez
Update README.md
6fc867a 
Updating Readme to show new way to install h5py, according to #49.
@jeenriquez
Copy link
Contributor Author

Reopening since having issues to do

pip install --no-binary=h5py h5py

In my mac I get

$ pip install --no-binary=h5py h5py
Collecting h5py
  Using cached https://files.pythonhosted.org/packages/74/5d/6f11a5fffc3d8884bb8d6c06abbee0b3d7c8c81bde9819979208ba823a47/h5py-2.8.0.tar.gz
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.org/simple/pkgconfig/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) -- Some packages may not be found!
    Couldn't find index page for 'pkgconfig' (maybe misspelled?)
    Download error on https://pypi.org/simple/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) -- Some packages may not be found!
    No local packages or working download links found for pkgconfig
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/7y/bw6r9jvd021_h03n0cqfsx9r0000gn/T/pip-install-XehpFX/h5py/setup.py", line 168, in <module>
        cmdclass = CMDCLASS,
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/__init__.py", line 130, in setup
        _install_setup_requires(attrs)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/__init__.py", line 125, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
        replace_conflicting=True,
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 773, in resolve
        replace_conflicting=replace_conflicting
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1056, in best_match
        return self.obtain(req, installer)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1068, in obtain
        return installer(requirement)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/dist.py", line 581, in fetch_build_egg
        return cmd.easy_install(req)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 670, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pkgconfig')
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/7y/bw6r9jvd021_h03n0cqfsx9r0000gn/T/pip-install-XehpFX/h5py/

and in the GBT cluster (Ubuntu 16)

Collecting h5py
  Downloading https://files.pythonhosted.org/packages/74/5d/6f11a5fffc3d8884bb8d6c06abbee0b3d7c8c81bde9819979208ba823a47/h5py-2.8.0.tar.gz (274kB)
    100% |████████████████████████████████| 276kB 3.5MB/s 
Requirement already satisfied: numpy>=1.7 in /opt/pyve/eepy/lib/python2.7/site-packages (from h5py) (1.15.1)
Requirement already satisfied: six in /opt/pyve/eepy/lib/python2.7/site-packages (from h5py) (1.11.0)
Skipping bdist_wheel for h5py, due to binaries being disabled for it.
Installing collected packages: h5py
  Running setup.py install for h5py ... error
    Complete output from command /opt/pyve/eepy/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-88vL0c/h5py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-e_6oh0/install-record.txt --single-version-externally-managed --compile --install-headers /opt/pyve/eepy/include/site/python2.7/h5py:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/h5py
    copying h5py/__init__.py -> build/lib.linux-x86_64-2.7/h5py
    copying h5py/ipy_completer.py -> build/lib.linux-x86_64-2.7/h5py
    copying h5py/version.py -> build/lib.linux-x86_64-2.7/h5py
    copying h5py/highlevel.py -> build/lib.linux-x86_64-2.7/h5py
    copying h5py/h5py_warnings.py -> build/lib.linux-x86_64-2.7/h5py
    creating build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/datatype.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/__init__.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/attrs.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/dims.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/filters.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/selections2.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/base.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/files.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/group.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/selections.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/compat.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/dataset.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    creating build/lib.linux-x86_64-2.7/h5py/tests
    copying h5py/tests/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests
    copying h5py/tests/common.py -> build/lib.linux-x86_64-2.7/h5py/tests
    creating build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_file.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_datatype.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_group.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_dataset.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
.
.
.
.
.
    warning: h5py/api_types_hdf5.pxd:494:4: 'H5T_CSET_ASCII' redeclared
    warning: h5py/api_types_hdf5.pxd:637:2: 'H5Z_filter_t' redeclared
    warning: h5py/api_types_hdf5.pxd:672:6: 'H5Z_ERROR_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:673:6: 'H5Z_DISABLE_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:674:6: 'H5Z_ENABLE_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:675:6: 'H5Z_NO_EDC' redeclared
    building 'h5py.defs' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/tmp
    creating build/temp.linux-x86_64-2.7/tmp/pip-install-88vL0c
    creating build/temp.linux-x86_64-2.7/tmp/pip-install-88vL0c/h5py
    creating build/temp.linux-x86_64-2.7/tmp/pip-install-88vL0c/h5py/h5py
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DH5_USE_16_API -I./h5py -I/usr/local/hdf5/include -I/tmp/pip-install-88vL0c/h5py/lzf -I/opt/pyve/eepy/local/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c /tmp/pip-install-88vL0c/h5py/h5py/defs.c -o build/temp.linux-x86_64-2.7/tmp/pip-install-88vL0c/h5py/h5py/defs.o
    In file included from /opt/pyve/eepy/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1821:0,
                     from /opt/pyve/eepy/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                     from /opt/pyve/eepy/local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from /tmp/pip-install-88vL0c/h5py/h5py/api_compat.h:26,
                     from /tmp/pip-install-88vL0c/h5py/h5py/defs.c:607:
    /opt/pyve/eepy/local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it by " \
      ^
    In file included from /tmp/pip-install-88vL0c/h5py/h5py/defs.c:607:0:
    /tmp/pip-install-88vL0c/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/opt/pyve/eepy/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-88vL0c/h5py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-e_6oh0/install-record.txt --single-version-externally-managed --compile --install-headers /opt/pyve/eepy/include/site/python2.7/h5py" failed with error code 1 in /tmp/pip-install-88vL0c/h5py/

@jeenriquez jeenriquez reopened this Sep 10, 2018
@telegraphic
Copy link
Contributor

Hey @jeenriquez

  1. Can you try updating pip on your Mac? It looks like the main error here is that pip is using an old version of SSL? https://stackoverflow.com/questions/49768770/not-able-to-install-python-packages-ssl-tlsv1-alert-protocol-version?noredirect=1&lq=1

  2. I reckon this is probably due to Ubuntu 16.04 deciding to install hdf5 libraries in a weird non-standard way. I would try this:

######
# HDF5 fixup
# Ubuntu 16.04 has a crazy hdf5 setup, needs some massaging, and extra flags to install h5py
RUN ln -s /usr/lib/x86_64-linux-gnu/libhdf5_serial.so /usr/lib/x86_64-linux-gnu/libhdf5.so
RUN ln -s /usr/lib/x86_64-linux-gnu/libhdf5_serial_hl.so /usr/lib/x86_64-linux-gnu/libhdf5_hl.so
RUN CFLAGS=-I/usr/include/hdf5/serial pip install h5py --no-binary=h5py h5py

@telegraphic
Copy link
Contributor

(on point 2) -- obviously you shouldn't need to do this. But this isn't an issue with blimpy per se. Maybe we should have a wiki page with common installation issues?)

@gijzelaerr
Copy link
Member

gijzelaerr commented Sep 10, 2018 •
edited
Loading

this is the error: pi_compat.h:27:18: fatal error: hdf5.h: No such file or directory

it doesn't find the hdf5 libraries. do you have that installed? try brew install hdf5

EDIT: ah wait, i'm mixing up the errors.

@gijzelaerr
Copy link
Member

Is this still a issue?

@gijzelaerr gijzelaerr removed their assignment Oct 15, 2018
@telegraphic telegraphic self-assigned this Nov 29, 2018
@telegraphic telegraphic mentioned this issue Nov 30, 2018
Open
@telegraphic
Copy link
Contributor

Closing as this is being moved to documentation issue

FX196 pushed a commit to FX196/blimpy that referenced this issue Jul 6, 2019
@jeenriquez
Update README.md
73fb1c6 
Updating Readme to show new way to install h5py, according to UCBerkeleySETI#49.
zrd7527 pushed a commit to zrd7527/Research that referenced this issue Jul 27, 2020
@jeenriquez
Update README.md
1296fef 
Updating Readme to show new way to install h5py, according to UCBerkeleySETI/blimpy#49.

Former-commit-id: 6fc867a7f7ff7469f7b073ff1f8506a169f76720
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@telegraphic telegraphic

@jeenriquez jeenriquez

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gijzelaerr @telegraphic @jeenriquez