CADC cadc.get_images failures #2242

keflavich · 2021-12-05T22:28:40Z

~~EDIT: Errors have changed, see the comment below: #2242 (comment)~~

===============

astroquery/cadc/tests/test_cadctap.py ....s........                                                                                                                                                [ 48%]
astroquery/cadc/tests/test_cadctap_remote.py ....ssssFsF..s                                                                                                                                        [100%]

================================================================================================ FAILURES ================================================================================================
_____________________________________________________________________________________ TestCadcClass.test_get_images ______________________________________________________________________________________

self = <astroquery.cadc.tests.test_cadctap_remote.TestCadcClass object at 0xd2c382b38>

    @pytest.mark.skipif(one_test, reason='One test mode')
    @pytest.mark.skipif(not pyvo_OK, reason='not pyvo_OK')
    def test_get_images(self):
        cadc = Cadc()
        coords = '08h45m07.5s +54d18m00s'
        radius = 0.005*u.deg
>       images = cadc.get_images(coords, radius, collection='CFHT')

astroquery/cadc/tests/test_cadctap_remote.py:213:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
astroquery/utils/class_or_instance.py:25: in f
    return self.fn(obj, *args, **kwds)
astroquery/cadc/core.py:346: in get_images
    get_url_list, show_progress)
astroquery/cadc/core.py:395: in get_images_async
    images_urls = self.get_image_list(query_result, coordinates, radius)
astroquery/cadc/core.py:459: in get_image_list
    urlencode({'ID': pid_sublist}, True)))
../pyvo/pyvo/dal/query.py:284: in from_result_url
    votableparse(cls._from_result_url(result_url, session).read),
../../miniconda3/lib/python3.7/site-packages/astropy/utils/decorators.py:521: in wrapper
    return function(*args, **kwargs)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/table.py:167: in parse
    config=config, pos=(1, 1)).parse(iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:3568: in parse
    iterator, tag, data, config, pos)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:3478: in _add_resource
    resource.parse(self, iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:3280: in parse
    iterator, tag, data, config, pos)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:3235: in _add_group
    group.parse(iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:2075: in parse
    iterator, tag, data, config, pos)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:2057: in _add_param
    param.parse(iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:1472: in parse
    self.values.parse(iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:1019: in parse
    self.min = data['value']
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:933: in min
    self._min = self._field.converter.parse(min)[0]
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/converters.py:580: in parse
    warn_or_raise(E02, E02, (self._items, len(parts)), config, pos)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/exceptions.py:92: in warn_or_raise
    vo_warn(warning_class, args, config, pos, stacklevel=stacklevel+1)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/exceptions.py:133: in vo_warn
    _suppressed_warning(warning, config, stacklevel=stacklevel+1)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

warning = E02('?:?:?: E02: Incorrect number of elements in array. Expected multiple of 2, got 1'), config = {'_warning_counts': {<class 'astropy.io.votable.exceptions.E02'>: 1}}, stacklevel = 3

    def _suppressed_warning(warning, config, stacklevel=2):
        warning_class = type(warning)
        config.setdefault('_warning_counts', dict()).setdefault(warning_class, 0)
        config['_warning_counts'][warning_class] += 1
        message_count = config['_warning_counts'][warning_class]
        if message_count <= MAX_WARNINGS:
            if message_count == MAX_WARNINGS:
                warning.formatted_message += \
                    ' (suppressing further warnings of this type...)'
>           warn(warning, stacklevel=stacklevel+1)
E           astropy.io.votable.exceptions.E02: ?:?:?: E02: Incorrect number of elements in array. Expected multiple of 2, got 1

../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/exceptions.py:73: E02
__________________________________________________________________________________ TestCadcClass.test_get_images_async ___________________________________________________________________________________

self = <astroquery.cadc.tests.test_cadctap_remote.TestCadcClass object at 0xd2d0feeb8>

    @pytest.mark.skipif(one_test, reason='One test mode')
    @pytest.mark.skipif(not pyvo_OK, reason='not pyvo_OK')
    def test_get_images_async(self):
        cadc = Cadc()
        coords = '01h45m07.5s +23d18m00s'
        radius = '0.05 deg'
>       readable_objs = cadc.get_images_async(coords, radius, collection="CFHT")

astroquery/cadc/tests/test_cadctap_remote.py:260:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
astroquery/cadc/core.py:395: in get_images_async
    images_urls = self.get_image_list(query_result, coordinates, radius)
astroquery/cadc/core.py:459: in get_image_list
    urlencode({'ID': pid_sublist}, True)))
../pyvo/pyvo/dal/query.py:284: in from_result_url
    votableparse(cls._from_result_url(result_url, session).read),
../../miniconda3/lib/python3.7/site-packages/astropy/utils/decorators.py:521: in wrapper
    return function(*args, **kwargs)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/table.py:167: in parse
    config=config, pos=(1, 1)).parse(iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:3568: in parse
    iterator, tag, data, config, pos)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:3478: in _add_resource
    resource.parse(self, iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:3280: in parse
    iterator, tag, data, config, pos)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:3235: in _add_group
    group.parse(iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:2075: in parse
    iterator, tag, data, config, pos)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:2057: in _add_param
    param.parse(iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:1472: in parse
    self.values.parse(iterator, config)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:1019: in parse
    self.min = data['value']
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py:933: in min
    self._min = self._field.converter.parse(min)[0]
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/converters.py:580: in parse
    warn_or_raise(E02, E02, (self._items, len(parts)), config, pos)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/exceptions.py:92: in warn_or_raise
    vo_warn(warning_class, args, config, pos, stacklevel=stacklevel+1)
../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/exceptions.py:133: in vo_warn
    _suppressed_warning(warning, config, stacklevel=stacklevel+1)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

warning = E02('?:?:?: E02: Incorrect number of elements in array. Expected multiple of 2, got 1'), config = {'_warning_counts': {<class 'astropy.io.votable.exceptions.E02'>: 1}}, stacklevel = 3

    def _suppressed_warning(warning, config, stacklevel=2):
        warning_class = type(warning)
        config.setdefault('_warning_counts', dict()).setdefault(warning_class, 0)
        config['_warning_counts'][warning_class] += 1
        message_count = config['_warning_counts'][warning_class]
        if message_count <= MAX_WARNINGS:
            if message_count == MAX_WARNINGS:
                warning.formatted_message += \
                    ' (suppressing further warnings of this type...)'
>           warn(warning, stacklevel=stacklevel+1)
E           astropy.io.votable.exceptions.E02: ?:?:?: E02: Incorrect number of elements in array. Expected multiple of 2, got 1

../../miniconda3/lib/python3.7/site-packages/astropy/io/votable/exceptions.py:73: E02
======================================================================================== short test summary info =========================================================================================
FAILED astroquery/cadc/tests/test_cadctap_remote.py::TestCadcClass::test_get_images - astropy.io.votable.exceptions.E02: ?:?:?: E02: Incorrect number of elements in array. Expected multiple of 2, got 1
FAILED astroquery/cadc/tests/test_cadctap_remote.py::TestCadcClass::test_get_images_async - astropy.io.votable.exceptions.E02: ?:?:?: E02: Incorrect number of elements in array. Expected multiple of ...
========================================================================== 2 failed, 18 passed, 7 skipped in 211.59s (0:03:31) ===========================================================================

These look like upstream (VO) issues? I'll x-post in pyvo

The text was updated successfully, but these errors were encountered:

andamian · 2021-12-08T17:49:07Z

I've had problems accessing the relevant CADC logs but I'm still looking into this.

andamian · 2022-03-18T20:07:39Z

Just to add some info:
2 of them (test_get_images, test_get_images_async) fail with a VOTable parse error which can be caused either by service or VOTable incompatibilities with the spec. It requires further investigation.

warning = E02('?:?:?: E02: Incorrect number of elements in array. Expected multiple of 2, got 1')
config = {'_warning_counts': {<class 'astropy.io.votable.exceptions.E02'>: 1}}
stacklevel = 3

def _suppressed_warning(warning, config, stacklevel=2):
    warning_class = type(warning)
    config.setdefault('_warning_counts', dict()).setdefault(warning_class, 0)
    config['_warning_counts'][warning_class] += 1
    message_count = config['_warning_counts'][warning_class]
    if message_count <= conf.max_warnings:
        if message_count == conf.max_warnings:
            warning.formatted_message += \
                ' (suppressing further warnings of this type...)'

      warn(warning, stacklevel=stacklevel+1)

E astropy.io.votable.exceptions.E02: ?:?:?: E02: Incorrect number of elements in array. Expected multiple of 2, got 1
The third one (test_list_jobs) is failing in PyVO with an unexpected element. Again incompatibilities with the spec in either the service or PyVO - to be investigated.

      warn(warning, stacklevel=stacklevel+1)

E pyvo.utils.xml.exceptions.UnknownElementWarning: None:2:0: UnknownElementWarning: Unknown element jobs
``

bsipocz · 2022-03-29T01:26:09Z

I don't see these two errors atm, instead, I see a different DALQueryError:

====================================================================== test session starts ======================================================================
platform darwin -- Python 3.10.0, pytest-6.2.5, py-1.11.0, pluggy-1.0.0

Running tests with astroquery version 0.4.7.dev7630_testrun.
Running tests in astroquery/cadc docs/cadc.

Date: 2022-03-28T18:23:26

Platform: macOS-10.13.6-x86_64-i386-64bit

Executable: /Users/bsipocz/.pyenv/versions/3.10.0/bin/python

Full Python Version: 
3.10.0 (default, Nov  1 2021, 16:57:20) [Clang 10.0.0 (clang-1000.11.45.5)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.21.4
Matplotlib: 3.4.3
Astropy: 5.0.2
APLpy: not available
pyregion: 2.1.1
regions: 0.5
pyVO: 1.3
mocpy: 0.10.0
astropy-healpix: 0.6
vamdclib: not available
astropy-helpers: 4.0.1

Using Astropy options: remote_data: any.

rootdir: /private/var/folders/dc/hsm7tqpx2d57n7vb3k1l81xw0000gq/T/astroquery-test-j5w_llg5/lib/python3.10/site-packages, configfile: setup.cfg
plugins: arraydiff-0.3, remotedata-0.3.2, hypothesis-6.24.2, mock-3.6.1, doctestplus-0.11.0, filter-subpackage-0.1.1, openfiles-0.5.0, astropy-header-0.1.2, cov-3.0.0
collected 28 items                                                                                                                                              

astroquery/cadc/tests/test_cadctap.py ....s........                                                                                                       [ 46%]
astroquery/cadc/tests/test_cadctap_remote.py ....ssssxsxF.s                                                                                               [ 96%]
docs/cadc/cadc.rst s                                                                                                                                      [100%]

=========================================================================== FAILURES ============================================================================
___________________________________________________________________ TestCadcClass.test_async ____________________________________________________________________

self = <astroquery.cadc.tests.test_cadctap_remote.TestCadcClass object at 0x118ada290>

    @pytest.mark.skipif(one_test, reason='One test mode')
    @pytest.mark.skipif(not pyvo_OK, reason='not pyvo_OK')
    def test_async(self):
        # test async calls
        cadc = Cadc()
    
        # run the query in sync mode first
        expected = cadc.exec_sync(
            "select top 3 observationID from caom2.Observation where "
            "collection='IRIS' order by observationID")
    
        # now run the query in async mode
        job = cadc.create_async(
            "select top 3 observationID from caom2.Observation where "
            "collection='IRIS' order by observationID")
        job = job.run().wait()
>       job.raise_if_error()

astroquery/cadc/tests/test_cadctap_remote.py:283: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyvo.dal.tap.AsyncTAPJob object at 0x118ada950>

    def raise_if_error(self):
        """
        raise a exception if theres an error
    
        Raises
        ------
        DALQueryError
            if theres an error
        """
        if self.phase in {"ERROR", "ABORTED"}:
>           raise DALQueryError("Query Error", self.phase, self.url)
E           pyvo.dal.exceptions.DALQueryError: Query Error

/Users/bsipocz/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pyvo/dal/tap.py:824: DALQueryError
==================================================================== short test summary info ====================================================================
FAILED astroquery/cadc/tests/test_cadctap_remote.py::TestCadcClass::test_async - pyvo.dal.exceptions.DALQueryError: Query Error

andamian · 2022-04-08T23:52:14Z

The CADC services have experience networking outages but things should work (more or less) now. The tests are now passing.

bsipocz · 2022-04-11T20:42:09Z

Perfect, I can confirm that all tests pass or xfail now.

astroquery/cadc/tests/test_cadctap.py ....s........                                                                                                   [ 46%]
astroquery/cadc/tests/test_cadctap_remote.py ....ssssxsx..s                                                                                           [ 96%]
docs/cadc/cadc.rst s                                                                                                                                  [100%]

=================================================== 18 passed, 8 skipped, 2 xfailed in 163.96s (0:02:43) ====================================================

bsipocz · 2022-09-27T23:49:45Z

reopening as these never got actually fixed, but got an xfail marker instead.

The same error is triggered for the narrative docs, and ultimately I think is related to astropy/pyvo#361, too

bsipocz · 2023-08-08T21:43:42Z

I don't see this any more for astroquery, so am closing it now. Upstream there still seem to be some issues to sort out, so the pyvo issues should stay open.

keflavich added vo cadc labels Dec 5, 2021

This was referenced Dec 5, 2021

Failing remote tests Q4 2021 #2203

Closed

CADC astroquery failure in vo reading? astropy/pyvo#280

Closed

bsipocz mentioned this issue Mar 18, 2022

CADC int tests failing #2325

Closed

bsipocz added bug Upstream labels Mar 18, 2022

bsipocz closed this as completed Apr 11, 2022

bsipocz changed the title ~~CADC failures Dec 2021~~ CADC cadc.get_images failures Sep 27, 2022

bsipocz reopened this Sep 27, 2022

bsipocz closed this as completed Aug 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CADC cadc.get_images failures #2242

CADC cadc.get_images failures #2242

keflavich commented Dec 5, 2021 •

edited by bsipocz

Loading

andamian commented Dec 8, 2021

andamian commented Mar 18, 2022

bsipocz commented Mar 29, 2022

andamian commented Apr 8, 2022

bsipocz commented Apr 11, 2022

bsipocz commented Sep 27, 2022

bsipocz commented Aug 8, 2023

CADC cadc.get_images failures #2242

CADC cadc.get_images failures #2242

Comments

keflavich commented Dec 5, 2021 • edited by bsipocz Loading

andamian commented Dec 8, 2021

andamian commented Mar 18, 2022

bsipocz commented Mar 29, 2022

andamian commented Apr 8, 2022

bsipocz commented Apr 11, 2022

bsipocz commented Sep 27, 2022

bsipocz commented Aug 8, 2023

keflavich commented Dec 5, 2021 •

edited by bsipocz

Loading