ElementTree cannot handle package entries from PyPI with attribute data-yanked

[lexming]

Downloading the sources of a Python package from PyPI will fail if some of the entries listed in the HTML file retrieved from PyPI's simple API have the extra attribute data-yanked.

More info on yanked packages: https://www.python.org/dev/peps/pep-0592/

This happens with

• Python v2.7.5 from CentOS 7.7
• Python v2.7.16 from Easybuild
• Python v3.7.4 from Easybuild

At the present time, this can be tested with IPython-7.13.0-foss-2020a-Python-3.8.2.eb from 2020a branch

$ eb4 --debug IPython-7.13.0-foss-2020a-Python-3.8.2.eb 
== temporary log file in case of crash /tmp/eb-wh9MvS/easybuild-lPPWux.log
== resolving dependencies ...
== processing EasyBuild easyconfig /theia/home/brussel/101/vsc10122/src/easybuild-easyconfigs/easybuild/easyconfigs/IPython-7.13.0-foss-2020a-Python-3.8.2.eb
== building and installing IPython/7.13.0-foss-2020a-Python-3.8.2...
== fetching files...
/tmp/eb-wh9MvS/tmp2GjUvY/ipython_urls.html
ERROR: Traceback (most recent call last):
  File "/theia/home/brussel/101/vsc10122/src/easybuild-framework/easybuild/main.py", line 114, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 3225, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 3129, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 2968, in run_step
    step_method(self)()
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 1795, in fetch_step
    self.exts = self.fetch_extension_sources(skip_checksums=skip_checksums)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 520, in fetch_extension_sources
    src_fn = self.obtain_file(fn, extension=True, urls=source_urls, force_download=force_download)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 727, in obtain_file
    alt_url = derive_alt_pypi_url(fullurl)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/tools/filetools.py", line 520, in derive_alt_pypi_url
    cand_urls = pypi_source_urls(pkg_name)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/tools/filetools.py", line 499, in pypi_source_urls
    parsed_html = ElementTree.parse(urls_html)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1182, in parse
    tree.parse(source, parser)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse
    parser.feed(data)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1642, in feed
    self._raiseerror(v)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 237, column 267

Line 237 of the HTML file is the first entry with the data-yanked attribute

[...]
    <a href="https://files.pythonhosted.org/packages/5b/e3/4b3082bd7f6908af828561b0129b5064bff4a13e6acadb321fc2d939a605/ipython-7.0.1.tar.gz#sha256=9fed506c3772c875a3048bc134a25e6f5e997b1569b2636f6a5d891f34cbfd46" data-requires-python="&gt;=3.5">ipython-7.0.1.tar.gz</a><br/>
    <a href="https://files.pythonhosted.org/packages/ea/56/d75d25b30936bff3f6b4d7b80cfd399b703e2162f31afee926f66ff77b13/ipython-7.1.0-py3-none-any.whl#sha256=107859b062f6360d7bf0063c275511c638b633622954ba9a401ffa4a0c6aba1b" data-requires-python="&gt;=3.5" data-yanked>ipython-7.1.0-py3-none-any.whl</a><br/>

[boegel]

@lexming I'm a bit surprised that I'm not seeing this...

Can you share the output of rpm -qf /usr/lib64/python2.7/xml/etree/ElementTree.py?

[boegel]

OK, nevermind, easy to reproduce locally...

This occurs for Python packages that have yanked releases, see https://pypi.org/help/#yanked .