Cannot access URLs

[johentsch]

Overview

I am curating a data project where we make datapackages available both on GitHub and on Zenodo. So far, I have never managed to access data on the web using the frictionless framework.

Setup

I have installed the framework (version 5.16.0) using pip install -U frictionless[zenodo,visidata].

GitHub Fail

Our workflow publishes datapackages with every GitHub release such as this one here.

Since everyone can access the file

• https://github.com/DCMLab/dcml_corpora/releases/download/v2.1/dcml_corpora.datapackage.json which describes the TSV data contained in
• https://github.com/DCMLab/dcml_corpora/releases/download/v2.1/dcml_corpora.zip

I would expect to be able to

frictionless explore https://github.com/DCMLab/dcml_corpora/releases/download/v2.1/dcml_corpora.datapackage.json

or to

frictionless validate https://github.com/DCMLab/dcml_corpora/releases/download/v2.1/dcml_corpora.datapackage.json

but in both cases I get

╭─ Error ─────────────────────────────────────────────╮
│ [error] Repo and user is required                   │
╰─────────────────────────────────────────────────────╯

I hope it's just that I overlooked the relevant bit in the documentation which I have extensively searched. Nevertheless, from my understanding of Frictionless this should "just work".

Zenodo Fail

Likewise, take this Zenodo record.

It provides the datapackage

• https://zenodo.org/records/10181503/files/mozart_piano_sonatas.datapackage.json which describes
• https://zenodo.org/records/10181503/files/mozart_piano_sonatas.zip

so I would like to

frictionless explore https://zenodo.org/records/10181503/files/mozart_piano_sonatas.datapackage.json
frictionless validate https://zenodo.org/records/10181503/files/mozart_piano_sonatas.datapackage.json

but I get

╭─ Error ─────────────────────────────────────────────╮
│ [error] Record is required.                         │
╰─────────────────────────────────────────────────────╯

Am I doing something wrong?

[roll]

Thanks for reporting it's a bug!

[pierrecamilleri]

I made a failed attempt to fix the github links, so for the next attempt (or if anyone wants to try to fix this), here is what happens : due to the github.com domain, frictionless thinks it needs to interact with a Github Portal, which in turn expects a URL shaped like github.com/user/repo with no additional URL segment and therefore throws an exception.

Instead, the URL provided in the issue is a plain URL pointing to a plain JSON file, and I suspect the same holds for the zenodo URL.

A possible solution might be to check if the URL has the expected shape before dispatching it to a data portal (which probably happens in some factory.py file => correction: the create_adapter function decides if the source is appropriate for the plugin or if it needs to be passed over to other plugins).