Add a short initial section on configuring providers #440
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This section is not very extensive, mainly because I lack knowledge about how things work. It is a start though, and it documents the cfg magic that is required to interact with an XNAT server
mih
requested changes
Apr 1, 2020
There was a problem hiding this comment.
Thx much! I think we should aim for something more general that is a better fit for the handbook, and does not attempt to replace the (admittedly non-existing technical docs). I left a few comments re a potential direction in which to take it. Please let me know what you think about that.
| With a provider configuration in place, commands such as :command:`datalad download-url` | ||
| or :command:`datalad add-urls` can work with urls of custom providers, and | ||
| :command:`datalad get` is enabled to retrieve file contents from these sources. | ||
| The configuration can either be done in the terminal upon a prompt from the |
There was a problem hiding this comment.
do you mean s/configuration/credential entry/ ? Is there a way to configure "providers" on the fly?
There was a problem hiding this comment.
Yes (I only know from trying it blindly). I creates the same file in the same location as you have, and you need to provide the information that you put into the file. Here's how looks in the terminal when downloading without the cfg file in place:
datalad download-url https://nmrxnat.ime.kfa-juelich.de:8443/xnat/data/projects/1000BRAINS/subjects/XNATINM4_S03059/experiments/XNATINM4_E04029/scans/1/resources/85041/files/0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm
[INFO ] Downloading 'https://nmrxnat.ime.kfa-juelich.de:8443/xnat/data/projects/1000BRAINS/subjects/XNATINM4_S03059/experiments/XNATINM4_E04029/scans/1/resources/85041/files/0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm' into '/tmp/xnat2/'
Authenticated access to https://nmrxnat.ime.kfa-juelich.de:8443/xnat/data/projects/1000BRAINS/subjects/XNATINM4_S03059/experiments/XNATINM4_E04029/scans/1/resources/85041/files/0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm has failed.
Would you like to setup a new provider configuration to access url? (choices: [yes], no): yes
New provider name
Unique name to identify 'provider' for https://nmrxnat.ime.kfa-juelich.de:8443/xnat/data/projects/1000BRAINS/subjects/XNATINM4_S03059/experiments/XNATINM4_E04029/scans/1/resources/85041/files/0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm [nmrxnat.ime.kfa-juelich.de]: xnat
New provider regular expression
A (Python) regular expression to specify for which URLs this provider should be used [https://nmrxnat\.ime\.kfa\-juelich\.de:8443/xnat/data/projects/1000BRAINS/subjects/XNATINM4_S03059/experiments/XNATINM4_E04029/scans/1/resources/85041/files/0015911870_1\.3\.12\.2\.1107\.5\.2\.32\.35135\.2011102112040130362336780\.dcm]: https://nmrxnat.ime.kfa-juelich.de:8443/xnat/.*
Authentication type
What authentication type to use (choices: aws-s3, bearer_token, html_form, http_auth, http_basic_auth, http_digest_auth, loris-token, nda-s3, none, xnat): http_basic_auth
Credential
What type of credential should be used? (choices: aws-s3, loris-token, nda-s3, token, [user_password]):
Save provider configuration file
Following configuration will be written to /home/adina/.config/datalad/providers/xnat.cfg:
# Provider configuration file created to initially access
# https://nmrxnat.ime.kfa-juelich.de:8443/xnat/data/projects/1000BRAINS/subjects/XNATINM4_S03059/experiments/XNATINM4_E04029/scans/1/resources/85041/files/0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm
[provider:xnat]
url_re = https://nmrxnat.ime.kfa-juelich.de:8443/xnat/.*
authentication_type = http_basic_auth
# Note that you might need to specify additional fields specific to the
# authenticator. Fow now "look into the docs/source" of <class 'datalad.downloaders.http.HTTPBasicAuthAuthenticator'>
# http_basic_auth_
credential = xnat
[credential:xnat]
# If known, specify URL or email to how/where to request credentials
# url = ???
type = user_password
(choices: [yes], no): yes
You need to authenticate with 'xnat' credentials.
user: ???
password:
password (repeat):
[INFO ] http session: Authenticating into session for https://nmrxnat.ime.kfa-juelich.de:8443/xnat/data/projects/1000BRAINS/subjects/XNATINM4_S03059/experiments/XNATINM4_E04029/scans/1/resources/85041/files/0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm
https://nmrxnat.ime.kfa-juelich.de:8443/xnat/data/projects/1000BRAINS/subjects/XNATINM4_S03059/experiments/XNATINM4_E04029/scans/1/resources/85041/files/0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm: 0%| | 0.00/611k download_url(ok): /tmp/xnat2/0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm (file)
add(ok): 0015911870_1.3.12.2.1107.5.2.32.35135.2011102112040130362336780.dcm (file)
save(ok): . (dataset)
action summary:
add (ok: 1)
download_url (ok: 1)
save (ok: 1)
|
Thanks a lot for the comments. I've given it a first try. I think its very sensible to not have an xnat example but more general setup, I'll probably only need to read up in detail about the authentication method (thanks for the links!) and expand the section a bit. |
mih
approved these changes
Apr 10, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is quickly written down and without much substance to it (quite difficult to dig through the docstring-less
downloaderscode base). If it contains too many errors, we can add the XNAT configuration as a gist for now and wait until I know more about file transfer and credential management.