From 3f6ea88db97a1a875682d6f5521f8e24f1c4068d Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 1 Apr 2020 17:07:23 +0200 Subject: [PATCH] Add a short initial section on configuring providers This section is not very extensive, mainly because I lack knowledge about how things work. It is a start though, and it documents the cfg magic that is required to interact with an XNAT server --- docs/basics/101-146-providers.rst | 86 +++++++++++++++++++++++++++++++ docs/basics/basics10toc.rst | 3 +- 2 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 docs/basics/101-146-providers.rst diff --git a/docs/basics/101-146-providers.rst b/docs/basics/101-146-providers.rst new file mode 100644 index 000000000..a3951f888 --- /dev/null +++ b/docs/basics/101-146-providers.rst @@ -0,0 +1,86 @@ +.. _providers: + +Configuring new data providers +------------------------------ + +DataLad can download files via the ``http``, ``https``, ``ftp``, and ``s3`` +protocol. If data retrieval requires authentication, DataLad's provides an interface to +query, request, and store the most common type of credentials. An example of this +is shown in the usecase :ref:`usecase_HCP_dataset`: :command:`datalad get` will +ask for AWS S3 credential to retrieve data that is stored in S3 buckets from +the command line once, and -- given valid credentials -- download the files +and all future files from this bucket right away. There +are a number of natively supported types of authentication (a list is given in +the hidden section below) and out-of-the box access to a broad range +of providers such as `S3 `_ or +`LORIS `_. +If you attempt to retrieve data from supported providers that require authentication, +you will be prompted for your credentials from the command line at the time of +the first download, and subsequent downloads handle authentication in the background. + +If a provider requires authentication but it is not known to DataLad yet, the +download will fail. However, the provider can be configured by +the user to enable interactions with it. +With a provider configuration in place, commands such as :command:`datalad download-url` +or :command:`datalad add-urls` can work with urls of custom providers, and +:command:`datalad get` is enabled to retrieve file contents from these sources. +The configuration can either be done in the terminal upon a prompt from the +command line when a download fails due to a missing provider configuration, +or by placing provider-specific configuration files into ``.datalad/providers``. + +In order to configure a provider configuration, one needs to input a +provider name, a regular expression that can match a url one would want to download +from, an authentication type, and a credential type. The example below sheds some +light one this. + +.. findoutmore:: Which authentication and credential types are possible? + + The following credential types are supported: + + - ``'user_password'`` + - ``'aws-s3'`` + - ``'nda-s3'`` + - ``'token'`` + - ``'loris-token'`` + + The following authentication types are supported: + + - ``'html_form'`` + - ``'http_auth'`` + - ``'http_basic_auth'`` + - ``'http_digest_auth'`` + - ``'bearer_token'`` + - ``'aws-s3'`` + - ``'nda-s3'`` + - ``'loris-token'`` + + Some pointers to read up more on these authetication types are here: + `http basic access `_, + `http and html form-based authentication `_, + `digest access authentication `_, + `http bearer token authentication `_. + +Example: DataLad access to an XNAT server +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +`XNAT `_ is an open source imaging informatics platform +and it allows for data download after authentication. +In order to allow file retrieval from a custom deployed +`XNAT server `_ with DataLad, a file following a +similar structure as the example below (with adjusted urls and ports) +can be placed into ``.datalad/providers/xnat.cfg``: + +.. code-block:: bash + + [provider:xnat] + url_re = https://nmrxnat.ime.kfa-juelich.de:8443/xnat/.* + credential = nmrxnat.ime.kfa-juelich.de:8443/xnat + authentication_type = http_basic_auth + + [credential:nmrxnat.ime.kfa-juelich.de:8443/xnat] + type = user_password + +In the dataset that this file was placed into, downloading commands such as +:command:`datalad download-url` that point to +``https://nmrxnat.ime.kfa-juelich.de:8443/xnat`` will work and ask (once) for +the user's user name and password. \ No newline at end of file diff --git a/docs/basics/basics10toc.rst b/docs/basics/basics10toc.rst index 41f6e2cd8..9aa0c2105 100644 --- a/docs/basics/basics10toc.rst +++ b/docs/basics/basics10toc.rst @@ -15,4 +15,5 @@ 101-179-gitignore 101-144-intro_extensions - 101-145-hooks \ No newline at end of file + 101-145-hooks + 101-146-providers \ No newline at end of file