Streamline observations download

doc/sphinx/source/community/dataset.rst

@@ -15,13 +15,15 @@ Dataset documentation
 The documentation required for a CMORizer script is the following:
 
 - Make sure that the new dataset is added to the list of
-  :ref:`supported_datasets`
+  :ref:`supported_datasets` and to the file datasets.yml_.
 - The in code documentation should contain clear instructions on how to obtain
-  the data
+  the data.
 - A BibTeX file named ``<dataset>.bibtex`` defining the reference for the new
   dataset should be placed in the directory ``esmvaltool/references/``, see
   :ref:`adding_references` for detailed instructions.
 
+.. _datasets.yml: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/datasets.yml
+
 For more general information on writing documentation, see :ref:`documentation`.
 
 .. _dataset-test:
@@ -37,7 +39,9 @@ To test a pull request for a new CMORizer script:
 
 #. Download the data following the instructions included in the script and place
    it in the ``RAWOBS`` path specified in your ``config-user.yml``
-#. Run the CMORizer script by running ``cmorize_obs -c <config-file> -o <dataset>``
+#. If available, use the downloading script by running
+   ``esmvaltool data download --config_file <config-file>  <dataset>``
+#. Run the cmorization by running ``esmvaltool data format <config-file> <dataset>``
 #. Copy the resulting data to the ``OBS`` (for CMIP5 compliant data) or ``OBS6``
    (for CMIP6 compliant data) path specified in your
    ``config-user.yml``
@@ -74,6 +78,8 @@ Dataset description
 Check that new dataset has been added to the table of observations defined in
 the ESMValTool guide user’s guide in section :ref:`inputdata`
 (generated from ``doc/sphinx/source/input.rst``).
+Check that the new dataset has also been added to the file `datasets.yml
+<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/datasets.yml>`__.
 
 BibTeX info file
 ----------------
@@ -87,11 +93,24 @@ recipe_check_obs.yml
 Check that new dataset has been added to the testing recipe
 ``esmvaltool/recipes/examples/recipe_check_obs.yml``
 
+Downloader script
+-----------------
+
+If present, check that the new downloader script
+``esmvaltool/cmorizers/data/downloaders/datasets/<dataset>.py``
+meets standards.
+This includes the following items:
+
+* Code quality checks
+
+  1. Code quality
+  2. No Codacy errors reported
+
 CMORizer script
 ---------------
 
 Check that the new CMORizer script
-``esmvaltool/cmorizers/obs/cmorize_obs_<dataset>.{py,ncl}``
+``esmvaltool/cmorizers/data/formatters/datasets/<dataset>.{py,ncl}``
 meets standards.
 This includes the following items:
 
@@ -110,13 +129,21 @@ Config file
 -----------
 
 If present, check config file ``<dataset>.yml`` in
-``esmvaltool/cmorizers/obs/cmor_config/`` for correctness.
+``esmvaltool/cmorizers/data/cmor_config/`` for correctness.
 Use ``yamllint`` to check for syntax errors and common mistakes.
 
+Run downloader script
+---------------------
+
+If available, make sure the downloader script is working by running
+``esmvaltool data download --config_file <config-file> <dataset>``
+
+
 Run CMORizer
 ------------
 
-Make sure CMORizer is working by running ``cmorize_obs -c <config-file> -o <dataset>``
+Make sure CMORizer is working by running
+``esmvaltool data format --config_file <config-file> <dataset>``
 
 Check output of CMORizer
 ------------------------

doc/sphinx/source/develop/dataset.rst

@@ -12,6 +12,7 @@ data set for the use in ESMValTool.
 | `1. Check if your variable is CMOR standard`_
 | `2. Edit your configuration file`_
 | `3. Store your dataset in the right place`_
+| `3.1 Downloader script (optional)`_
 | `4. Create a cmorizer for the dataset`_
 | `4.1 Cmorizer script written in python`_
 | `4.2 Cmorizer script written in NCL`_
@@ -75,14 +76,62 @@ for downloading (e.g. providing contact information, licence agreements)
 and using the observations. The unformatted (raw) observations
 should then be stored then in the appropriate of these three folders.
 
+For each additional dataset, an entry needs to be made to the file 
+`datasets.yml
+<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/datasets.yml>`_.
+The dataset entry should contain:
+
+- the correct ``tier`` information;
+- the ``source`` of the raw data;
+- the ``last_access`` date;
+- the ``info`` that explain how to download the data.
+
+Note that these fields should be identical to the content of the header
+of the cmorizing script (see Section `4. Create a cmorizer for the dataset`_).
+
+3.1 Downloader script (optional)
+--------------------------------
+
+A Python script can be written to download raw observations 
+from source and store the data in the appropriate tier subdirectory of the
+folder ``RAWOBS`` automatically.
+There are many downloading scripts available in 
+`/esmvaltool/cmorizers/data/downloaders/datasets/
+<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/data/downloaders/datasets/>`_
+where several data download mechanisms are provided:
+
+- A `wget` get based downloader for http(s) downloads, with a specific derivation for NASA datasets.
+- A `ftp` downloader with a specific derivation for ESACCI datasets available from CEDA.
+- A Climate Data Store downloader based on `cdsapi`.
+
+Note that the name of this downloading script has to be identical to the
+name of the dataset.
+
+Depending on the source server, the downloading script needs to contain paths to
+raw observations, filename patterns and various necessary fields to retrieve 
+the data.
+Default ``start_date`` and ``end_date`` can be provided in cases where raw data 
+are stored in daily, monthly, and yearly files.
+
+The downloading script for the given dataset can be run with:
+
+.. code-block:: console
+
+ esmvaltool data download --config_file <config-user.yml>  <dataset-name>
+
+.. note::
+  The options ``--start`` and ``--end`` can be added to the command above to 
+  restrict the download of raw data to a time range. They will be ignored is a specific dataset
+  does not support it (i.e. because it is provided as a single file).
+
 4. Create a cmorizer for the dataset
 ====================================
 
-There are many cmorizing scripts available in `/esmvaltool/cmorizers/obs/
-<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/obs/>`_
+There are many cmorizing scripts available in 
+`/esmvaltool/cmorizers/data/formatters/datasets/
+<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/formatters/datasets/>`_
 where solutions to many kinds of format issues with observational data are
-addressed. Most of these scripts are written in NCL at the moment, but more
-and more examples for Python-based cmorizing scripts become available.
+addressed. These scripts are either written in Python or in NCL.
 
 .. note::
   NCL support will terminate soon, so new cmorizer scripts should preferably be
@@ -99,16 +148,17 @@ and one written in NCL, are explained in more detail.
 -------------------------------------
 
 Find here an example of a cmorizing script, written for the ``MTE`` dataset
-that is available at the MPI for Biogeochemistry in Jena: `cmorize_obs_mte.py
-<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/obs/cmorize_obs_mte.py>`_.
+that is available at the MPI for Biogeochemistry in Jena: `mte.py
+<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/formatters/datasets/mte.py>`_.
 
 All the necessary information about the dataset to write the filename
 correctly, and which variable is of interest, is stored in a separate
 configuration file: `MTE.yml
-<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/obs/cmor_config/MTE.yml>`_
-in the directory ``ESMValTool/esmvaltool/cmorizers/obs/cmor_config/``. Note
-that the name of this configuration file has to be identical to the name of
-your data set. It is recommended that you set ``project`` to ``OBS6`` in the
+<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/cmor_config/MTE.yml>`_
+in the directory ``ESMValTool/esmvaltool/cmorizers/data/cmor_config/``. Note
+that both the name of this configuration file and the cmorizing script have to be
+identical to the name of your dataset. 
+It is recommended that you set ``project`` to ``OBS6`` in the
 configuration file. That way, the variables defined in the CMIP6 CMOR table,
 augmented with the custom variables described above, are available to your script.
 
@@ -124,28 +174,31 @@ If a single dataset has more than one reference,
 it is possible to add tags as a list e.g. ``reference: ['tag1', 'tag2']``.
 The third part in the configuration file defines the variables that are supposed to be cmorized.
 
-The actual cmorizing script ``cmorize_obs_mte.py`` consists of a header with
+The actual cmorizing script ``mte.py`` consists of a header with
 information on where and how to download the data, and noting the last access
 of the data webpage.
 
 The main body of the CMORizer script must contain a function called
 
 .. code-block:: python
 
-   def cmorization(in_dir, out_dir, cfg, config_user):
+   def cmorization(in_dir, out_dir, cfg, cfg_user, start_date, end_date):
 
 with this exact call signature. Here, ``in_dir`` corresponds to the input
 directory of the raw files, ``out_dir`` to the output directory of final
-reformatted data set and ``cfg`` to the configuration dictionary given by
-the  ``.yml`` configuration file. The return value of this function is ignored. All
+reformatted data set, ``cfg`` to the dataset-specific configuration file,
+``cfg_user`` to the user configuration file, ``start_date`` to the start
+of the period to format, and ``end_date`` to the end of the period to format.
+If not needed, the last three arguments can be ignored using underscores.
+The return value of this function is ignored. All
 the work, i.e. loading of the raw files, processing them and saving the final
 output, has to be performed inside its body. To simplify this process, ESMValTool
 provides a set of predefined utilities.py_, which can be imported into your CMORizer
 by
 
 .. code-block:: python
 
-   from . import utilities as utils
+   from esmvaltool.cmorizers.data import utilities as utils
 
 Apart from a function to easily save data, this module contains different kinds
 of small fixes to the data attributes, coordinates, and metadata which are
@@ -157,16 +210,16 @@ that code style). For example, the function ``_get_filepath`` converts the raw
 filepath to the correct one and the function ``_extract_variable`` extracts and
 saves a single variable from the raw data.
 
-.. _utilities.py: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/obs/utilities.py
+.. _utilities.py: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/utilities.py
 
 
 4.2 Cmorizer script written in NCL
 ----------------------------------
 
 Find here an example of a cmorizing script, written for the ``ESACCI XCH4``
 dataset that is available on the Copernicus Climate Data Store:
-`cmorize_obs_cds_xch4.ncl
-<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/obs/cmorize_obs_cds_xch4.ncl>`_.
+`cds_xch4.ncl
+<https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/formatters/datasets/cds_xch4.ncl>`_.
 
 The first part of the script collects all the information about the dataset
 that are necessary to write the filename correctly and to understand which
@@ -183,20 +236,14 @@ CMOR_TABLE.
   through the loading of the script interface.ncl_. There are similar
   functions available for python scripts.
 
-.. _interface.ncl: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/obs/interface.ncl
+.. _interface.ncl: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/formatters/interface.ncl
 
-.. _utilities.ncl: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/obs/utilities.ncl
+.. _utilities.ncl: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/formatters/utilities.ncl
 
 In the second part of the script each variable defined in ``VAR`` is separately
 extracted from the original data file and processed. Most parts of the code are
 commented, and therefore it should be easy to follow. ESMValTool provides a set
-of predefined utilities.ncl_, which can be imported into your CMORizer
-by
-
-.. code-block:: NCL
-
-   loadscript(getenv("esmvaltool_root") + "/esmvaltool/cmorizers/obs/utilities.ncl")
-
+of predefined utilities.ncl_, which are imported by default into your CMORizer.
 This module contains different kinds of small fixes to the data attributes,
 coordinates, and metadata which are necessary for the data field to be
 CMOR-compliant.
@@ -208,20 +255,31 @@ The cmorizing script for the given dataset can be run with:
 
 .. code-block:: console
 
- cmorize_obs -c <config-user.yml> -o <dataset-name>
+ esmvaltool data format --config_file <config-user.yml> <dataset-name>
 
 
 .. note::
 
    The output path given in the configuration file is the path where
    your cmorized dataset will be stored. The ESMValTool will create a folder
-   with the correct tier information (see Section `2. Edit your configuration file`_) if that tier folder is not
-   already available, and then a folder named after the data set. In this
-   folder the cmorized data set will be stored as a netCDF file.
+   with the correct tier information 
+   (see Section `2. Edit your configuration file`_) if that tier folder is not
+   already available, and then a folder named after the dataset. 
+   In this folder the cmorized data set will be stored as a NetCDF file.
+   The cmorized dataset will be automatically moved to the correct tier
+   subfolder of your OBS or OBS6 directory if the option 
+   ``--install=True`` is used in the command above and no such directory
+   was already created.
 
 If your run was successful, one or more NetCDF files are produced in your
 output directory.
 
+If a downloading script is available for the dataset, the downloading and
+the cmorizing scripts can be run in a single command with:
+
+.. code-block:: console
+
+ esmvaltool data prepare --config_file <config-user.yml> <dataset-name>
 
 6. Naming convention of the observational data files
 ====================================================
@@ -265,8 +323,8 @@ The different parts of the name are explained in more detail here:
   (``mon``);
 - xch4: Is the name of the variable. Each observational data file is supposed
   to only include one variable per file;
-- 200301-201612: Is the period the dataset spans with ``200301`` being the
-  start year and month, and ``201612`` being the end year and month;
+- 200301-201812: Is the period the dataset spans with ``200301`` being the
+  start year and month, and ``201812`` being the end year and month;
 
 .. note::
    There is a different naming convention for ``obs4MIPs`` data (see the exact

doc/sphinx/source/input.rst

@@ -36,6 +36,7 @@ or, if you need longer term access or more computational resources, the
 
 If the options above are not available to you, ESMValTool also offers a feature
 to make it easy to download CMIP6, CMIP5, CMIP3, CORDEX, and obs4MIPs from ESGF.
+ESMValTool also provides support to download some observational dataset from source.
 
 The chapter in the ESMValCore documentation on
 :ref:`finding data <esmvalcore:findingdata>` explains how to
@@ -71,7 +72,54 @@ Observations
 
 Observational and reanalysis products in the standard CF/CMOR format used in CMIP and required by the ESMValTool are available via the obs4MIPs and ana4mips projects at the ESGF (e.g., https://esgf-data.dkrz.de/projects/esgf-dkrz/). Their use is strongly recommended, when possible.
 
-Other datasets not available in these archives can be obtained by the user from the respective sources and reformatted to the CF/CMOR standard. ESMValTool currently support two ways to perform this reformatting (aka 'CMORization'). The first is to use a CMORizer script to generate a local pool of reformatted data that can readily be used by the ESMValTool. The second way is to implement specific 'fixes' for your dataset. In that case, the reformatting is performed 'on the fly' during the execution of an ESMValTool recipe (note that one of the first preprocessor tasks is 'CMOR checks and fixes'). Below, both methods are explained in more detail.
+Other datasets not available in these archives can be obtained by the user from the respective sources
+and reformatted to the CF/CMOR standard.
+The list of datasets supported by ESMValTool can be obtained with:
+
+.. code-block:: bash
+
+    esmvaltool data list
+
+Datasets for which auto-download is supported can be downloaded with:
+
+.. code-block:: bash
+
+    esmvaltool data download --config_file [CONFIG_FILE] [DATASET_LIST]
+
+Note that all Tier3 and some Tier2 datasets for which auto-download is supported
+will require an authentification. In such cases enter your credentials in your
+``~/.netrc`` file as explained
+`here <https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html>`_.
+
+An entry to the ``~/.netrc`` should look like:
+
+.. code-block:: bash
+
+    machine [server_name] login [user_name] password [password]
+
+Make sure that the permissions of the ``~/.netrc`` file are set so only you and administrators
+can read it, i.e. 
+
+.. code-block:: bash
+
+    chmod 600 ~/.netrc
+    ls -l ~/.netrc
+
+The latter command should show ``-rw-------``.
+
+For other datasets, downloading instructions can be obtained with:
+
+.. code-block:: bash
+
+    esmvaltool data info [DATASET]
+
+ESMValTool currently support two ways to perform this reformatting (aka 'CMORization').
+The first is to use a CMORizer to generate a local pool of reformatted data that can
+readily be used by the ESMValTool.
+The second way is to implement specific 'fixes' for your dataset.
+In that case, the reformatting is performed 'on the fly' during the execution of an ESMValTool
+recipe (note that one of the first preprocessor tasks is 'CMOR checks and fixes').
+Below, both methods are explained in more detail.
 
 Using a CMORizer script
 -----------------------
@@ -89,7 +137,7 @@ To CMORize one or more datasets, run:
 
 .. code-block:: bash
 
-    cmorize_obs -c [CONFIG_FILE] -o [DATASET_LIST]
+    esmvaltool data format --config_file [CONFIG_FILE] [DATASET_LIST]
 
 The path to the raw data to be CMORized must be specified in the
 :ref:`user configuration file<config-user>` as RAWOBS.
@@ -117,7 +165,7 @@ may be ``sat`` (satellite data), ``reanaly`` (reanalysis data),
 ``ground`` (ground observations), ``clim`` (derived climatologies),
 ``campaign`` (aircraft campaign).
 
-At the moment, cmorize_obs supports Python and NCL scripts.
+At the moment, ``esmvaltool data format`` supports Python and NCL scripts.
 
 .. _cmorization_as_fix: