FCDR_ST_ensemble

Code for calculation of the FCDR ST ensemble from the AVHRR Easy-FCDR harmonisation parameters and covariance matrix.

Contents

• setup.py - main build script to be run with Python 3.6
• convert_func.py - functions for radiance <--> brightness temperature conversion functions plus measurement equations
• ensemble_func.py - functions for calculation of the ensemble
• inmyvenv.sh - conda environment activation script
• plot_cov2ensemble.py - plotting routines for cov2ensemble calculations
• plot_ensemble.py - plotting routines for ensemble calculations
• run_test_cov2ensemble.sh - submission script for test_cov2ensemble.py
• run_test_ensemble2bt.sh - submission script for test_ensemble2bt.py
• sample_binormal.py - 2D-normal distribution (joint) Monte Carlo sampling test
• test_cov2ensemble.py - generate ensemble given a covariance matrix and parameter mean values
• test_ensemble2bt.py - generate ensemble brightness temperatures given a radiance ensemble

Installation from Sources

We recommend installing FCDR_ST_ensemble into an isolated Python 3 environment, because this approach avoids clashes with existing versions of 3rd-party Python package requirements. For example, using (Miniconda or Anaconda) which will usually also avoid platform-specific issues caused by module native binaries.

The first step is to clone latest FCDR_ST_ensemble code and step into the check out directory:

$ git clone https://github.com/FIDUCEO/FCDR_ST_ensemble.git
$ cd FCDR_ST_ensemble

Using Conda

Conda is the package manager used by the Miniconda or Anaconda Python distributions.

To create a new Conda environment FCDR_ST_ensemble-env in your Anaconda/Miniconda installation directory, type:

$ conda env create

If you want the environment to be installed in another location, e.g. due to disk space limitations, type:

$ conda env create --prefix some/other/location/for/FCDR_ST_ensemble

Next step is to activate the new environment. On Linux/Darwin type:

$ source activate FCDR_ST_ensemble-env

In case you used another location use it instead of the name FCDR_ST_ensemble. Windows users can omit the source command and just type

> activate FCDR_ST_ensemble-env

You can now safely install FCDR_ST_ensemble sources into the new FCDR_ST_ensemble-env environment.

(FCDR_ST_ensemble) $ python setup.py install

Using Standard Python

If you run it with the standard CPython installation, make sure you use a 64-bit version. FCDR_ST_ensemble relies on new Python language features and therefore requires Python 3.6+.

FCDR_ST_ensemble can be run from sources directly, once the following module requirements are resolved:

• matplotlib
• netCDF4
• numba
• numpy
• pandas
• scipy
• seaborn
• sklearn
• xarray

The most up-to-date list of module requirements is found in the project's environment.yml file.

To install FCDR_ST_ensemble into an existing Python 3.6+ environment just for the current user, use

$ python3 setup.py install --user

To install FCDR_ST_ensemble for development and for the current user, use

$ python3 setup.py develop --user

There is a known issue on Windows when installing into an existing Python environment. Installation may fail due to an unresolved dependency to the h5py package, which expects pre-installed HDF-5 C-libraries to be present on your computer. You may get around this by pre-installing the FCDR_ST_ensemble dependencies (which you'll find in setup.py) on your own, for example by using Christoph Gohlke's Unofficial Windows Binaries for Python Extension Packages.

Getting started

To test the installation, first run the FCDR_ST_ensemble command-line interface. Type

$ FCDR_ST_ensemble -h

IPython notebooks for various FCDR_ST_ensemble use cases are on the way, they will appear in the project's notebooks folder.

To use them interactively, you'll need to install Jupyter and run its Notebook app:

$ conda install jupyter
$ jupyter notebook

Open the notebooks folder and select a use case.

Conda Deployment

There will be a dedicated repository FCDR_ST_ensemble-conda that will provide scripts and configuration files to build FCDR_ST_ensemble's Conda packages and stand-alone installer.

Development

Contributors

Thanks go to the members of the FIDUCEO project consortium for making the data required available.

Unit-testing

Unit testing will be performed using pytest and its coverage plugin pytest-cov.

To run the unit-tests with coverage, type

$ export NUMBA_DISABLE_JIT=1
$ py.test --cov=FCDR_ST_ensemble test

We need to set environment variable NUMBA_DISABLE_JIT to disable JIT compilation by numba, so that coverage reaches the actual Python code. We use Numba's JIT compilation to speed up numeric Python number crunching code.

Generating the Documentation

The documentation will be generated with the Sphinx tool to create a ReadTheDocs. If there is a need to build the docs locally, some additional software packages are required:

$ conda install sphinx sphinx_rtd_theme mock
$ conda install -c conda-forge sphinx-argparse
$ pip install sphinx_autodoc_annotation

To regenerate the HTML docs, type

$ cd doc
$ make html

License

The code is distributed under terms and conditions of the MIT license.

Contact information

• Michael Taylor (https://patternizer.github.io)