Initial commit

.gitignore

@@ -0,0 +1,14 @@
+__pycache__
+.DS_Store
+*.log
+*.pdf
+.ipynb_checkpoints
+.vscode
+
+datefile.txt
+account.ini
+
+data/ground_truth/*.csv
+data/shapefiles/*/*
+sentinel_api
+plots

LICENSE

@@ -0,0 +1,30 @@
+BSD 3-Clause License
+
+Copyright (c) 2019, Philipp Wagner, Felix M. Riese, Sina Keller. Karlsruhe
+Institute of Technology.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,68 @@
+# CAOS Sentinel-2 Pipeline in Python
+
+Pipeline for the processing of Sentinel-2 data and ground truth in the CAOS-Lux project
+
+Workflow for creating a dataset out of soil moisture data and Sentinel-2A
+images for the CAOS-lux soil moisture dataset in the Attert region of Luxemburg.
+The data was aquired from 2014 to 2018.
+
+**License:** [3-Clause BSD license](LICENSE)
+
+**Authors:**
+
+* [Philipp Wagner](https://github.com/philipp01wagner)
+* [Felix M. Riese](https://github.com/felixriese)
+* [Sina Keller](https://github.com/sinakeller)
+
+**Citation:** See [Citation](#citation) and [bibliography.bib](bibliography.bib).
+
+## Requirements
+
+The dependencies and requirements can be found in `/dependencies`.
+
+Install dependencies with:
+
+```bash
+./install_dependencies_and_scihub_account.sh
+```
+
+This script asks also for the Scihub account data which is stored in
+`account.ini`.
+
+## Howto
+
+To create a dataset the script `py/create_dataset.py` has to be executed with
+the desired date as command line argument in the format YYYYMMDD, e.g.:
+
+```bash
+python create_dataset.py 20161204
+```
+
+Alternative is to give a file as argument with a date in every line.
+A new directory named after the date will be created in the directory
+`/processed_data`. The script will download the desired S2A file, extract the
+zip, and construct gtiffs for the measured area and the prediction area.
+The IPython Notebooks provide an analysis of the constructed data. They include
+visualization of soil moisture and Sentinel-bands, further pre-processing and
+filtering and regression/classification where the soil moisture is predicted
+based on the Sentinel-data. For the machine learning part, the common
+regressors/classificators are used:
+
+-  Random Forest,
+-  Extra Trees,
+-  Support Vector Machine,
+-  Multilayer Perceptron.
+
+## Citation
+
+Philipp Wagner, Felix M. Riese and Sina Keller, "CAOS Sentinel-2 Pipeline in
+Python", Code, Zenodo, 2020.
+
+```tex
+@misc{wagner2020caos,
+  author       = {Wagner, Philipp and Riese, Felix~M. and Keller, Sina},
+  title        = {CAOS Sentinel-2 Pipeline in Python},
+  year         = 2020,
+  publisher    = {Zenodo}
+}
+```

dependencies/dependencies_linux.txt

@@ -0,0 +1,117 @@
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+@EXPLICIT
+https://repo.anaconda.com/pkgs/main/linux-64/blas-1.0-mkl.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/ca-certificates-2018.03.07-0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/intel-openmp-2019.1-144.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-8.2.0-hdf63c60_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libgfortran-ng-7.3.0-hdf63c60_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libstdcxx-ng-8.2.0-hdf63c60_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/poppler-data-0.4.9-0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/bzip2-1.0.6-h14c3975_5.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/expat-2.2.6-he6710b0_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/freexl-1.0.5-h14c3975_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/geos-3.6.2-heeff764_2.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/giflib-5.1.4-h14c3975_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/icu-58.2-h9c2bf20_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jpeg-9b-h024ee3a_2.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/json-c-0.13.1-h1bed415_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libffi-3.2.1-hd88cf55_4.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libuuid-1.0.3-h1bed415_2.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libxcb-1.13-h1bed415_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/mkl-2018.0.3-1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/ncurses-6.1-he6710b0_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/openssl-1.0.2p-h14c3975_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pcre-8.42-h439df22_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pixman-0.34.0-hceecf20_3.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/proj4-5.0.1-h14c3975_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/xz-5.2.4-h14c3975_4.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.11-h7b6447c_3.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/glib-2.56.2-hd408876_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/hdf4-4.2.13-h3ca952b_2.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/hdf5-1.8.18-h6792536_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libboost-1.67.0-h46d08c1_4.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libedit-3.1.20170329-h6b74fdf_2.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libpng-1.6.35-hbc83047_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libssh2-1.8.0-h9cfc8f7_4.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libtiff-4.0.9-he85c1e1_2.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libxml2-2.9.8-h26e45fe_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/readline-7.0-h7b6447c_5.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.8-hbc83047_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/xerces-c-3.2.2-h780794e_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/dbus-1.13.2-h714fa37_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/freetype-2.9.1-h8a8886c_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/gstreamer-1.14.0-hb453b48_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/kealib-1.4.7-h5472223_5.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/krb5-1.16.1-hc83ff2d_6.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libcurl-7.61.0-h1ad7b7a_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libkml-1.3.0-h590aaf7_4.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/openjpeg-2.3.0-h05c96fa_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/sqlite-3.25.3-h7b6447c_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/curl-7.61.0-h84994c4_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/fontconfig-2.13.0-h9420a91_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/gst-plugins-base-1.14.0-hbbd80ab_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libpq-10.5-h1ad7b7a_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libspatialite-4.3.0a-he475c7f_19.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/python-3.7.0-h6e4f718_3.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/affine-2.2.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/asn1crypto-0.24.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/atomicwrites-1.2.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/attrs-18.2.0-py37h28b3542_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/cairo-1.14.12-h8948797_3.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/certifi-2018.10.15-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/chardet-3.0.4-py37_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/click-7.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/docutils-0.14-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/idna-2.7-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jmespath-0.9.3-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/kiwisolver-1.0.1-py37hf484d3e_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libdap4-3.19.1-h6ec2957_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libnetcdf-4.6.1-h015f1c5_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/numpy-base-1.15.4-py37h81de0dd_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pluggy-0.8.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/py-1.7.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pycparser-2.19-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pyparsing-2.3.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pysocks-1.6.8-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pytz-2018.7-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/qt-5.9.6-h8703b6f_2.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/sip-4.19.8-py37hf484d3e_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/six-1.11.0-py37_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/tornado-5.1.1-py37h7b6447c_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/tqdm-4.28.1-py37h28b3542_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/cffi-1.11.5-py37he75722e_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/click-plugins-1.0.4-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/cligj-0.5.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/cycler-0.10.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/more-itertools-4.3.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/munch-2.3.2-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/poppler-0.65.0-h581218d_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pyqt-5.9.2-py37h05f1152_2.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/python-dateutil-2.7.5-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/python-utils-2.3.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/setuptools-40.6.2-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/cryptography-2.3.1-py37hc365091_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/libgdal-2.2.4-heea9cce_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pytest-4.0.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/wheel-0.32.3-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pip-18.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pyopenssl-18.0.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pytest-runner-4.2-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/progressbar2-3.37.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/urllib3-1.23-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/botocore-1.12.35-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/requests-2.20.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/s3transfer-0.1.13-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/boto3-1.9.35-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/gdal-2.2.4-py37h04863e7_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/matplotlib-3.0.1-py37h5429711_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/mkl_fft-1.0.6-py37h7dd41cf_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/mkl_random-1.0.1-py37h4414c95_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/numpy-1.15.4-py37h1d66e8a_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pandas-0.23.4-py37h04863e7_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/shapely-1.6.4-py37h7ef4460_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/snuggs-1.4.2-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/fiona-1.7.12-py37h3f37509_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/rasterio-0.36.0-py37h3f37509_2.tar.bz2