Ichnosat is an Open Source plugin based platform able to download and process EO satellite data. It is written in Python and C++, has been designed and developed by Raffaele Bua for Sardegna Clima Onlus NPO. Downloads and Processes are multithreaded and schedulable. The Platform is configurable and the processor is extendible via C++ plugins.
Docker containers are included in the package, facilitating the installation and running.
Source code is released under the MIT license and it is available on GitHub.
The downloader gets automatically products from Sentinel-2 on AWS portal. Such a downloader is multithreaded and can be fully customized in terms of AOI (Area of Interest) and Products Sensing Time.
The processor gets products from the downloader and processes products via C++ plugins. Current version contains a single plugin able to extract from Sentinel-2 data the NDVI exploiting GDAL library.
Ichnosat processor is multithreaded, extendible via C++ plugins and customizable.
-
Cross-platform support
- Ichnosat Platform provides the Docker containers
-
Downloader
- Get Sentinel-2 data from Sentinel-2 on AWS
- Customizable features:
- Tiles to download (UTM-MGRS Tile-ID)
- Files to download for each tile (Image files and/or metadata)
- Dates interval (Sensing time)
- Number of parallel downloads at the same time
-
Processor
- Plugin based: the processors are extendible C++ Shared Libraries
- Customizable features:
- Number of parallel processings at the same time
-
Ichnosat Platform
- Automatic plugin compilation
- HTTP APIs to manage the system
- Customizable cron for scheduling run
-
Graphical User Interface (Web based)
- Run Downloader and Processor
- Show product lists
- Compile plugins
- Manage Database
-
Testing
- Unit tests run at Platform start-up
-
API-documentation
- Autogenerated API-documentation via Sphinx from comments in the code (PEP 257 Convention)
Downloader and Processor are the main modules of this platform. The schema with all modules involved follows:
This module is deputed to retrieve the list of available products and download them, on the basis of the configurations set by user. The possible configurations to filter the list of products are:
- List of tiles
- List of files to download for each tile
- Sensing time interval (start - end )
The goal of this component is to process downloaded products. Every processing algorithm is represented by C++ plugins. This means that Processor module is extendible. Plugins are dynamic shared library (Linux environment)
The processor runs as http server to receive requests via http:
When the POST /process http is received the Processor starts a processing task, spreading a pool of threads. The number of threads is configurable via conf.cfg file.
If you want to develop a new plugin for your processing purposes, please follows the tutorial:
System Manager is the high level module to manage other modules.
Database module contains the classes of ORM. In this version (0.1) is used SqlAlchemy ORM.
The description of the main folders in the source code follows:
|-- Dockerfile # Docker file of ichnosat platform
|-- LICENSE.TXT # MIT license text
|-- README.md # README of the project (for github main page)
|-- auto-docs # Documentation sources and images
|-- data_local
| |-- db # Postgresql database files (mounted by docker-compose)
| |-- inbox # Temporary downloaded products by the Downloader
| |-- log # Ichnosat logs
| |-- outbox # Processed products folder
| |-- supervisord # Supervisord log folder
|-- docker-compose.yml # Docker Compose definition file
|-- src
| |-- core # Main Modules source code (Downloader, Processor, System Manager)
| |-- data # Database and Logger source code
| |-- gui # Web GUI interface source code
| |-- presentation # External interface (http) source code
| `-- tests # Tests
`-- vendors # Third part dependencies
- Docker and docker compose: virtual image container
- SqlAlchemy: ORM framework for python
- Supervisord: Task manager
- Valgrind: Process monitoring
- Sphinx: Automatic Documentation for python
- Flask: light webserver
- Nginx: web server
- Postgresql: database
- GDAL: C/C++ library for satellite image processing
- OpenJPG: library to manage jpeg2000 images
- Python 3.4
- crontab
- debian
- pip
- C++11