Skip to content

Installation

Jump to bottom
Lewis John McGibbney edited this page Mar 16, 2023 · 20 revisions

Download

Simply grab the most recent version from our public release site. You can then run the software stack in one of two ways. We strongly recommend running the tagbase-server stack with Docker.

Running with Docker

Docker enables rapid simplified deployment of Tagbase by removing all services setup and configuration e.g. PostgreSQL, tagbase-server, etc. This is achieved via Docker Compose; a tool for defining and running multi-container Docker applications.

See below for prerequisite installation requirements.

Prerequisites

Either download the latest release of tabase-server OR clone the source code with Git

Once you've acquired the tagbase-server source code on your workstation, you need to navigate to the source root directory e.g.

$ cd tagbase-server

Configuration Management

Ensure Docker has enough memory

N.B. Due to the size of the input datasets we ingest into tagbase-server, it is essential that the container running the service has sufficient available memory. 8GiB should be sufficient however you are strongly encouraged to experiment as more memory may be required..

See this for Mac:

MEMORY By default, Docker for Mac is set to use 2 GB runtime memory, allocated from the total available memory on your Mac. You can increase the RAM on the app to get faster performance by setting this number higher...

For Windows:

Memory - Change the amount of memory the Docker for Windows Linux VM uses

docker-compose configuration

tagbase-server relies on one simple docker-compose configuration file named .env. You must create this file in the same directory as docker-compose.yml before you can deploy the tagbase-server stack. Here is an example of the required .env configuration

PGADMIN_DEFAULT_EMAIL=...
PGADMIN_DEFAULT_PASSWORD=...
POSTGRES_PASSWORD=...                <<<<<<<<< if you wish to change this, see note 1 below
POSTGRES_PORT=...
#SLACK_BOT_CHANNEL=..
#SLACK_BOT_TOKEN=...                 
#webhook=...                         <<<<<<<<< please see note 2 below

NOTE 1: The default password can be changed here but is also needs to be changed in tagbase_schema.sql.

NOTE 2: See documentation for tagbase-server and Slack integration

Specifying TLS Certificates

See the relevant documentation.

Deployment

To orchestrate and deploy the Tagbase services execute the following from this root directory:

N.B. Make sure you make the relevant substitutions for the ENV VARS below

docker-compose build --build-arg POSTGRES_PASSWORD="${POSTGRES_PASSWORD}" --build-arg POSTGRES_PORT="${POSTGRES_PORT}" --build-arg NGINX_USER="${NGINX_USER}" --build-arg NGINX_PASS="${NGINX_PASS}"

Now deploy the composition...

docker-compose up

N.B. you can run with -d to disconnect from running processes

After a short while, you will see now have a running docker composition comprising the following stack

docker ps
...
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                    PORTS                                              NAMES
7eebb8b7b993   tagbase-server-nginx            "/docker-entrypoint.…"   14 minutes ago   Up 14 minutes             0.0.0.0:81->81/tcp, 80/tcp, 0.0.0.0:443->443/tcp   tagbase-server-nginx-1
1c4eb8354030   tagbase-server-tagbase_server   "gunicorn tagbase_se…"   14 minutes ago   Up 14 minutes             5433/tcp                                           tagbase-server-tagbase_server-1
d91fb770abf1   tagbase-server-docker-cron      "./entrypoint.sh"        14 minutes ago   Up 14 minutes                                                                tagbase-server-docker-cron-1
3d3f07463f02   dpage/pgadmin4:6.17             "/entrypoint.sh"         14 minutes ago   Up 14 minutes             80/tcp, 443/tcp, 5434/tcp                          tagbase-server-pgadmin4-1
f62787c8157e   tagbase-server-postgres         "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes (healthy)   0.0.0.0:5432->5432/tcp                             tagbase-server-postgres-1

To stop the docker-compose deployment, simply open a new terminal, navigate to the tagbase-server root directory and execute

docker-compose stop

You will see the services graciously shutdown.

Tagbase Server

Navigate to https://localhost/docs/ to see the landing page. From here you can find hyperlinks to

  • /docs: visualize and interact with tagbase-server REST APIs.
  • /pgadmin4: administration and development platform for Tagbase PostgreSQL instances.
  • /tagbase/${version}: REST API base path.

It should be noted that the OpenAPI specification is self-documenting. Reading it will really help as it fully explains how to ingest data via GET and POST and how to interact with tags ingested into tagbase-server.

N.B. The REST server is capable of ingesting data via GET (over file, ftp, http and https protocols) and via POST (.txt and binary files e.g. .zip, etc.)

See here for more information on ingestion patterns.

pgAdmin

Navigate to https://localhost/pgadmin4 and enter

username tagbase password tagbase

You can now:

  • Add New Server
  • General Tab --> name: tagbase
  • Connection Tab --> Host name/address: postgres
  • Connection Tab --> Port: 5432
  • Connection Tab --> Maintenance database: postgres
  • Connection Tab --> Username: ...

On the left hand side navigation panel, you will now see the persistent connection to the tagbase database.

Generating Materialized views

Upon successful ingestion of files into Tagbase, you are required to generate materialized views in order to access the 'application ready' tagbase data.

N.B. Previously, it was necessary to execute a data migration command which essentially populated initial staging data around the DB. This is now managed by a trigger such that all we need to worry about it generating materialized views.

PostgreSQL Materialized Views extend the concept of database views; virtual tables which represent data of the underlying tables, to the next level that allows views to store data physically, and we call those views materialized views. A materialized view caches the result of a complex expensive query and then allows you to refresh this result periodically.

You can generate the Tagbase materialized views by opening tagbase-materialized-views.sql

... and executing the contents as a query within the pgAdmin4 Query Tool.

Similar to the ingestion and migration routines, generation of the materialized views may take a while so be patient. Once it has completed however, you can browse the materialized views.

N.B. It should be noted that materialized views can only be generated once... this process should not be executed every time a file is ingested!

Running Manually (To be updated)

See https://wiki.earthdata.nasa.gov/display/OIIP/Manual+Installation+of+tagbase-server (Link not active right now)

Clone this wiki locally