diff --git a/doc/integrator/configuration.rst b/doc/integrator/configuration.rst index 1bf64a7bdb..937e44d592 100644 --- a/doc/integrator/configuration.rst +++ b/doc/integrator/configuration.rst @@ -22,3 +22,4 @@ This chapter describes advanced configuration settings. ngeo_new_interface multiple_databases windows_tools + objectstorage diff --git a/doc/integrator/objectstorage.rst b/doc/integrator/objectstorage.rst new file mode 100644 index 0000000000..6e65414328 --- /dev/null +++ b/doc/integrator/objectstorage.rst @@ -0,0 +1,109 @@ +.. _integrator_objectstorage: + +Working with Object storage (like S3) +===================================== + +Generalities +------------ + +In this we will consider to use the S3-like from Exoscale. + +First of all we should set the following variables on the ``geoportal``, ``config`` and ``qgisserver`` services. + +* ``AWS_ACCESS_KEY_ID``: The project access key. +* ``AWS_SECRET_ACCESS_KEY``: The project secret key. +* ``AWS_DEFAULT_REGION=ch-dk-2``: The region used by Exoscale. +* ``AWS_S3_ENDPOINT=sos-ch-dk-2.exo.io``: The endpoint used by Exoscale. + +For better performance, we should set the following variables on the ``geoportal`` and ``qgisserver`` services. + +* ``CPL_VSIL_CURL_USE_CACHE=TRUE`` +* ``CPL_VSIL_CURL_CACHE_SIZE=128000000`` +* ``CPL_VSIL_CURL_USE_HEAD=FALSE`` +* ``GDAL_DISABLE_READDIR_ON_OPEN=TRUE`` + + +Use the aws client to list the files: + +.. prompt:: bash + + aws --endpoint-url https://sos-ch-dk-2.exo.io/ --region ch-dk-2 s3 ls s3:/// + + +Create the vrt file for a raster layer: + +.. prompt:: bash + + gdalbuildvrt `aws --endpoint-url https://sos-ch-dk-2.exo.io/ --region ch-dk-2 s3 ls \ + s3:////*.geotiff|awk '{print $4}'` s3:////index.vrt + +MapServer +--------- + +Create the shape index file for a raster layer: + +.. prompt:: bash + + gdaltindex mapserver/index.shp `aws --endpoint-url https://sos-ch-dk-2.exo.io/ --region ch-dk-2 s3 ls \ + s3:////*.geotiff|awk '{print $4}'` + + +Add the following config in the ``mapserver/mapserver.map.tmpl`` file: + +.. code:: + + CONFIG "CPL_VSIL_CURL_USE_CACHE" "TRUE" + CONFIG "CPL_VSIL_CURL_CACHE_SIZE" "128000000" + CONFIG "CPL_VSIL_CURL_USE_HEAD" "FALSE" + CONFIG "GDAL_DISABLE_READDIR_ON_OPEN" "TRUE" + + CONFIG "AWS_ACCESS_KEY_ID" "${AWS_ACCESS_KEY_ID}" + CONFIG "AWS_SECRET_ACCESS_KEY" "${AWS_SECRET_ACCESS_KEY}" + CONFIG "AWS_DEFAULT_REGION" "${AWS_DEFAULT_REGION}" + CONFIG "AWS_S3_ENDPOINT" "${AWS_S3_ENDPOINT}" + +Use the shape index in the layer: + +.. code:: + + TYPE RASTER + STATUS ON + PROCESSING "RESAMPLE=AVERAGE" + CONNECTIONTYPE OGR + TILEINDEX "index.shp" + TILEITEM "LOCATION" + +Add a vector layer for the object storage: + +.. code:: + + CONNECTIONTYPE OGR + CONNECTION "/vsis3///.shp" + DATA "" + +`Some more information `_ + +QGIS client +----------- + +Open settings, Option and define the following environment variables: + +* ``AWS_ACCESS_KEY_ID``: The project access key. +* ``AWS_SECRET_ACCESS_KEY``: The project secret key. +* ``AWS_DEFAULT_REGION=ch-dk-2``: The region used by Exoscale. +* ``AWS_S3_ENDPOINT=sos-ch-dk-2.exo.io``: The endpoint used by Exoscale. + +On Windows also add: + +* ``GDAL_HTTP_UNSAFESSL=YES`` + +Then you can add a raster layer with: + +* Open Data Source Manager, +* Raster, +* Protocol: HTTP(S), cloud, etc., +* Type: AWS S3 +* Bucket or container: +* Object key: /index.vrt + +You can add a vector layer in an analogous manner. diff --git a/docker/build/Dockerfile b/docker/build/Dockerfile index c999164458..8032370304 100644 --- a/docker/build/Dockerfile +++ b/docker/build/Dockerfile @@ -28,6 +28,7 @@ RUN \ COPY sudoers /etc/ COPY bin/* /usr/bin/ +COPY etc/bash_completion.d/* /etc/bash_completion.d/ WORKDIR /src diff --git a/docker/build/etc/bash_completion.d/aws_bash_completer b/docker/build/etc/bash_completion.d/aws_bash_completer new file mode 100644 index 0000000000..6a18c75f9e --- /dev/null +++ b/docker/build/etc/bash_completion.d/aws_bash_completer @@ -0,0 +1 @@ +complete -C aws_completer aws diff --git a/docker/build/requirements.txt b/docker/build/requirements.txt index ee87b41307..9158136b73 100644 --- a/docker/build/requirements.txt +++ b/docker/build/requirements.txt @@ -1,5 +1,6 @@ alabaster==0.7.10 # Doc theme alembic==0.9.9 # geoportal +awscli==1.16.80 # Work with Object storage Babel==2.5.3 # i18n beautifulsoup4==4.6.3 # admin tests boto3==1.7.4 # Tile generation