AusSeabed is a national seabed mapping coordination program. The program aims to serve the Australian community that relies on seabed data by coordinating collection efforts in Australian waters and improving data access.
This repository contains AWS lambda modules and AWS step function that process bathymetry products. The terraform code for deploying the underlying infrastructure is housed in the https://github.com/ausseabed/ausseabed-aws-foundation/ repository.
Most notably, the step functions that act as the interface for processing bathymetry are in: pipelines/step_functions/
The inputs for update_L3_warehouse.asl.json are:
{
"cat-url": "https://catalogue.ausseabed.gov.au/rest",
"bucket": "ausseabed-public-warehouse-bathymetry",
"proceed": true
}The inputs for process_L3.asl.json (generated by the previous workflow)
{
"bucket": "ausseabed-public-warehouse-bathymetry",
"s3_scaling_factor": "1",
"s3_src_tif": "s3://ausseabed-public-bathymetry/L3/4861_fk200429/FK200429_Transit_to_McDermott_30m_CUBE_WGS84_floatingPtGeo.tif",
"product-id": 154,
"s3_hillshade_dest_tif": "s3://ausseabed-public-warehouse-bathymetry/L3/c1bc994f-82e2-440c-877b-dbeb74c69a58/asdf_000_asfd_Ellipsoid_hs.tif",
"s3_dest_shp": "s3://ausseabed-public-warehouse-bathymetry/L3/c1bc994f-82e2-440c-877b-dbeb74c69a58/asdf_000_asfd_Ellipsoid.shp",
"s3_dest_tif": "s3://ausseabed-public-warehouse-bathymetry/L3/c1bc994f-82e2-440c-877b-dbeb74c69a58/asdf_000_asfd_Ellipsoid_cog.tif",
"uuid": "c1bc994f-82e2-440c-877b-dbeb74c69a58",
"cat-url": "https://catalogue.ausseabed.gov.au/rest"
}The inputs for process_L2.asl.json are:
{
"target-location": "s3://ausseabed-public-warehouse-bathymetry/L2/",
"uuid": "125d8c3b-1e07-4652-a01a-cb3d3aef880c",
"working-directory-root": "D:\\",
"l2-gsf-location": "s3://ausseabed-public-bathymetry/Clearing House/Geoscience Australia/transit/FK200930/EM302_L2_Patch/",
"product-name": "Great_Barrier_Reef_Cape_York_2020",
"vessel-file": "s3://ausseabed-public-bathymetry/Clearing House/Geoscience Australia/Vessel Files/RV_Falkor_EM302.hvf",
"license-server": "172.31.23.28",
"depth-ranges-file": "s3://ausseabed-public-bathymetry/Clearing House/Geoscience Australia/depth_ranges.txt",
"cube-config-file": "s3://ausseabed-public-bathymetry/Clearing House/Geoscience Australia/CUBEParams_AusSeabed_2019.xml",
"MSL-reference": "s3://ausseabed-public-bathymetry/Clearing House/Geoscience Australia/s45e135inv.asc",
"resolution": "64m"
}The continuous integration server (CircleCI) compiles, publishes and deploys the Product Catalogue to the development environment.
The CI server builds the lambdas and infrastructure and publishes them to non-production environment on any commit to master.
The Step Functions use the latest version of lambda functions. So once, a lambda is deployed to AWS, it will be used in any new Step Function workflows.
git checkout master
git pull
git tag prod/deploy/0.1.1
git push origin prod/deploy/0.1.1
If the size of the imagery in the L3 pipeline is too big for the standard GDAL image to handle. Switch to the gdalbigtiff image which uses an EFS for local storage. The required change should go in pipelines/step_function/process_L3.asl.json "TaskDefinition": ... It happens rarely enough that it hasn't been parameterised.