🤝 Implement AWS Transfer Family Server

[Gary-H9]

User Story

As a… User of the Analytical Platform
I want to be able to ingest data into the platform
So that… I can use all of the tooling etc within the platform

As a AP Product Engineer
I want to the platform to be able to ingest data in a controlled, precise and monitored fashion.
So that we can provide a better service for our users and build a foundational offering within the platform.

This ticket builds on this previously raised Feature Request.

Value / Purpose

Data ingestion will be a foundational part of the AP offering going forward. This piece of work will create the foundations which this offering will be built on.

Useful Contacts

Jacob W / Julia / Gary

Proposal

Create the ingestion route as outlined here.

Additional Information

For the AWS Transfer Family Server - Electronic Monitoring already use this functionality.

Definition of Done

• 📝 Documentation has been written / updated
• Modernisation Platform AWS Account created (WIP)
• AWS Transfer Family Server Terraform'ed
• IAM Roles defined + Terraform'ed
• User testing arranged
• Testing completed

[jacobwoffenden]

29/02/24 summary:

• @Gary-H9 requested new Modernisation Platform environment Creation of analytical-platform-ingestion  modernisation-platform#6364

[jacobwoffenden]

05/03/24 summary:

• @jacobwoffenden 🦪 Add ingestion scanner repository #3602
• @Gary-H9
  • initial configuration of ingestion scanning container - WIP PR
  • Addition of CloudWatch EventBridge, lambda function and associated testings of the above in concert with these
  • Modernisation Platform environment build in progress

[jacobwoffenden]

06/03/24:

• Successfully tested flow for updating and then consuming ClamAV definitions
• Successfully tested scanning files using S3 bucket notifications
  • Clean files moved to a processed bucket, while infected are moved to quarantine
    • Files are deleted from landing once moved

TODO:

• Add object policy to deny access to quarantined
• Configure Transfer Family server
  • Home directory
  • Move into VPC and give an EIP
  • Custom hostname?
  • Restrict users to specific IP ranges
• Notification for quarantined objects
  • Can we use a tag from the transfer user to send an email? and in the future a message to AP dashboard Split into 🔧 🗣️ Ingestion Notifications #3770

Notes:

• Electronic Monitoring create a server per provider, we don't want to do this, could be hard to scale and manager via AP dashboard

EDIT @jacobwoffenden:

I've managed to get this working by editing the IAM policy for the user to include more S3 permissions, and added KMS permissions

EDIT 2 @jacobwoffenden:

• figure out how/where to run aws transfer update-server to set DirectoryListingOptimization to ENABLED. this is not exposed in Terraform MP ClickOps'd it

[Gary-H9]

07/03/24:

• Converted the lambda function to bash from python. Noted faster runs as a result. Tested both scenarios successfully.
• Planned architecture relating to notifications in variety of scenarios.

[jacobwoffenden]

11/04/24 summary:

• More work on scan functionality
  • sends a message to SNS when quarantining a file
• Started work on notify functionality
  • Triggers on SNS publish
  • Plumbed into Data Platform's GOV.UK Notify account, have tested basic functionality
• Drafted architecture for "supplier data", or information about supplier we don't want to store in public Terraform, this will be superseded by AP dashboard eventually
• Liaised with GDS about getting access to Analytical Platform's notify account

TODO:

• SNS redrive logic
• clarify naming/terminology (data contact, data owner, etc.)

[Gary-H9]

14th March summary:

• Refactored existing code + added code for new lambda and required components
• Removed calls to SNS for this iteration of the product - this can be looked at after MVP - ticket created for this
• Created ingestion-transfer handler.py
• Created a test bucket in analytical-platform-dev called dev-ingestion-testing to test cross account file move from the above lambda. Manually updated the policy on this but it needs further work.

[jacobwoffenden]

18/03/2024 summary (plus a bit more):

• 🔭 Adds Analytical Platform's Ingestion accounts to Observability Platform modernisation-platform-environments#5371
• 🌐 Extend Analytical Platform Ingestion's VPC modernisation-platform-environments#5400

Gary:

• continued development on the ingestion-transfer image.

[Gary-H9]

• Closed development branch/PR in data-platform repository
• Migrating code from the above branch into Modernisation Platform Environments repository. Merged - development and production built.
• Updated DNS record for both environments in analytical-platform-production R53.

[Gary-H9]

Created documentation relating to the solution in user-guidance (🚧) and in our new runbooks documentation.

[jacobwoffenden]

@ministryofjustice/modernisation-platform have enabled optimised directories manually in both transfer servers (dev and prod)

[jacobwoffenden]

hashicorp/terraform-provider-aws#35851

[Gary-H9]

Awaiting user information to allow testing. In the meantime egress has been completed.

[Gary-H9]

• Locked down S3 Bucket Permissions on the Quarantine bucket
• Egress functionality
• No update from users RE testing.

[jacobwoffenden]

Pending details from BOLD to begin onboarding

[jacobwoffenden]

Data Engineering's https://github.com/moj-analytical-services/iam_builder needs updating to add kms which is needed to add KMS permissions to their Airflow role

[jacobwoffenden]

Analytical Platform team to update Airflow IAM role with permissions to access KMS key

• arn:aws:iam::593291632749:role/airflow_dev_bold_rr_essex_police

[jacobwoffenden]

Implemented regional KMS

• 🌐 Set BOLD egress KMS keys to multi region #4106
• 🇮🇪 Add eu-west-1 replica of BOLD KMS #4108

[jacobwoffenden]

Moving to blocked:

• BOLD Airflow role needs access to KMS, release made to IAM Build (https://github.com/moj-analytical-services/iam_builder/releases/tag/v4.3.0)
• DE's mojap-land S3 buckets need their policy updating to allow ingestion service

[michaeljcollinsuk]

Moving back to in progress as @julialawrence is working on a new request from BOLD

[jacobwoffenden]

Blocked by #3765

[jacobwoffenden]

Closing as we've tested end-to-end, we're waiting on BOLD to perform their end-to-end testing which is out of scope for this issue.