Protective Monitoring: Integration

[SimonPPledger]

User Story

As a product manager
I need to ensure that we have an appropriate level of protective monitoring
So that we have as secure a service as possible.

This ticket is to work with the protective monitoring team to capture the following levels of detail - for PRODUCTION only.
We are looking to provide the following info:

VPC flow logs
Networking live data
Network Firewall logs
Route 53

First step will me a conversation with Leo

Value / Purpose

This is to help security highlight issues

Useful Contacts

leonardo.marini@justice.gov.uk

Additional Information

For the integration work:

1. Route 53 - this should just require a configuration change in route 53

2. We then need to get the following 2 logs into a singel group in the tool (see below for previously used terraform):

• VPC flow logs
• Networking live data

3. And these logs into a separate group

• Network Firewall logs

This is the terraform code used by the LAN team.
module

Proposal / Unknowns

No response

Definition of Done

• Route 53 config config changes made
• Route 53 info ingested by monitoring tool
• Log files made available to the the monitoring tool
• Log files ingested by monitoring tool
• User docs have been updated
• Another team member has reviewed

[davidkelliott]

Route53 logs are also in cloudwatch, so we might want to do one of the other cloudwatch integrations first, then when we start to do route53 we will have the knowledge of the cloudwatch integration already so it may help decide which method is better.

[dms1981]

Hi @SimonPPledger. Just so I'm clear, this would cover us exporting log information into a remote tool?

We can do that for the VPC flow logs with a little bit of coordination with the requestor.

For Route53 I'd need to know exactly what they're after. There are a few different kinds of Route 53 logs - resolve log queries and public dns queries are the sort of thing I'd assume the requestor would want.

Likewise I'm confident we can give them the Network Firewall logs they're after, but it's worth understanding what kind of granularity they want from the logs. They may, for example, want logs for traffic that we deny (but do not log at present).

• https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-firehose.html
• https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/query-logs.html
• https://docs.aws.amazon.com/network-firewall/latest/developerguide/logging-kinesis.html

[mikereiddigital]

I've been through and identified the following resources that are relevant:

Flow Logs: (core-vpc-production) (excluding default)

(also ref https://github.com/ministryofjustice/modernisation-platform/blob/42fe9153176eaa9fec75485561a1604955712f63/source/runbooks/querying-vpc-flow-logs.html.md.erb#L3)

arn:aws:logs:eu-west-2:278663825216:log-group:hmpps-production-vpc-flow-logs-b0500bc4
arn:aws:logs:eu-west-2:278663825216:log-group:hmcts-production-vpc-flow-logs-a2064786
arn:aws:logs:eu-west-2:278663825216:log-group:hq-production-vpc-flow-logs-b4c5e7fc
arn:aws:logs:eu-west-2:278663825216:log-group:laa-production-vpc-flow-logs-e380d929
arn:aws:logs:eu-west-2:278663825216:log-group:platforms-production-vpc-flow-logs-817e4e1c
arn:aws:logs:eu-west-2:278663825216:log-group:cjse-production-vpc-flow-logs-99fb731b
arn:aws:logs:eu-west-2:278663825216:log-group:cica-production-vpc-flow-logs-ad30cffc
arn:aws:logs:eu-west-2:278663825216:log-group:opg-production-vpc-flow-logs-5fbb18c1


Networking Live Data - Flow Logs (core-network-services)

fl-04e17b9b7ee09e6c7 - live_data-vpc-flow-logs-qoa52v4z


Network Firewall Logs (core-network-services)

fw-live-data-inline-inspection-logs-ulpiqfaq
NOMS-Transit-Live-DR-VPN-VNG-1-vpn-attachment-logs
NOMS-Transit-Live-DR-VPN-VNG-2-vpn-attachment-logs	


Route 53 DNS Logs 

- core-vpc-production for intra-vpc requests only
- not logged in core-network-services

I'm out tomorrow morning so I'll talk to @dms1981 to confirm this scope is correct.

[mikereiddigital]

Spoke to Leo via slack. Short initial chat organised for tomorrow at 2.30pm

Teams call - https://teams.microsoft.com/l/meetup-join/19%3ameeting_MGY1NjgwYTgtY2E4OC00NTM0LTgyYzgtYjQ5YzFkZTdiZTU1%40thread.v2/0?context=%7b%22Tid%22%3a%22c6874728-71e6-41fe-a9e1-2e8c36776ad8%22%2c%22Oid%22%3a%226e4e862b-b9a9-4735-8386-a706da19fddf%22%7d

[mikereiddigital]

Content of email from Leo:

Hi Mike,
 
In preparation for the meeting we have today I suggest you start looking into the two integration technologies we will have to implement:
 
Route53 - Logs are essentially sent over to an S3 bucket that is then paired with XSIAM.
We need to work a bit together and we can do this activity in our XSIAM PreProduction/Lab instance. Specific details of the activity are here:
Ingest Network Route 53 Logs from Amazon S3 • Cortex XSIAM Administrator Guide • Reader • Palo Alto Networks documentation portal https://docs-cortex.paloaltonetworks.com/r/Cortex-XSIAM/Cortex-XSIAM-Administrator-Guide/Ingest-Network-Route-53-Logs-from-Amazon-S3
 
 
AWS with CloudWatch – Logs aggregated in CloudWatch are streamed to XSIAM using an Amazon Kinesis Firehose.
Step by step implementation is described at the link below and we are happy to work on this with you, also we are trying to collect some terraform code to share with you.
https://docs-cortex.paloaltonetworks.com/r/Cortex-XSIAM/Cortex-XSIAM-Administrator-Guide/Ingest-Logs-from-Amazon-CloudWatch
We will need 2 streams, one for the network data and the other for the firewalls.

Also:

• The way data will be parsed means that different delivery streams will be set up on their side for different categories of data.

[mikereiddigital]

Questions:

• Which account for vpc flow log data? core-vpc-production or core-logging?

[mikereiddigital]

Email from Leo as a follow up to the meeting:

Hi Mike,
 
It was a pleasure meeting you today.
 
We created the connectors you will use for XSIAM
 
Preproduction/Lab
name: Modernisation Platforms Network
api: https://api-justiceukpreprod.xdr.uk.paloaltonetworks.com/logs/v1/aws
secret_key: XSIAM-PP-MODP-NETWORK-CLOUD-WATCH-KEY
 
name: Modernisation Platforms Firewalls
api: https://api-justiceukpreprod.xdr.uk.paloaltonetworks.com/logs/v1/aws
secret_key: XSIAM-PP-MODP-FIREWALL-CLOUD-WATCH-KEY
 
 
Live
name: Modernisation Platforms Network
api: https://api-justiceuk.xdr.uk.paloaltonetworks.com/logs/v1/aws
secret_key: XSIAM-LIVE-MODP-NETWORK-CLOUD-WATCH-KEY
 
name: Modernisation Platforms Firewalls
api: https://api-justiceuk.xdr.uk.paloaltonetworks.com/logs/v1/aws
secret_key: XSIAM-LIVE-MODP-FIREWALL-CLOUD-WATCH-KEY
 
 
The secret keys are stored in our Azure Key Vault, if you could provide us with the list of users and emails that will need access we will grant permission and provide the final link to them.
 
Kind Regards,
Leo

[mikereiddigital]

PR with the initial set of changes for firehose - ministryofjustice/modernisation-platform-terraform-member-vpc#328

SecOps have granted us access to the security keys and added manually a secrets manager resource in core-vpc-development to hold the key for their preprod network endpoint.

[mikereiddigital]

Both https://github.com/ministryofjustice/modernisation-platform/tree/adds-firehose-resources and https://github.com/ministryofjustice/modernisation-platform-terraform-member-vpc/tree/add-aws-firehose are producing a clean plan. Next steps:

• Clean commits
• Set the scope to just one member vpc
• new PR, plan & approvals and apply as a test.
• Once confirmed working, new PR to include other vpcs and apply.
• Then move onto network firewalls logs.

[mikereiddigital]

Working build in core-vpc-sandbox - https://github.com/ministryofjustice/modernisation-platform-terraform-member-vpc/tree/add-aws-firehose

Next step to identify the scope of what's needed to apply to development.

[mikereiddigital]

Note:
│ Warning: Deprecated attribute
│
│ on .terraform/modules/vpc/firehose.tf line 190, in resource "aws_cloudwatch_log_subscription_filter" "nacs_server_xsiam_subscription":
│ 190: log_group_name = aws_flow_log.cloudwatch.log_group_name

This is from the plan & shows that property in question is deprecated however the terraform documentation shows otherwise - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_subscription_filter

[mikereiddigital]

Latest PR of the module - ministryofjustice/modernisation-platform-terraform-member-vpc#338. We still use the build_firehose variable as this determines which environ ments the resources are built in or not.

[mikereiddigital]

New release for the VPC module -https://github.com/ministryofjustice/modernisation-platform-terraform-member-vpc/releases/tag/v2.3.0

[mikereiddigital]

PR for the modernisation-platform repo changes - #6449

[mikereiddigital]

PR applied and resources built in dev & production. Will clear down sandbox before end of today.

[mikereiddigital]

Sandbox cleared.

[mikereiddigital]

PR for the 2nd part of the changes - #6485

[mikereiddigital]

PR for secret - #6505. This single secret will hold all of the endpoint and key details.

[mikereiddigital]

New commit that covers a just the firewall inspection logs & review comments.

[mikereiddigital]

Firewall inspection log data now being transferred. Tomorrow will be looking at the firewall vpc flow log transfers - which will not take long, and then we can move onto R53 data.

[mikereiddigital]

PR applied for the firewall vpc flow log data - #6545

[mikereiddigital]

Chat with Leo at 12.30 to go through the R53 records.

[mikereiddigital]

Additional VPC flow logs:

core-logging - 2 vpcs - see branch link below.
core-security - 2 vpcs (live & non-live
core-shared-services - 2 vpcs ""

For each of the above:

• 2 data items in secrets.tf for the secret reference
• 1 local item for the secrets
• new file - firehose.tf with the firehose module call & locals.

A single iteration of the module creates 19 resources.

https://github.com/ministryofjustice/modernisation-platform/tree/firehose-additional-vpcs

any others? Modernisation has just default vpc. Accounts that don't use shared vpcs such as bichard7?

[mikereiddigital]

Meeting with Leo:

• R53 can be sent via firehose. They will send us the new endpoints & secrets. Update - we have the new keys for the R53 transfers. The endpoints are the same as networks.
• Non-prod envs are not in scope currently.
• Need to confirm how & when we will get the VPC flow log data for the unrestricted member accounts.
• Cloudtrail logs is also a requirement which was recently added and needs to be discussed as part of a separate ticket?

[mikereiddigital]

Tidied up the added data & locals in core-vpc to reduce the number of items used.

[mikereiddigital]

We have added the transfer of R53 resolver logs & have confirmed they are transmitting. We need to trigger a small update to the firewall inspection & flow logs plus those for shared so the module changes are applied. Once that is done, we will create a new ticket for the cloudtrail logs as that is a new requirement that wasn't covered in the original scope.

[mikereiddigital]

Had a final chat with Leo before he goes on leave. Summarised where we are:

• Member shared VPC flow logs & r53 resolver logs (core-vpc)
• Firewall inspection logs & their VPC flow logs (core-network-services)
• Shared account VPC flow logs (core-logging, core-security, core-shared-services)

He knows that the cloudtrail events are being covered in a separate ticket. Also he mentioned to contact this team - monitoring-and-integration-platform@justice.gov.uk - if anything changes.

Leo is back on Wednesday next week.

[dms1981]

Cloudtrail integration discussed in #6614