POC Datasync #8357

ep-93 · 2024-10-28T13:17:12Z

A reference to the issue / Description of it

#7725

How does this PR fix the problem?

Makes Cooker the main account, with sprinkler data syncing to it.

This PR create an S3 bucket to store the inventory data.
Create IAM roles and policies to allow cross-account access.
Configure Resource Data Sync to aggregate inventory data into the S3 bucket.
Use Athena to query the inventory data stored in S3.

How has this been tested?

Yes, and shown to Simon.

Deployment Plan / Instructions

Not to be deployed, but as a POC ready for the rolling out of this.

github-actions · 2024-10-28T13:19:43Z

`Trivy Scan` Failed

Show Output

```hcl

Trivy will check the following folders:
terraform/environments/cooker
terraform/environments/sprinkler

Running Trivy in terraform/environments/cooker
2024-10-28T13:19:24Z INFO [vulndb] Need to update DB
2024-10-28T13:19:24Z INFO [vulndb] Downloading vulnerability DB...
2024-10-28T13:19:24Z INFO [vulndb] Downloading artifact... repo="public.ecr.aws/aquasecurity/trivy-db:2"
2024-10-28T13:19:26Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"
2024-10-28T13:19:26Z INFO [vuln] Vulnerability scanning is enabled
2024-10-28T13:19:26Z INFO [misconfig] Misconfiguration scanning is enabled
2024-10-28T13:19:26Z INFO [misconfig] Need to update the built-in checks
2024-10-28T13:19:26Z INFO [misconfig] Downloading the built-in checks...
156.02 KiB / 156.02 KiB [------------------------------------------------------] 100.00% ? p/s 100ms2024-10-28T13:19:27Z INFO [secret] Secret scanning is enabled
2024-10-28T13:19:27Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-10-28T13:19:27Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.56/docs/scanner/secret#recommendation for faster secret detection
2024-10-28T13:19:28Z INFO [terraform scanner] Scanning root module file_path="."
2024-10-28T13:19:28Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="networking"
2024-10-28T13:19:29Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.aws_s3_object.user_public_keys" value="cty.NilVal"
2024-10-28T13:19:29Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.data.aws_subnet.local_account" value="cty.NilVal"
2024-10-28T13:19:29Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.dynamic.tag" value="cty.NilVal"
2024-10-28T13:19:29Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.dynamic.tag" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.expiration" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.transition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.noncurrent_version_expiration" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.noncurrent_version_transition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.principals" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.condition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.expiration" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.transition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.noncurrent_version_expiration" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.noncurrent_version_transition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.principals" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.bastion_linux.module.s3-bucket.dynamic.condition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.expiration" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.transition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.noncurrent_version_expiration" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.noncurrent_version_transition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.principals" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.condition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.expiration" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.transition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.noncurrent_version_expiration" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.noncurrent_version_transition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.principals" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.s3-bucket.dynamic.condition" value="cty.NilVal"
2024-10-28T13:19:30Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.ram-ec2-retagging[0].data.aws_subnet.host" value="cty.NilVal"
2024-10-28T13:19:31Z INFO [terraform executor] Ignore finding rule="aws-s3-encryption-customer-key" range="github.com/ministryofjustice/modernisation-platform-terraform-bastion-linux?ref=95ed3c3f454e2014a62990aacd5d68c64d026f11/github.com/ministryofjustice/modernisation-platform-terraform-s3-bucket?ref=568694e50e03630d99cb569eafa06a0b879a1239/main.tf:171-179"
2024-10-28T13:19:31Z INFO [terraform executor] Ignore finding rule="aws-s3-encryption-customer-key" range="github.com/ministryofjustice/modernisation-platform-terraform-s3-bucket?ref=v8.2.0/main.tf:187-197"
2024-10-28T13:19:31Z INFO Number of language-specific files num=0
2024-10-28T13:19:31Z INFO Detected config files num=4

datasync.tf (terraform)

Tests: 8 (SUCCESSES: 0, FAILURES: 7, EXCEPTIONS: 1)
Failures: 7 (HIGH: 7, CRITICAL: 0)

HIGH: Database does not have encryption configured.
════════════════════════════════════════
Data can be read if the Athena Database is compromised. Athena databases and workspace result sets should be encrypted at rests. These databases and query sets are generally derived from data in S3 buckets and should have the same level of at rest protection.

See https://avd.aquasec.com/misconfig/avd-aws-0006
────────────────────────────────────────
datasync.tf:121-124
────────────────────────────────────────
121 ┌ resource "aws_athena_database" "ssm_inventory_db" {
122 │ name = "ssm_inventory_db"
123 │ bucket = "ssm-inventory-sync-bucket-236861075084-euw2"
124 └ }
────────────────────────────────────────

HIGH: Workgroup does not have encryption configured.
════════════════════════════════════════
Data can be read if the Athena Database is compromised. Athena databases and workspace result sets should be encrypted at rests. These databases and query sets are generally derived from data in S3 buckets and should have the same level of at rest protection.

See https://avd.aquasec.com/misconfig/avd-aws-0006
────────────────────────────────────────
datasync.tf:126-138
────────────────────────────────────────
126 ┌ resource "aws_athena_workgroup" "ssm_inventory_workgroup" {
127 │ name = "ssm_inventory_workgroup"
128 │ description = "Workgroup for querying SSM Inventory data"
129 │ state = "ENABLED"
130 │
131 │ configuration {
132 │ enforce_workgroup_configuration = true
133 │
134 └ result_configuration {
...
────────────────────────────────────────

HIGH: No public access block so not blocking public acls
════════════════════════════════════════
S3 buckets should block public ACLs on buckets and any objects they contain. By blocking, PUTs with fail if the object has any public ACL a.

See https://avd.aquasec.com/misconfig/avd-aws-0086
────────────────────────────────────────
datasync.tf:4-6
────────────────────────────────────────
4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
5 │ bucket = "ssm-inventory-sync-bucket-euw2"
6 └ }
────────────────────────────────────────

HIGH: No public access block so not blocking public policies
════════════════════════════════════════
S3 bucket policy should have block public policy to prevent users from putting a policy that enable public access.

See https://avd.aquasec.com/misconfig/avd-aws-0087
────────────────────────────────────────
datasync.tf:4-6
────────────────────────────────────────
4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
5 │ bucket = "ssm-inventory-sync-bucket-euw2"
6 └ }
────────────────────────────────────────

HIGH: Bucket does not have encryption enabled
════════════════════════════════════════
S3 Buckets should be encrypted to protect the data that is stored within them if access is compromised.

See https://avd.aquasec.com/misconfig/avd-aws-0088
────────────────────────────────────────
datasync.tf:4-6
────────────────────────────────────────
4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
5 │ bucket = "ssm-inventory-sync-bucket-euw2"
6 └ }
────────────────────────────────────────

HIGH: No public access block so not blocking public acls
════════════════════════════════════════
S3 buckets should ignore public ACLs on buckets and any objects they contain. By ignoring rather than blocking, PUT calls with public ACLs will still be applied but the ACL will be ignored.

See https://avd.aquasec.com/misconfig/avd-aws-0091
────────────────────────────────────────
datasync.tf:4-6
────────────────────────────────────────
4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
5 │ bucket = "ssm-inventory-sync-bucket-euw2"
6 └ }
────────────────────────────────────────

HIGH: No public access block so not restricting public buckets
════════════════════════════════════════
S3 buckets should restrict public policies for the bucket. By enabling, the restrict_public_buckets, only the bucket owner and AWS Services can access if it has a public policy.

See https://avd.aquasec.com/misconfig/avd-aws-0093
────────────────────────────────────────
datasync.tf:4-6
────────────────────────────────────────
4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
5 │ bucket = "ssm-inventory-sync-bucket-euw2"
6 └ }
────────────────────────────────────────

trivy_exitcode=1

Running Trivy in terraform/environments/sprinkler
2024-10-28T13:19:31Z INFO [vuln] Vulnerability scanning is enabled
2024-10-28T13:19:31Z INFO [misconfig] Misconfiguration scanning is enabled
2024-10-28T13:19:31Z INFO [secret] Secret scanning is enabled
2024-10-28T13:19:31Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-10-28T13:19:31Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.56/docs/scanner/secret#recommendation for faster secret detection
2024-10-28T13:19:32Z INFO [terraform scanner] Scanning root module file_path="."
2024-10-28T13:19:32Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="networking"
2024-10-28T13:19:32Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.ram-ec2-retagging[0].data.aws_subnet.host" value="cty.NilVal"
2024-10-28T13:19:32Z INFO Number of language-specific files num=0
2024-10-28T13:19:32Z INFO Detected config files num=1
trivy_exitcode=1

</details> #### `Checkov Scan` Failed
<details><summary>Show Output</summary>

```hcl

*****************************

Checkov will check the following folders:
terraform/environments/cooker
terraform/environments/sprinkler

*****************************

Running Checkov in terraform/environments/cooker
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
2024-10-28 13:19:35,419 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-bastion-linux?ref=95ed3c3f454e2014a62990aacd5d68c64d026f11:None (for external modules, the --download-external-modules flag is required)
2024-10-28 13:19:35,419 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-s3-bucket?ref=v8.2.0:None (for external modules, the --download-external-modules flag is required)
2024-10-28 13:19:35,443 [MainThread  ] [WARNI]  [ArmLocalGraph] created 0 vertices
2024-10-28 13:19:35,443 [MainThread  ] [WARNI]  [ArmLocalGraph] created 0 edges
terraform scan results:

Passed checks: 43, Failed checks: 10, Skipped checks: 1

Check: CKV_AWS_77: "Ensure Athena Database is encrypted at rest (default is unencrypted)"
	FAILED for resource: aws_athena_database.ssm_inventory_db
	File: /datasync.tf:121-124
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-general-policies/bc-aws-general-29

		121 | resource "aws_athena_database" "ssm_inventory_db" {
		122 |   name   = "ssm_inventory_db"
		123 |   bucket = "ssm-inventory-sync-bucket-236861075084-euw2"
		124 | }

Check: CKV_AWS_159: "Ensure that Athena Workgroup is encrypted"
	FAILED for resource: aws_athena_workgroup.ssm_inventory_workgroup
	File: /datasync.tf:126-138
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-general-policies/ensure-that-athena-workgroup-is-encrypted

		126 | resource "aws_athena_workgroup" "ssm_inventory_workgroup" {
		127 |   name        = "ssm_inventory_workgroup"
		128 |   description = "Workgroup for querying SSM Inventory data"
		129 |   state       = "ENABLED"
		130 | 
		131 |   configuration {
		132 |     enforce_workgroup_configuration = true
		133 | 
		134 |     result_configuration {
		135 |       output_location = "s3://ssm-inventory-sync-bucket-euw2/athena_results/"
		136 |     }
		137 |   }
		138 | }

Check: CKV_AWS_355: "Ensure no IAM policies documents allow "*" as a statement's resource for restrictable actions"
	FAILED for resource: aws_iam_policy.quicksight_access
	File: /datasync.tf:161-183
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-iam-policies/bc-aws-355

		161 | resource "aws_iam_policy" "quicksight_access" {
		162 |   name        = "QuickSightAthenaS3Access"
		163 |   description = "Access for QuickSight to Athena and S3"
		164 | 
		165 |   policy = jsonencode({
		166 |     Version = "2012-10-17",
		167 |     Statement = [
		168 |       {
		169 |         Effect   = "Allow",
		170 |         Action   = ["athena:StartQueryExecution", "athena:GetQueryResults", "athena:GetQueryExecution"],
		171 |         Resource = "*"
		172 |       },
		173 |       {
		174 |         Effect   = "Allow",
		175 |         Action   = ["s3:GetObject", "s3:ListBucket"],
		176 |         Resource = [
		177 |           "arn:aws:s3:::ssm-inventory-sync-bucket-euw2",
		178 |           "arn:aws:s3:::ssm-inventory-sync-bucket-euw2/*"
		179 |         ]
		180 |       }
		181 |     ]
		182 |   })
		183 | }

Check: CKV_AWS_290: "Ensure IAM policies does not allow write access without constraints"
	FAILED for resource: aws_iam_policy.quicksight_access
	File: /datasync.tf:161-183
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-iam-policies/bc-aws-290

		161 | resource "aws_iam_policy" "quicksight_access" {
		162 |   name        = "QuickSightAthenaS3Access"
		163 |   description = "Access for QuickSight to Athena and S3"
		164 | 
		165 |   policy = jsonencode({
		166 |     Version = "2012-10-17",
		167 |     Statement = [
		168 |       {
		169 |         Effect   = "Allow",
		170 |         Action   = ["athena:StartQueryExecution", "athena:GetQueryResults", "athena:GetQueryExecution"],
		171 |         Resource = "*"
		172 |       },
		173 |       {
		174 |         Effect   = "Allow",
		175 |         Action   = ["s3:GetObject", "s3:ListBucket"],
		176 |         Resource = [
		177 |           "arn:aws:s3:::ssm-inventory-sync-bucket-euw2",
		178 |           "arn:aws:s3:::ssm-inventory-sync-bucket-euw2/*"
		179 |         ]
		180 |       }
		181 |     ]
		182 |   })
		183 | }

Check: CKV2_AWS_62: "Ensure S3 buckets should have event notifications enabled"
	FAILED for resource: aws_s3_bucket.ssm_inventory_bucket
	File: /datasync.tf:4-6
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-logging-policies/bc-aws-2-62

		4 | resource "aws_s3_bucket" "ssm_inventory_bucket" {
		5 |   bucket = "ssm-inventory-sync-bucket-euw2"
		6 | }

Check: CKV_AWS_18: "Ensure the S3 bucket has access logging enabled"
	FAILED for resource: aws_s3_bucket.ssm_inventory_bucket
	File: /datasync.tf:4-6
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/s3-policies/s3-13-enable-logging

		4 | resource "aws_s3_bucket" "ssm_inventory_bucket" {
		5 |   bucket = "ssm-inventory-sync-bucket-euw2"
		6 | }

Check: CKV2_AWS_6: "Ensure that S3 bucket has a Public Access block"
	FAILED for resource: aws_s3_bucket.ssm_inventory_bucket
	File: /datasync.tf:4-6
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-networking-policies/s3-bucket-should-have-public-access-blocks-defaults-to-false-if-the-public-access-block-is-not-attached

		4 | resource "aws_s3_bucket" "ssm_inventory_bucket" {
		5 |   bucket = "ssm-inventory-sync-bucket-euw2"
		6 | }

Check: CKV_AWS_144: "Ensure that S3 bucket has cross-region replication enabled"
	FAILED for resource: aws_s3_bucket.ssm_inventory_bucket
	File: /datasync.tf:4-6
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-general-policies/ensure-that-s3-bucket-has-cross-region-replication-enabled

		4 | resource "aws_s3_bucket" "ssm_inventory_bucket" {
		5 |   bucket = "ssm-inventory-sync-bucket-euw2"
		6 | }

Check: CKV_AWS_145: "Ensure that S3 buckets are encrypted with KMS by default"
	FAILED for resource: aws_s3_bucket.ssm_inventory_bucket
	File: /datasync.tf:4-6
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-general-policies/ensure-that-s3-buckets-are-encrypted-with-kms-by-default

		4 | resource "aws_s3_bucket" "ssm_inventory_bucket" {
		5 |   bucket = "ssm-inventory-sync-bucket-euw2"
		6 | }

Check: CKV2_AWS_61: "Ensure that an S3 bucket has a lifecycle configuration"
	FAILED for resource: aws_s3_bucket.ssm_inventory_bucket
	File: /datasync.tf:4-6
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-logging-policies/bc-aws-2-61

		4 | resource "aws_s3_bucket" "ssm_inventory_bucket" {
		5 |   bucket = "ssm-inventory-sync-bucket-euw2"
		6 | }


checkov_exitcode=1

*****************************

Running Checkov in terraform/environments/sprinkler
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
2024-10-28 13:19:39,293 [MainThread  ] [WARNI]  [ArmLocalGraph] created 0 vertices
2024-10-28 13:19:39,293 [MainThread  ] [WARNI]  [ArmLocalGraph] created 0 edges
terraform scan results:

Passed checks: 19, Failed checks: 0, Skipped checks: 0


checkov_exitcode=1

`CTFLint Scan` Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.9.1)
tflint will check the following folders:
terraform/environments/cooker
terraform/environments/sprinkler

*****************************

Running tflint in terraform/environments/cooker
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

*****************************

Running tflint in terraform/environments/sprinkler
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

`Trivy Scan` Failed

Show Output

*****************************

Trivy will check the following folders:
terraform/environments/cooker
terraform/environments/sprinkler

*****************************

Running Trivy in terraform/environments/cooker
2024-10-28T13:19:24Z	INFO	[vulndb] Need to update DB
2024-10-28T13:19:24Z	INFO	[vulndb] Downloading vulnerability DB...
2024-10-28T13:19:24Z	INFO	[vulndb] Downloading artifact...	repo="public.ecr.aws/aquasecurity/trivy-db:2"
2024-10-28T13:19:26Z	INFO	[vulndb] Artifact successfully downloaded	repo="public.ecr.aws/aquasecurity/trivy-db:2"
2024-10-28T13:19:26Z	INFO	[vuln] Vulnerability scanning is enabled
2024-10-28T13:19:26Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-10-28T13:19:26Z	INFO	[misconfig] Need to update the built-in checks
2024-10-28T13:19:26Z	INFO	[misconfig] Downloading the built-in checks...
156.02 KiB / 156.02 KiB [------------------------------------------------------] 100.00% ? p/s 100ms2024-10-28T13:19:27Z	INFO	[secret] Secret scanning is enabled
2024-10-28T13:19:27Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-10-28T13:19:27Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.56/docs/scanner/secret#recommendation for faster secret detection
2024-10-28T13:19:28Z	INFO	[terraform scanner] Scanning root module	file_path="."
2024-10-28T13:19:28Z	WARN	[terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.	module="root" variables="networking"
2024-10-28T13:19:29Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.aws_s3_object.user_public_keys" value="cty.NilVal"
2024-10-28T13:19:29Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.data.aws_subnet.local_account" value="cty.NilVal"
2024-10-28T13:19:29Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.dynamic.tag" value="cty.NilVal"
2024-10-28T13:19:29Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.dynamic.tag" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.expiration" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.transition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.noncurrent_version_expiration" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.noncurrent_version_transition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.principals" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.condition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.expiration" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.transition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.noncurrent_version_expiration" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.noncurrent_version_transition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.principals" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.bastion_linux.module.s3-bucket.dynamic.condition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.expiration" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.transition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.noncurrent_version_expiration" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.noncurrent_version_transition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.principals" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.condition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.expiration" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.transition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.noncurrent_version_expiration" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.noncurrent_version_transition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.principals" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.s3-bucket.dynamic.condition" value="cty.NilVal"
2024-10-28T13:19:30Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.ram-ec2-retagging[0].data.aws_subnet.host" value="cty.NilVal"
2024-10-28T13:19:31Z	INFO	[terraform executor] Ignore finding	rule="aws-s3-encryption-customer-key" range="github.com/ministryofjustice/modernisation-platform-terraform-bastion-linux?ref=95ed3c3f454e2014a62990aacd5d68c64d026f11/github.com/ministryofjustice/modernisation-platform-terraform-s3-bucket?ref=568694e50e03630d99cb569eafa06a0b879a1239/main.tf:171-179"
2024-10-28T13:19:31Z	INFO	[terraform executor] Ignore finding	rule="aws-s3-encryption-customer-key" range="github.com/ministryofjustice/modernisation-platform-terraform-s3-bucket?ref=v8.2.0/main.tf:187-197"
2024-10-28T13:19:31Z	INFO	Number of language-specific files	num=0
2024-10-28T13:19:31Z	INFO	Detected config files	num=4

datasync.tf (terraform)
=======================
Tests: 8 (SUCCESSES: 0, FAILURES: 7, EXCEPTIONS: 1)
Failures: 7 (HIGH: 7, CRITICAL: 0)

HIGH: Database does not have encryption configured.
════════════════════════════════════════
Data can be read if the Athena Database is compromised. Athena databases and workspace result sets should be encrypted at rests. These databases and query sets are generally derived from data in S3 buckets and should have the same level of at rest protection.


See https://avd.aquasec.com/misconfig/avd-aws-0006
────────────────────────────────────────
 datasync.tf:121-124
────────────────────────────────────────
 121 ┌ resource "aws_athena_database" "ssm_inventory_db" {
 122 │   name   = "ssm_inventory_db"
 123 │   bucket = "ssm-inventory-sync-bucket-236861075084-euw2"
 124 └ }
────────────────────────────────────────


HIGH: Workgroup does not have encryption configured.
════════════════════════════════════════
Data can be read if the Athena Database is compromised. Athena databases and workspace result sets should be encrypted at rests. These databases and query sets are generally derived from data in S3 buckets and should have the same level of at rest protection.


See https://avd.aquasec.com/misconfig/avd-aws-0006
────────────────────────────────────────
 datasync.tf:126-138
────────────────────────────────────────
 126 ┌ resource "aws_athena_workgroup" "ssm_inventory_workgroup" {
 127 │   name        = "ssm_inventory_workgroup"
 128 │   description = "Workgroup for querying SSM Inventory data"
 129 │   state       = "ENABLED"
 130 │ 
 131 │   configuration {
 132 │     enforce_workgroup_configuration = true
 133 │ 
 134 └     result_configuration {
 ...   
────────────────────────────────────────


HIGH: No public access block so not blocking public acls
════════════════════════════════════════
S3 buckets should block public ACLs on buckets and any objects they contain. By blocking, PUTs with fail if the object has any public ACL a.


See https://avd.aquasec.com/misconfig/avd-aws-0086
────────────────────────────────────────
 datasync.tf:4-6
────────────────────────────────────────
   4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
   5 │   bucket = "ssm-inventory-sync-bucket-euw2"
   6 └ }
────────────────────────────────────────


HIGH: No public access block so not blocking public policies
════════════════════════════════════════
S3 bucket policy should have block public policy to prevent users from putting a policy that enable public access.


See https://avd.aquasec.com/misconfig/avd-aws-0087
────────────────────────────────────────
 datasync.tf:4-6
────────────────────────────────────────
   4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
   5 │   bucket = "ssm-inventory-sync-bucket-euw2"
   6 └ }
────────────────────────────────────────


HIGH: Bucket does not have encryption enabled
════════════════════════════════════════
S3 Buckets should be encrypted to protect the data that is stored within them if access is compromised.


See https://avd.aquasec.com/misconfig/avd-aws-0088
────────────────────────────────────────
 datasync.tf:4-6
────────────────────────────────────────
   4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
   5 │   bucket = "ssm-inventory-sync-bucket-euw2"
   6 └ }
────────────────────────────────────────


HIGH: No public access block so not blocking public acls
════════════════════════════════════════
S3 buckets should ignore public ACLs on buckets and any objects they contain. By ignoring rather than blocking, PUT calls with public ACLs will still be applied but the ACL will be ignored.


See https://avd.aquasec.com/misconfig/avd-aws-0091
────────────────────────────────────────
 datasync.tf:4-6
────────────────────────────────────────
   4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
   5 │   bucket = "ssm-inventory-sync-bucket-euw2"
   6 └ }
────────────────────────────────────────


HIGH: No public access block so not restricting public buckets
════════════════════════════════════════
S3 buckets should restrict public policies for the bucket. By enabling, the restrict_public_buckets, only the bucket owner and AWS Services can access if it has a public policy.


See https://avd.aquasec.com/misconfig/avd-aws-0093
────────────────────────────────────────
 datasync.tf:4-6
────────────────────────────────────────
   4 ┌ resource "aws_s3_bucket" "ssm_inventory_bucket" {
   5 │   bucket = "ssm-inventory-sync-bucket-euw2"
   6 └ }
────────────────────────────────────────


trivy_exitcode=1

*****************************

Running Trivy in terraform/environments/sprinkler
2024-10-28T13:19:31Z	INFO	[vuln] Vulnerability scanning is enabled
2024-10-28T13:19:31Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-10-28T13:19:31Z	INFO	[secret] Secret scanning is enabled
2024-10-28T13:19:31Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-10-28T13:19:31Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.56/docs/scanner/secret#recommendation for faster secret detection
2024-10-28T13:19:32Z	INFO	[terraform scanner] Scanning root module	file_path="."
2024-10-28T13:19:32Z	WARN	[terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.	module="root" variables="networking"
2024-10-28T13:19:32Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.ram-ec2-retagging[0].data.aws_subnet.host" value="cty.NilVal"
2024-10-28T13:19:32Z	INFO	Number of language-specific files	num=0
2024-10-28T13:19:32Z	INFO	Detected config files	num=1
trivy_exitcode=1

ep-93 added 2 commits October 23, 2024 14:57

Working syncing bucket and systems manager

a12503a

Remove acc numbers

21949c4

ep-93 changed the title ~~Poc/smi datasync~~ POC Datasync Oct 28, 2024

ep-93 mentioned this pull request Oct 28, 2024

PoC: Use AWS Systems Manager Inventory with Resource Data Sync #7725

Closed

7 tasks

ep-93 closed this Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

POC Datasync #8357

POC Datasync #8357

ep-93 commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024

datasync.tf (terraform)

POC Datasync #8357

POC Datasync #8357

Conversation

ep-93 commented Oct 28, 2024 • edited Loading

A reference to the issue / Description of it

How does this PR fix the problem?

How has this been tested?

Deployment Plan / Instructions

github-actions bot commented Oct 28, 2024

Trivy Scan Failed

datasync.tf (terraform)

CTFLint Scan Success

Trivy Scan Failed

ep-93 commented Oct 28, 2024 •

edited

Loading

`Trivy Scan` Failed

`CTFLint Scan` Success

`Trivy Scan` Failed