Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DPR2-1643: Fix replay pipeline #9214

Merged
merged 2 commits into from
Jan 6, 2025

Conversation

koladeadewuyi-moj
Copy link
Contributor

@koladeadewuyi-moj koladeadewuyi-moj commented Jan 2, 2025

The replay pipeline

  1. Copies the batch and CDC files back to the raw zone
  2. Processes the batch files
  3. Processes the CDC files
  4. Checks that all CDC files have been processed

It fails on step 4 because the batch files are also present in the raw zone and are included in the check. To solve this,

  • Argument dpr.allowed.s3.file.extensions is replaced with dpr.allowed.s3.file.regex to allow a finer control of which files are included in the check
  • The value is set to \\d+-\\d+.parquet to match only the CDC files.

@koladeadewuyi-moj koladeadewuyi-moj requested review from a team as code owners January 2, 2025 15:47
@github-actions github-actions bot added the environments-repository Used to exclude PRs from this repo in our Slack PR update label Jan 2, 2025
@koladeadewuyi-moj koladeadewuyi-moj had a problem deploying to digital-prison-reporting-test January 2, 2025 15:49 — with GitHub Actions Error
@koladeadewuyi-moj koladeadewuyi-moj had a problem deploying to digital-prison-reporting-development January 2, 2025 15:49 — with GitHub Actions Error
Copy link
Contributor

github-actions bot commented Jan 2, 2025

Trivy Scan Success

Show Output ```hcl

Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline


Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
2025-01-02T15:49:22Z INFO [vulndb] Need to update DB
2025-01-02T15:49:22Z INFO [vulndb] Downloading vulnerability DB...
2025-01-02T15:49:22Z INFO [vulndb] Downloading artifact... repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-02T15:49:24Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-02T15:49:24Z INFO [vuln] Vulnerability scanning is enabled
2025-01-02T15:49:24Z INFO [misconfig] Misconfiguration scanning is enabled
2025-01-02T15:49:24Z INFO [misconfig] Need to update the built-in checks
2025-01-02T15:49:24Z INFO [misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [---------------------------------------------------------] 100.00% ? p/s 0s2025-01-02T15:49:24Z INFO [secret] Secret scanning is enabled
2025-01-02T15:49:24Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-02T15:49:24Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-02T15:49:25Z INFO [terraform scanner] Scanning root module file_path="."
2025-01-02T15:49:25Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="archive_job_trigger_name, data_ingestion_pipeline, dms_replication_task_arn, domain, glue_archive_job, glue_hive_table_creation_jobname, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_reporting_hub_batch_jobname, glue_reporting_hub_cdc_jobname, glue_s3_data_deletion_job, glue_s3_file_transfer_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, glue_stop_glue_instance_job, glue_switch_prisons_hive_data_location_job, glue_trigger_activation_job, pipeline_notification_lambda_function, replication_task_id, s3_curated_bucket_id, s3_curated_path, s3_glue_bucket_id, s3_raw_archive_bucket_id, s3_raw_bucket_id, s3_structured_bucket_id, s3_structured_path, s3_temp_reload_bucket_id, step_function_execution_role_arn, stop_dms_task_job"
2025-01-02T15:49:25Z INFO Number of language-specific files num=0
2025-01-02T15:49:25Z INFO Detected config files num=1
trivy_exitcode=0


Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2025-01-02T15:49:25Z INFO [vuln] Vulnerability scanning is enabled
2025-01-02T15:49:25Z INFO [misconfig] Misconfiguration scanning is enabled
2025-01-02T15:49:25Z INFO [secret] Secret scanning is enabled
2025-01-02T15:49:25Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-02T15:49:25Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-02T15:49:26Z INFO [terraform scanner] Scanning root module file_path="."
2025-01-02T15:49:26Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2025-01-02T15:49:26Z INFO Number of language-specific files num=0
2025-01-02T15:49:26Z INFO Detected config files num=1
trivy_exitcode=0

</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>

```hcl

*****************************

Checkov will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39

checkov_exitcode=0

*****************************

Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39

checkov_exitcode=0

CTFLint Scan Success

Show Output
*****************************

Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.9.1)
tflint will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running tflint in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

*****************************

Running tflint in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

Trivy Scan Success

Show Output
*****************************

Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
2025-01-02T15:49:22Z	INFO	[vulndb] Need to update DB
2025-01-02T15:49:22Z	INFO	[vulndb] Downloading vulnerability DB...
2025-01-02T15:49:22Z	INFO	[vulndb] Downloading artifact...	repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-02T15:49:24Z	INFO	[vulndb] Artifact successfully downloaded	repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-02T15:49:24Z	INFO	[vuln] Vulnerability scanning is enabled
2025-01-02T15:49:24Z	INFO	[misconfig] Misconfiguration scanning is enabled
2025-01-02T15:49:24Z	INFO	[misconfig] Need to update the built-in checks
2025-01-02T15:49:24Z	INFO	[misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [---------------------------------------------------------] 100.00% ? p/s 0s2025-01-02T15:49:24Z	INFO	[secret] Secret scanning is enabled
2025-01-02T15:49:24Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-02T15:49:24Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-02T15:49:25Z	INFO	[terraform scanner] Scanning root module	file_path="."
2025-01-02T15:49:25Z	WARN	[terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.	module="root" variables="archive_job_trigger_name, data_ingestion_pipeline, dms_replication_task_arn, domain, glue_archive_job, glue_hive_table_creation_jobname, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_reporting_hub_batch_jobname, glue_reporting_hub_cdc_jobname, glue_s3_data_deletion_job, glue_s3_file_transfer_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, glue_stop_glue_instance_job, glue_switch_prisons_hive_data_location_job, glue_trigger_activation_job, pipeline_notification_lambda_function, replication_task_id, s3_curated_bucket_id, s3_curated_path, s3_glue_bucket_id, s3_raw_archive_bucket_id, s3_raw_bucket_id, s3_structured_bucket_id, s3_structured_path, s3_temp_reload_bucket_id, step_function_execution_role_arn, stop_dms_task_job"
2025-01-02T15:49:25Z	INFO	Number of language-specific files	num=0
2025-01-02T15:49:25Z	INFO	Detected config files	num=1
trivy_exitcode=0

*****************************

Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2025-01-02T15:49:25Z	INFO	[vuln] Vulnerability scanning is enabled
2025-01-02T15:49:25Z	INFO	[misconfig] Misconfiguration scanning is enabled
2025-01-02T15:49:25Z	INFO	[secret] Secret scanning is enabled
2025-01-02T15:49:25Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-02T15:49:25Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-02T15:49:26Z	INFO	[terraform scanner] Scanning root module	file_path="."
2025-01-02T15:49:26Z	WARN	[terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.	module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2025-01-02T15:49:26Z	INFO	Number of language-specific files	num=0
2025-01-02T15:49:26Z	INFO	Detected config files	num=1
trivy_exitcode=0

@koladeadewuyi-moj koladeadewuyi-moj force-pushed the DPR2-1643_check-only-cdc-files-in-replay branch from b214fec to 239d943 Compare January 3, 2025 14:57
@koladeadewuyi-moj koladeadewuyi-moj had a problem deploying to digital-prison-reporting-development January 3, 2025 14:59 — with GitHub Actions Error
@koladeadewuyi-moj koladeadewuyi-moj had a problem deploying to digital-prison-reporting-test January 3, 2025 14:59 — with GitHub Actions Error
Copy link
Contributor

github-actions bot commented Jan 3, 2025

Trivy Scan Success

Show Output ```hcl

Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline


Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
2025-01-03T14:59:28Z INFO [vulndb] Need to update DB
2025-01-03T14:59:28Z INFO [vulndb] Downloading vulnerability DB...
2025-01-03T14:59:28Z INFO [vulndb] Downloading artifact... repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-03T14:59:31Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-03T14:59:31Z INFO [vuln] Vulnerability scanning is enabled
2025-01-03T14:59:31Z INFO [misconfig] Misconfiguration scanning is enabled
2025-01-03T14:59:31Z INFO [misconfig] Need to update the built-in checks
2025-01-03T14:59:31Z INFO [misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [---------------------------------------------------------] 100.00% ? p/s 0s2025-01-03T14:59:31Z INFO [secret] Secret scanning is enabled
2025-01-03T14:59:31Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-03T14:59:31Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-03T14:59:32Z INFO [terraform scanner] Scanning root module file_path="."
2025-01-03T14:59:32Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="archive_job_trigger_name, data_ingestion_pipeline, dms_replication_task_arn, domain, glue_archive_job, glue_hive_table_creation_jobname, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_reporting_hub_batch_jobname, glue_reporting_hub_cdc_jobname, glue_s3_data_deletion_job, glue_s3_file_transfer_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, glue_stop_glue_instance_job, glue_switch_prisons_hive_data_location_job, glue_trigger_activation_job, pipeline_notification_lambda_function, replication_task_id, s3_curated_bucket_id, s3_curated_path, s3_glue_bucket_id, s3_raw_archive_bucket_id, s3_raw_bucket_id, s3_structured_bucket_id, s3_structured_path, s3_temp_reload_bucket_id, step_function_execution_role_arn, stop_dms_task_job"
2025-01-03T14:59:32Z INFO Number of language-specific files num=0
2025-01-03T14:59:32Z INFO Detected config files num=1
trivy_exitcode=0


Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2025-01-03T14:59:32Z INFO [vuln] Vulnerability scanning is enabled
2025-01-03T14:59:32Z INFO [misconfig] Misconfiguration scanning is enabled
2025-01-03T14:59:32Z INFO [secret] Secret scanning is enabled
2025-01-03T14:59:32Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-03T14:59:32Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-03T14:59:33Z INFO [terraform scanner] Scanning root module file_path="."
2025-01-03T14:59:33Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2025-01-03T14:59:33Z INFO Number of language-specific files num=0
2025-01-03T14:59:33Z INFO Detected config files num=1
trivy_exitcode=0

</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>

```hcl

*****************************

Checkov will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39

checkov_exitcode=0

*****************************

Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39

checkov_exitcode=0

CTFLint Scan Success

Show Output
*****************************

Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.9.1)
tflint will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running tflint in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

*****************************

Running tflint in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

Trivy Scan Success

Show Output
*****************************

Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
2025-01-03T14:59:28Z	INFO	[vulndb] Need to update DB
2025-01-03T14:59:28Z	INFO	[vulndb] Downloading vulnerability DB...
2025-01-03T14:59:28Z	INFO	[vulndb] Downloading artifact...	repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-03T14:59:31Z	INFO	[vulndb] Artifact successfully downloaded	repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-03T14:59:31Z	INFO	[vuln] Vulnerability scanning is enabled
2025-01-03T14:59:31Z	INFO	[misconfig] Misconfiguration scanning is enabled
2025-01-03T14:59:31Z	INFO	[misconfig] Need to update the built-in checks
2025-01-03T14:59:31Z	INFO	[misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [---------------------------------------------------------] 100.00% ? p/s 0s2025-01-03T14:59:31Z	INFO	[secret] Secret scanning is enabled
2025-01-03T14:59:31Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-03T14:59:31Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-03T14:59:32Z	INFO	[terraform scanner] Scanning root module	file_path="."
2025-01-03T14:59:32Z	WARN	[terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.	module="root" variables="archive_job_trigger_name, data_ingestion_pipeline, dms_replication_task_arn, domain, glue_archive_job, glue_hive_table_creation_jobname, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_reporting_hub_batch_jobname, glue_reporting_hub_cdc_jobname, glue_s3_data_deletion_job, glue_s3_file_transfer_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, glue_stop_glue_instance_job, glue_switch_prisons_hive_data_location_job, glue_trigger_activation_job, pipeline_notification_lambda_function, replication_task_id, s3_curated_bucket_id, s3_curated_path, s3_glue_bucket_id, s3_raw_archive_bucket_id, s3_raw_bucket_id, s3_structured_bucket_id, s3_structured_path, s3_temp_reload_bucket_id, step_function_execution_role_arn, stop_dms_task_job"
2025-01-03T14:59:32Z	INFO	Number of language-specific files	num=0
2025-01-03T14:59:32Z	INFO	Detected config files	num=1
trivy_exitcode=0

*****************************

Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2025-01-03T14:59:32Z	INFO	[vuln] Vulnerability scanning is enabled
2025-01-03T14:59:32Z	INFO	[misconfig] Misconfiguration scanning is enabled
2025-01-03T14:59:32Z	INFO	[secret] Secret scanning is enabled
2025-01-03T14:59:32Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-03T14:59:32Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-03T14:59:33Z	INFO	[terraform scanner] Scanning root module	file_path="."
2025-01-03T14:59:33Z	WARN	[terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.	module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2025-01-03T14:59:33Z	INFO	Number of language-specific files	num=0
2025-01-03T14:59:33Z	INFO	Detected config files	num=1
trivy_exitcode=0

Copy link
Contributor

github-actions bot commented Jan 6, 2025

Trivy Scan Success

Show Output ```hcl

Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline


Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
2025-01-06T09:32:04Z INFO [vulndb] Need to update DB
2025-01-06T09:32:04Z INFO [vulndb] Downloading vulnerability DB...
2025-01-06T09:32:04Z INFO [vulndb] Downloading artifact... repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-06T09:32:06Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-06T09:32:06Z INFO [vuln] Vulnerability scanning is enabled
2025-01-06T09:32:06Z INFO [misconfig] Misconfiguration scanning is enabled
2025-01-06T09:32:06Z INFO [misconfig] Need to update the built-in checks
2025-01-06T09:32:06Z INFO [misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [---------------------------------------------------------] 100.00% ? p/s 0s2025-01-06T09:32:07Z INFO [secret] Secret scanning is enabled
2025-01-06T09:32:07Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-06T09:32:07Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-06T09:32:08Z INFO [terraform scanner] Scanning root module file_path="."
2025-01-06T09:32:08Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="archive_job_trigger_name, data_ingestion_pipeline, dms_replication_task_arn, domain, glue_archive_job, glue_hive_table_creation_jobname, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_reporting_hub_batch_jobname, glue_reporting_hub_cdc_jobname, glue_s3_data_deletion_job, glue_s3_file_transfer_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, glue_stop_glue_instance_job, glue_switch_prisons_hive_data_location_job, glue_trigger_activation_job, pipeline_notification_lambda_function, replication_task_id, s3_curated_bucket_id, s3_curated_path, s3_glue_bucket_id, s3_raw_archive_bucket_id, s3_raw_bucket_id, s3_structured_bucket_id, s3_structured_path, s3_temp_reload_bucket_id, step_function_execution_role_arn, stop_dms_task_job"
2025-01-06T09:32:08Z INFO Number of language-specific files num=0
2025-01-06T09:32:08Z INFO Detected config files num=1
trivy_exitcode=0


Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2025-01-06T09:32:08Z INFO [vuln] Vulnerability scanning is enabled
2025-01-06T09:32:08Z INFO [misconfig] Misconfiguration scanning is enabled
2025-01-06T09:32:08Z INFO [secret] Secret scanning is enabled
2025-01-06T09:32:08Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-06T09:32:08Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-06T09:32:09Z INFO [terraform scanner] Scanning root module file_path="."
2025-01-06T09:32:09Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2025-01-06T09:32:09Z INFO Number of language-specific files num=0
2025-01-06T09:32:09Z INFO Detected config files num=1
trivy_exitcode=0

</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>

```hcl

*****************************

Checkov will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39

checkov_exitcode=0

*****************************

Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39

checkov_exitcode=0

CTFLint Scan Success

Show Output
*****************************

Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.9.1)
tflint will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running tflint in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

*****************************

Running tflint in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

Trivy Scan Success

Show Output
*****************************

Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline

*****************************

Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/ingestion-pipeline
2025-01-06T09:32:04Z	INFO	[vulndb] Need to update DB
2025-01-06T09:32:04Z	INFO	[vulndb] Downloading vulnerability DB...
2025-01-06T09:32:04Z	INFO	[vulndb] Downloading artifact...	repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-06T09:32:06Z	INFO	[vulndb] Artifact successfully downloaded	repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-06T09:32:06Z	INFO	[vuln] Vulnerability scanning is enabled
2025-01-06T09:32:06Z	INFO	[misconfig] Misconfiguration scanning is enabled
2025-01-06T09:32:06Z	INFO	[misconfig] Need to update the built-in checks
2025-01-06T09:32:06Z	INFO	[misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [---------------------------------------------------------] 100.00% ? p/s 0s2025-01-06T09:32:07Z	INFO	[secret] Secret scanning is enabled
2025-01-06T09:32:07Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-06T09:32:07Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-06T09:32:08Z	INFO	[terraform scanner] Scanning root module	file_path="."
2025-01-06T09:32:08Z	WARN	[terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.	module="root" variables="archive_job_trigger_name, data_ingestion_pipeline, dms_replication_task_arn, domain, glue_archive_job, glue_hive_table_creation_jobname, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_reporting_hub_batch_jobname, glue_reporting_hub_cdc_jobname, glue_s3_data_deletion_job, glue_s3_file_transfer_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, glue_stop_glue_instance_job, glue_switch_prisons_hive_data_location_job, glue_trigger_activation_job, pipeline_notification_lambda_function, replication_task_id, s3_curated_bucket_id, s3_curated_path, s3_glue_bucket_id, s3_raw_archive_bucket_id, s3_raw_bucket_id, s3_structured_bucket_id, s3_structured_path, s3_temp_reload_bucket_id, step_function_execution_role_arn, stop_dms_task_job"
2025-01-06T09:32:08Z	INFO	Number of language-specific files	num=0
2025-01-06T09:32:08Z	INFO	Detected config files	num=1
trivy_exitcode=0

*****************************

Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2025-01-06T09:32:08Z	INFO	[vuln] Vulnerability scanning is enabled
2025-01-06T09:32:08Z	INFO	[misconfig] Misconfiguration scanning is enabled
2025-01-06T09:32:08Z	INFO	[secret] Secret scanning is enabled
2025-01-06T09:32:08Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-06T09:32:08Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-06T09:32:09Z	INFO	[terraform scanner] Scanning root module	file_path="."
2025-01-06T09:32:09Z	WARN	[terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.	module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2025-01-06T09:32:09Z	INFO	Number of language-specific files	num=0
2025-01-06T09:32:09Z	INFO	Detected config files	num=1
trivy_exitcode=0

@koladeadewuyi-moj koladeadewuyi-moj merged commit 470ac0b into main Jan 6, 2025
10 of 13 checks passed
@koladeadewuyi-moj koladeadewuyi-moj deleted the DPR2-1643_check-only-cdc-files-in-replay branch January 6, 2025 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
environments-repository Used to exclude PRs from this repo in our Slack PR update
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants