You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The batch job of the replay pipeline does not process diff files created by the reload pipeline. This is because the batch job looks for files matching the name pattern LOAD*.parquet whereas the reload diff files have the name pattern part-*.snappy.parquet.
Steps to reproduce:
Run the ingestion pipeline if there is no already ingested data
Run the reload pipeline. This generates the reload diff files
Run the replay pipeline. The batch job would fail to read the reload diff files
To fix this, an extra argument will be supplied to the batch job invocation during the replay pipeline: --dpr.batch.load.fileglobpattern : {part-*.snappy.parquet,LOAD*parquet}
Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2024-12-30T17:55:37Z INFO [vulndb] Need to update DB
2024-12-30T17:55:37Z INFO [vulndb] Downloading vulnerability DB...
2024-12-30T17:55:37Z INFO [vulndb] Downloading artifact... repo="public.ecr.aws/aquasecurity/trivy-db:2"
2024-12-30T17:55:40Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"
2024-12-30T17:55:40Z INFO [vuln] Vulnerability scanning is enabled
2024-12-30T17:55:40Z INFO [misconfig] Misconfiguration scanning is enabled
2024-12-30T17:55:40Z INFO [misconfig] Need to update the built-in checks
2024-12-30T17:55:40Z INFO [misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [------------------------------------------------------] 100.00% ? p/s 100ms2024-12-30T17:55:40Z INFO [secret] Secret scanning is enabled
2024-12-30T17:55:40Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-12-30T17:55:40Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2024-12-30T17:55:41Z INFO [terraform scanner] Scanning root module file_path="."
2024-12-30T17:55:41Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2024-12-30T17:55:41Z INFO Number of language-specific files num=0
2024-12-30T17:55:41Z INFO Detected config files num=1
trivy_exitcode=0
</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>
```hcl
*****************************
Checkov will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
checkov_exitcode=0
CTFLint Scan Success
Show Output
*****************************
Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version:0.9.1)
tflint will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running tflint in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0
Trivy Scan Success
Show Output
*****************************
Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2024-12-30T17:55:37Z INFO [vulndb] Need to update DB
2024-12-30T17:55:37Z INFO [vulndb] Downloading vulnerability DB...2024-12-30T17:55:37Z INFO [vulndb] Downloading artifact...repo="public.ecr.aws/aquasecurity/trivy-db:2"2024-12-30T17:55:40Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"2024-12-30T17:55:40Z INFO [vuln] Vulnerability scanning is enabled
2024-12-30T17:55:40Z INFO [misconfig] Misconfiguration scanning is enabled
2024-12-30T17:55:40Z INFO [misconfig] Need to update the built-in checks
2024-12-30T17:55:40Z INFO [misconfig] Downloading the built-in checks...160.80 KiB /160.80 KiB [------------------------------------------------------] 100.00%? p/s 100ms2024-12-30T17:55:40Z INFO [secret] Secret scanning is enabled
2024-12-30T17:55:40Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-12-30T17:55:40Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection2024-12-30T17:55:41Z INFO [terraformscanner] Scanning root module file_path="."2024-12-30T17:55:41Z WARN [terraformparser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.module="root"variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"2024-12-30T17:55:41Z INFO Number of language-specific files num=02024-12-30T17:55:41Z INFO Detected config files num=1trivy_exitcode=0
Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2024-12-30T19:42:30Z INFO [vulndb] Need to update DB
2024-12-30T19:42:30Z INFO [vulndb] Downloading vulnerability DB...
2024-12-30T19:42:30Z INFO [vulndb] Downloading artifact... repo="public.ecr.aws/aquasecurity/trivy-db:2"
2024-12-30T19:42:33Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"
2024-12-30T19:42:33Z INFO [vuln] Vulnerability scanning is enabled
2024-12-30T19:42:33Z INFO [misconfig] Misconfiguration scanning is enabled
2024-12-30T19:42:33Z INFO [misconfig] Need to update the built-in checks
2024-12-30T19:42:33Z INFO [misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [------------------------------------------------------] 100.00% ? p/s 100ms2024-12-30T19:42:34Z INFO [secret] Secret scanning is enabled
2024-12-30T19:42:34Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-12-30T19:42:34Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2024-12-30T19:42:35Z INFO [terraform scanner] Scanning root module file_path="."
2024-12-30T19:42:35Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2024-12-30T19:42:35Z INFO Number of language-specific files num=0
2024-12-30T19:42:35Z INFO Detected config files num=1
trivy_exitcode=0
</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>
```hcl
*****************************
Checkov will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
checkov_exitcode=0
CTFLint Scan Success
Show Output
*****************************
Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version:0.9.1)
tflint will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running tflint in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0
Trivy Scan Success
Show Output
*****************************
Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2024-12-30T19:42:30Z INFO [vulndb] Need to update DB
2024-12-30T19:42:30Z INFO [vulndb] Downloading vulnerability DB...2024-12-30T19:42:30Z INFO [vulndb] Downloading artifact...repo="public.ecr.aws/aquasecurity/trivy-db:2"2024-12-30T19:42:33Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"2024-12-30T19:42:33Z INFO [vuln] Vulnerability scanning is enabled
2024-12-30T19:42:33Z INFO [misconfig] Misconfiguration scanning is enabled
2024-12-30T19:42:33Z INFO [misconfig] Need to update the built-in checks
2024-12-30T19:42:33Z INFO [misconfig] Downloading the built-in checks...160.80 KiB /160.80 KiB [------------------------------------------------------] 100.00%? p/s 100ms2024-12-30T19:42:34Z INFO [secret] Secret scanning is enabled
2024-12-30T19:42:34Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-12-30T19:42:34Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection2024-12-30T19:42:35Z INFO [terraformscanner] Scanning root module file_path="."2024-12-30T19:42:35Z WARN [terraformparser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.module="root"variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"2024-12-30T19:42:35Z INFO Number of language-specific files num=02024-12-30T19:42:35Z INFO Detected config files num=1trivy_exitcode=0
Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2025-01-03T14:31:55Z INFO [vulndb] Need to update DB
2025-01-03T14:31:55Z INFO [vulndb] Downloading vulnerability DB...
2025-01-03T14:31:55Z INFO [vulndb] Downloading artifact... repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-03T14:31:57Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"
2025-01-03T14:31:57Z INFO [vuln] Vulnerability scanning is enabled
2025-01-03T14:31:57Z INFO [misconfig] Misconfiguration scanning is enabled
2025-01-03T14:31:57Z INFO [misconfig] Need to update the built-in checks
2025-01-03T14:31:57Z INFO [misconfig] Downloading the built-in checks...
160.80 KiB / 160.80 KiB [------------------------------------------------------] 100.00% ? p/s 100ms2025-01-03T14:31:57Z INFO [secret] Secret scanning is enabled
2025-01-03T14:31:57Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-03T14:31:57Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection
2025-01-03T14:31:58Z INFO [terraform scanner] Scanning root module file_path="."
2025-01-03T14:31:58Z WARN [terraform parser] Variable values was not found in the environment or variable files. Evaluating may not work correctly. module="root" variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"
2025-01-03T14:31:58Z INFO Number of language-specific files num=0
2025-01-03T14:31:58Z INFO Detected config files num=1
trivy_exitcode=0
</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>
```hcl
*****************************
Checkov will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running Checkov in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
checkov_exitcode=0
CTFLint Scan Success
Show Output
*****************************
Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version:0.9.1)
tflint will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running tflint in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0
Trivy Scan Success
Show Output
*****************************
Trivy will check the following folders:
terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
*****************************
Running Trivy in terraform/environments/digital-prison-reporting/modules/domains/replay-pipeline
2025-01-03T14:31:55Z INFO [vulndb] Need to update DB
2025-01-03T14:31:55Z INFO [vulndb] Downloading vulnerability DB...2025-01-03T14:31:55Z INFO [vulndb] Downloading artifact...repo="public.ecr.aws/aquasecurity/trivy-db:2"2025-01-03T14:31:57Z INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2"2025-01-03T14:31:57Z INFO [vuln] Vulnerability scanning is enabled
2025-01-03T14:31:57Z INFO [misconfig] Misconfiguration scanning is enabled
2025-01-03T14:31:57Z INFO [misconfig] Need to update the built-in checks
2025-01-03T14:31:57Z INFO [misconfig] Downloading the built-in checks...160.80 KiB /160.80 KiB [------------------------------------------------------] 100.00%? p/s 100ms2025-01-03T14:31:57Z INFO [secret] Secret scanning is enabled
2025-01-03T14:31:57Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-01-03T14:31:57Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.57/docs/scanner/secret#recommendation for faster secret detection2025-01-03T14:31:58Z INFO [terraformscanner] Scanning root module file_path="."2025-01-03T14:31:58Z WARN [terraformparser] Variable values was not found in the environment or variable files. Evaluating may not work correctly.module="root"variables="dms_replication_task_arn, glue_maintenance_compaction_job, glue_maintenance_retention_job, glue_s3_max_attempts, glue_s3_retry_max_wait_millis, glue_s3_retry_min_wait_millis, processed_files_check_max_attempts, processed_files_check_wait_interval_seconds, replication_task_id, s3_curated_path, s3_structured_path, step_function_execution_role_arn"2025-01-03T14:31:58Z INFO Number of language-specific files num=02025-01-03T14:31:58Z INFO Detected config files num=1trivy_exitcode=0
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
bot
added
the
environments-repository
koladeadewuyi-moj
temporarily deployed
to
digital-prison-reporting-test
GitHub Actions
Inactive
The batch job of the replay pipeline does not process diff files created by the reload pipeline. This is because the batch job looks for files matching the name pattern
LOAD*.parquetwhereas the reload diff files have the name patternpart-*.snappy.parquet.Steps to reproduce:
To fix this, an extra argument will be supplied to the batch job invocation during the replay pipeline:
--dpr.batch.load.fileglobpattern : {part-*.snappy.parquet,LOAD*parquet}