Dso 2239/ansible failure handling in ami builds

[shajida95]

No description provided.

[github-actions]

teams/nomis/rhel_6_10_weblogic_appserver_10_3 terragrunt plan on pull_request event #1102

module.imagebuilder.data.terraform_remote_state.imagebuilder_mp: Reading...
module.imagebuilder.data.terraform_remote_state.core_shared_services_production: Reading...
module.imagebuilder.data.aws_secretsmanager_secret.environment_management: Reading...
module.imagebuilder.data.terraform_remote_state.imagebuilder_mp: Read complete after 1s
module.imagebuilder.data.aws_secretsmanager_secret.environment_management: Read complete after 0s [id=<REDACTED>]
module.imagebuilder.data.aws_secretsmanager_secret_version.environment_management: Reading...
module.imagebuilder.data.aws_secretsmanager_secret_version.environment_management: Read complete after 1s [id=<REDACTED>]
module.imagebuilder.data.terraform_remote_state.core_shared_services_production: Read complete after 2s
module.imagebuilder.data.aws_imagebuilder_component.this["update-linux"]: Reading...
module.imagebuilder.data.aws_caller_identity.current: Reading...
module.imagebuilder.data.aws_caller_identity.current: Read complete after 0s [id=<REDACTED>]
module.imagebuilder.data.aws_kms_key.hmpps_ebs_encryption_cmk: Reading...
module.imagebuilder.data.aws_kms_key.hmpps_ebs_encryption_cmk: Read complete after 1s [id=<REDACTED>]
module.imagebuilder.data.aws_imagebuilder_component.this["update-linux"]: Read complete after 1s [id=<REDACTED>]
module.imagebuilder.aws_imagebuilder_infrastructure_configuration.this: Refreshing state... [id=<REDACTED>]
module.imagebuilder.aws_imagebuilder_image_recipe.this: Refreshing state... [id=<REDACTED>]
module.imagebuilder.aws_imagebuilder_distribution_configuration.this: Refreshing state... [id=<REDACTED>]
module.imagebuilder.aws_imagebuilder_image_pipeline.this: Refreshing state... [id=<REDACTED>]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.imagebuilder.aws_imagebuilder_distribution_configuration.this will be updated in-place
  ~ resource "aws_imagebuilder_distribution_configuration" "this" {
        id           = "arn:aws:imagebuilder:eu-west-2:374269020027:distribution-configuration/nomis-rhel-6-10-weblogic-appserver-10-3"
        name         = "nomis_rhel_6_10_weblogic_appserver_10_3"
      ~ tags         = {
            "application"                  = "NOMIS"
            "branch"                       = "main"
            "business-unit"                = "HMPPS"
            "image-pipeline"               = "nomis_rhel_6_10_weblogic_appserver_10_3"
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            "is-production"                = "true"
            "os-version"                   = "rhel 6.10"
            "owner"                        = "DSO: digital-studio-operations-team@digital.justice.gov.uk"
            "release-or-patch"             = "release"
            "source-code"                  = "https://github.com/ministryofjustice/modernisation-platform-ami-builds/tree/main/teams/nomis"
            "update-linux-version"         = "1.0.2"
        }
      ~ tags_all     = {
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            # (10 unchanged elements hidden)
        }
        # (4 unchanged attributes hidden)

      - distribution {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      + distribution {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
    }

  # module.imagebuilder.aws_imagebuilder_image_pipeline.this must be replaced
-/+ resource "aws_imagebuilder_image_pipeline" "this" {
      ~ arn                              = "arn:aws:imagebuilder:eu-west-2:374269020027:image-pipeline/nomis-rhel-6-10-weblogic-appserver-10-3" -> (known after apply)
      ~ date_created                     = "2023-11-03T12:58:46.349Z" -> (known after apply)
      + date_last_run                    = (known after apply)
      ~ date_next_run                    = "2023-12-02T00:00:00.000Z" -> (known after apply)
      ~ date_updated                     = "2023-11-03T12:58:46.349Z" -> (known after apply)
      ~ id                               = "arn:aws:imagebuilder:eu-west-2:374269020027:image-pipeline/nomis-rhel-6-10-weblogic-appserver-10-3" -> (known after apply)
      ~ image_recipe_arn                 = "arn:aws:imagebuilder:eu-west-2:374269020027:image-recipe/nomis-rhel-6-10-weblogic-appserver-10-3/0.2.6" # forces replacement -> (known after apply) # forces replacement
      ~ infrastructure_configuration_arn = "arn:aws:imagebuilder:eu-west-2:374269020027:infrastructure-configuration/nomis-rhel-6-10-weblogic-appserver-10-3-0-2-6" -> (known after apply)
        name                             = "nomis_rhel_6_10_weblogic_appserver_10_3"
      ~ platform                         = "Linux" -> (known after apply)
      ~ tags                             = {
            "application"                  = "NOMIS"
            "branch"                       = "main"
            "business-unit"                = "HMPPS"
            "image-pipeline"               = "nomis_rhel_6_10_weblogic_appserver_10_3"
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            "is-production"                = "true"
            "os-version"                   = "rhel 6.10"
            "owner"                        = "DSO: digital-studio-operations-team@digital.justice.gov.uk"
            "release-or-patch"             = "release"
            "source-code"                  = "https://github.com/ministryofjustice/modernisation-platform-ami-builds/tree/main/teams/nomis"
            "update-linux-version"         = "1.0.2"
        }
      ~ tags_all                         = {
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            # (10 unchanged elements hidden)
        }
        # (4 unchanged attributes hidden)

      ~ schedule {
          + timezone                           = (known after apply)
            # (2 unchanged attributes hidden)
        }

        # (1 unchanged block hidden)
    }

  # module.imagebuilder.aws_imagebuilder_image_recipe.this must be replaced
+/- resource "aws_imagebuilder_image_recipe" "this" {
      ~ arn              = "arn:aws:imagebuilder:eu-west-2:374269020027:image-recipe/nomis-rhel-6-10-weblogic-appserver-10-3/0.2.6" -> (known after apply)
      ~ date_created     = "2023-11-03T12:58:45.410Z" -> (known after apply)
      ~ id               = "arn:aws:imagebuilder:eu-west-2:374269020027:image-recipe/nomis-rhel-6-10-weblogic-appserver-10-3/0.2.6" -> (known after apply)
        name             = "nomis_rhel_6_10_weblogic_appserver_10_3"
      ~ owner            = "374269020027" -> (known after apply)
      ~ platform         = "Linux" -> (known after apply)
      ~ tags             = {
            "application"                  = "NOMIS"
            "branch"                       = "main"
            "business-unit"                = "HMPPS"
            "image-pipeline"               = "nomis_rhel_6_10_weblogic_appserver_10_3"
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            "is-production"                = "true"
            "os-version"                   = "rhel 6.10"
            "owner"                        = "DSO: digital-studio-operations-team@digital.justice.gov.uk"
            "release-or-patch"             = "release"
            "source-code"                  = "https://github.com/ministryofjustice/modernisation-platform-ami-builds/tree/main/teams/nomis"
            "update-linux-version"         = "1.0.2"
        }
      ~ tags_all         = {
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            # (10 unchanged elements hidden)
        }
      + user_data_base64 = (known after apply)
      ~ version          = "0.2.6" -> "0.2.7" # forces replacement
        # (2 unchanged attributes hidden)

      - block_device_mapping {
          - device_name = "/dev/sda1" -> null
          - no_device   = false -> null

          - ebs {
              - delete_on_termination = "true" -> null
              - encrypted             = "true" -> null
              - iops                  = 0 -> null
              - kms_key_id            = "arn:aws:kms:eu-west-2:374269020027:key/12984197-3371-4c21-8e43-a88a1581e691" -> null
              - throughput            = 0 -> null
              - volume_size           = 30 -> null
              - volume_type           = "gp3" -> null
            }
        }
      - block_device_mapping {
          - device_name = "/dev/sdb" -> null
          - no_device   = false -> null

          - ebs {
              - delete_on_termination = "true" -> null
              - encrypted             = "true" -> null
              - iops                  = 0 -> null
              - kms_key_id            = "arn:aws:kms:eu-west-2:374269020027:key/12984197-3371-4c21-8e43-a88a1581e691" -> null
              - throughput            = 0 -> null
              - volume_size           = 150 -> null
              - volume_type           = "gp3" -> null
            }
        }
      + block_device_mapping {
          + device_name = "/dev/sda1"

          + ebs {
              + delete_on_termination = "true"
              + encrypted             = "true"
              + kms_key_id            = "arn:aws:kms:eu-west-2:374269020027:key/12984197-3371-4c21-8e43-a88a1581e691"
              + volume_size           = 30
              + volume_type           = "gp3"
            }
        }
      + block_device_mapping {
          + device_name = "/dev/sdb"

          + ebs {
              + delete_on_termination = "true"
              + encrypted             = "true"
              + kms_key_id            = "arn:aws:kms:eu-west-2:374269020027:key/12984197-3371-4c21-8e43-a88a1581e691"
              + volume_size           = 150
              + volume_type           = "gp3"
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.imagebuilder.aws_imagebuilder_infrastructure_configuration.this must be replaced
-/+ resource "aws_imagebuilder_infrastructure_configuration" "this" {
      ~ arn                           = "arn:aws:imagebuilder:eu-west-2:374269020027:infrastructure-configuration/nomis-rhel-6-10-weblogic-appserver-10-3-0-2-6" -> (known after apply)
      ~ date_created                  = "2023-11-03T12:58:45.596Z" -> (known after apply)
      + date_updated                  = (known after apply)
      ~ id                            = "arn:aws:imagebuilder:eu-west-2:374269020027:infrastructure-configuration/nomis-rhel-6-10-weblogic-appserver-10-3-0-2-6" -> (known after apply)
      ~ name                          = "nomis_rhel_6_10_weblogic_appserver_10_3_0_2_6" -> "nomis_rhel_6_10_weblogic_appserver_10_3_0_2_7" # forces replacement
      ~ resource_tags                 = {
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            # (10 unchanged elements hidden)
        }
      ~ tags                          = {
            "application"                  = "NOMIS"
            "branch"                       = "main"
            "business-unit"                = "HMPPS"
            "image-pipeline"               = "nomis_rhel_6_10_weblogic_appserver_10_3"
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            "is-production"                = "true"
            "os-version"                   = "rhel 6.10"
            "owner"                        = "DSO: digital-studio-operations-team@digital.justice.gov.uk"
            "release-or-patch"             = "release"
            "source-code"                  = "https://github.com/ministryofjustice/modernisation-platform-ami-builds/tree/main/teams/nomis"
            "update-linux-version"         = "1.0.2"
        }
      ~ tags_all                      = {
          ~ "image-recipe"                 = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
          ~ "infrastructure-configuration" = "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.6" -> "nomis_rhel_6_10_weblogic_appserver_10_3/0.2.7"
            # (10 unchanged elements hidden)
        }
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 3 to add, 1 to change, 3 to destroy.

[github-actions]

commonimages/components terraform plan on pull_request event #190

data.aws_secretsmanager_secret.environment_management: Reading...
data.aws_secretsmanager_secret.environment_management: Read complete after 1s [id=<REDACTED>]
data.aws_secretsmanager_secret_version.environment_management: Reading...
data.aws_secretsmanager_secret_version.environment_management: Read complete after 0s [id=<REDACTED>]
data.aws_caller_identity.current: Reading...
data.aws_caller_identity.current: Read complete after 0s [id=<REDACTED>]
data.aws_kms_key.hmpps_ebs_encryption_cmk: Reading...
data.aws_kms_key.hmpps_ebs_encryption_cmk: Read complete after 1s [id=<REDACTED>]
aws_imagebuilder_component.this["aws_cli.yml"]: Refreshing state... [id=<REDACTED>]
aws_imagebuilder_component.this["python_3_9.yml"]: Refreshing state... [id=<REDACTED>]
aws_imagebuilder_component.this["ansible.yml"]: Refreshing state... [id=<REDACTED>]
aws_imagebuilder_component.this["yum_packages.yml"]: Refreshing state... [id=<REDACTED>]
aws_imagebuilder_component.this["prometheus_windows_exporter.yml"]: Refreshing state... [id=<REDACTED>]
aws_imagebuilder_component.this["powershell_core.yml"]: Refreshing state... [id=<REDACTED>]
aws_imagebuilder_component.this["python_3_6.yml"]: Refreshing state... [id=<REDACTED>]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+/- create replacement and then destroy

Terraform will perform the following actions:

  # aws_imagebuilder_component.this["ansible.yml"] must be replaced
+/- resource "aws_imagebuilder_component" "this" {
      ~ arn                   = "arn:aws:imagebuilder:eu-west-2:374269020027:component/ansible/0.0.7/1" -> (known after apply)
      ~ data                  = <<-EOT # forces replacement
            ---
            name: ansible
            description: Install ansible roles
            schemaVersion: 1.0
            parameters:
              - Version:
                  type: string
          -       default: 0.0.7
          +       default: 0.0.8
                  description: "Component version, increment if you make changes."
              - Platform:
                  type: string
                  default: "Linux"
                  description: Platform.
              - Ami:
                  type: string
                  description: "Name of AMI. There must be a group_vars/ami_$Ami file in the repo."
              - Branch:
                  type: string
                  default: main
                  description: Git branch to use when cloning the ansible repo
              - AnsibleRepo:
                  type: string
                  default: modernisation-platform-configuration-management
                  description: The ansible github repo to clone
              - AnsibleRepoDir:
                  type: string
                  default: ansible
                  description: The directory in the repo where the ansible code is found
              - AnsibleTags:
                  type: string
                  default: amibuild
                  description: The tags to run ansible with
              - AnsibleArgs:
                  type: string
                  default: ""
                  description: Any other additional arguments to pass into ansible
            phases:
              - name: build
                steps:
                  # Install ansible roles
                  - name: InstallAndRunAnsible
                    action: ExecuteBash
                    inputs:
                      commands:
                        - |
                          # do not set -u as it breaks on RedHat 6
                          set -e
                          PATH=$PATH:/usr/local/bin
                          run_ansible() {
                            # define all params here to make standalone testing of script easier
                            repo="{{ AnsibleRepo }}"
                            ami_tag="{{ Ami }}"
                            ansible_repo_dir="{{ AnsibleRepoDir }}"
                            ansible_tags="{{ AnsibleTags }}"
                            ansible_args="{{ AnsibleArgs }}"
                            branch="{{ Branch }}"
            
                            # clone ansible roles and playbook
                            yum install -y git
                            ansible_dir=$(mktemp -d)
                            echo "Cloning $repo into $ansible_dir using branch=$branch"
                            cd $ansible_dir
                            git clone -b $branch "https://github.com/ministryofjustice/$repo.git"
            
                            # set python version
                            # check if already installed (e.g. RHEL6/7 via python component)
                            if [[ $(which python3.9 2> /dev/null) ]]; then
                              python=$(which python3.9)
                            elif [[ $(which python3.6 2> /dev/null) ]]; then
                              python=$(which python3.6)
                            else
                              # otherwise just install via yum
                              yum install -y python39 || true
                              if [[ $(which python3.9 2> /dev/null) ]]; then
                                python=$(which python3.9)
                              else
                                echo "Python3.9/3.6 not found"
                                exit 1
                              fi
                            fi
                            echo "python: $python"
            
                            # install python dependencies outside of virtual env so ansible
                            # can be executed remotely
                            cd $ansible_dir/$repo/$ansible_repo_dir
                            $python -m pip install -r requirements.txt
            
                            # activate virtual environment
                            mkdir $ansible_dir/python-venv && cd "$_"
                            $python -m venv ansible
                            source ansible/bin/activate
                            $python -m pip install --upgrade pip
                            if [[ "$python" =~ 3.6 ]]; then
                              $python -m pip install wheel
                              $python -m pip install cryptography==2.3
                              export LC_ALL=en_US.UTF-8
                              $python -m pip install ansible-core==2.11.12
                            else
                              $python -m pip install ansible==6.0.0
                            fi
            
                            # install requirements in virtual env
                            cd $ansible_dir/$repo/$ansible_repo_dir
                            $python -m pip install -r requirements.txt
                            ansible-galaxy role install -r requirements.yml
                            ansible-galaxy collection install -r requirements.yml --force
            
                            # run ansible (note comma after localhost is deliberate)
                            ansible-playbook site.yml \
                            --connection=local \
                            --inventory localhost, \
                            --extra-vars "ansible_python_interpreter=$python" \
                            --extra-vars "target=localhost" \
                            --extra-vars "@group_vars/ami_$ami_tag.yml" \
                            --tags "$ansible_tags" \
                            --become $ansible_args
            
                            # Cleanup
                            deactivate
                            echo "Cleaning up $ansible_dir"
                            rm -rf $ansible_dir/$repo
                            rm -rf $ansible_dir/python-venv
                            rmdir $ansible_dir
                          }
                          echo "ansible $ami_tag start" | logger -p local3.info -t ami-component
          -               run_ansible 2>&1 | logger -p local3.info -t ami-component -s 2>&1
          +               run_ansible | logger -p local3.info -t ami-component -s 2>&1
                          echo "ansible $ami_tag end" | logger -p local3.info -t ami-component
        EOT
      ~ date_created          = "2023-11-08T12:58:46.688Z" -> (known after apply)
      ~ encrypted             = true -> (known after apply)
      ~ id                    = "arn:aws:imagebuilder:eu-west-2:374269020027:component/ansible/0.0.7/1" -> (known after apply)
        name                  = "ansible"
      ~ owner                 = "374269020027" -> (known after apply)
      - supported_os_versions = [] -> null
        tags                  = {
            "application"   = "n/a"
            "branch"        = "main"
            "business-unit" = "HMPPS"
            "is-production" = "true"
            "owner"         = "digital-studio-operations-team@digital.justice.gov.uk"
            "source-code"   = "https://github.com/ministryofjustice/modernisation-platform-ami-builds/tree/main/commonimages/components"
        }
      ~ type                  = "BUILD" -> (known after apply)
      ~ version               = "0.0.7" -> "0.0.8" # forces replacement
        # (5 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Warning: Value for undeclared variable

The root module does not declare a variable named "account_to_distribute_ami"
but a value was found in file "terraform.tfvars". If you meant to use this
value, add a "variable" block to the configuration.

To silence these warnings, use TF_VAR_... environment variables to provide
certain "global" settings to all configurations in your organization. To
reduce the verbosity of these warnings, use the -compact-warnings option.