aws_ec2 inventory running against the same host multiple times

[yamjoepobuda]

Summary

We recently bumped amazon.aws collection to 4.1.0 from 4.0.0 and started experiencing errors due to process locks and other similar errors. We found that the inventory file was being parsed incorrectly. Instead of processing each host once, it appears to be processing each host for every IP address it has, in addition to the usual tag we use to name hosts.

Issue Type

Bug Report

Component Name

aws_ec2

Ansible Version

$ ansible --version
ansible [core 2.12.7]
  config file = /etc/ansible/ansible-amer.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
  ansible collection location = /usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0]
  jinja version = 3.1.2
  libyaml = False

Collection Versions

+ ansible-galaxy collection list

# /usr/share/ansible/collections/ansible_collections
Collection            Version
--------------------- -------
amazon.aws            4.1.0  
ansible.netcommon     3.1.0  
ansible.posix         1.4.0  
ansible.utils         2.6.1  
ansible.windows       1.10.0 
chocolatey.chocolatey 1.3.0  
cisco.ios             3.3.0  
cisco.meraki          2.10.1 
commscope.icx         1.0.5  
community.aws         4.0.0  
community.crypto      2.4.0  
community.general     5.4.0  
community.network     4.0.1  
community.windows     1.10.0 
fortinet.fortios      2.1.6  

# /usr/local/lib/python3.10/dist-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.3.0  
ansible.netcommon             2.6.1  
ansible.posix                 1.4.0  
ansible.utils                 2.6.1  
ansible.windows               1.10.0 
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.13.0 
check_point.mgmt              2.3.0  
chocolatey.chocolatey         1.2.0  
cisco.aci                     2.2.0  
cisco.asa                     2.1.0  
cisco.dnac                    6.4.0  
cisco.intersight              1.0.19 
cisco.ios                     2.8.1  
cisco.iosxr                   2.9.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.6.2  
cisco.mso                     1.4.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.9.1  
cisco.ucs                     1.8.0  
cloud.common                  2.1.1  
cloudscale_ch.cloud           2.2.2  
community.aws                 2.5.0  
community.azure               1.1.0  
community.ciscosmb            1.0.5  
community.crypto              2.3.2  
community.digitalocean        1.19.0 
community.dns                 2.2.0  
community.docker              2.6.0  
community.fortios             1.0.0  
community.general             4.8.2  
community.google              1.0.0  
community.grafana             1.4.0  
community.hashi_vault         2.5.0  
community.hrobot              1.4.0  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.1.0  
community.mongodb             1.4.0  
community.mysql               2.3.8  
community.network             3.3.0  
community.okd                 2.2.0  
community.postgresql          1.7.4  
community.proxysql            1.4.0  
community.rabbitmq            1.2.1  
community.routeros            2.1.0  
community.sap                 1.0.0  
community.sap_libs            1.1.0  
community.skydive             1.0.0  
community.sops                1.2.2  
community.vmware              1.18.0 
community.windows             1.10.0 
community.zabbix              1.7.0  
containers.podman             1.9.3  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.14 
dellemc.enterprise_sonic      1.1.1  
dellemc.openmanage            4.4.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.17.0 
fortinet.fortimanager         2.1.5  
fortinet.fortios              2.1.6  
frr.frr                       1.0.4  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.4  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.3  
infoblox.nios_modules         1.2.2  
inspur.sm                     1.3.0  
junipernetworks.junos         2.10.0 
kubernetes.core               2.3.1  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.17.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.19.1
netapp.storagegrid            21.10.0
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.3.0  
netbox.netbox                 3.7.1  
ngine_io.cloudstack           2.2.4  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.1  
openstack.cloud               1.8.0  
openvswitch.openvswitch       2.1.0  
ovirt.ovirt                   1.6.6  
purestorage.flasharray        1.13.0 
purestorage.flashblade        1.9.0  
sensu.sensu_go                1.13.1 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.29.0 
theforeman.foreman            2.2.0  
vmware.vmware_rest            2.1.5  
vyos.vyos                     2.8.0  
wti.remote                    1.0.3  

AWS SDK versions

I originally thought this was related to boto*, as those versions also changed in our deployment at the same time as the collection. I pinned the versions below, before rolling back the amazon.aws collection.

+ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /usr/local/lib/python3.10/dist-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.24.42
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.27.42
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: awscli, boto3, s3transfer

Once the collection was rolled back and the issue was resolved, I unpinned boto* to validate. This is after unpinning:

+ pip show boto boto3 botocore
debug2: channel 0: written 31 to efd 6
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /usr/local/lib/python3.10/dist-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.24.44
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.27.44
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: awscli, boto3, s3transfer

Tl;dr 4.1.0 remains broken on both sets of boto versions above

Configuration

+ ansible-config dump --only-changed
ANSIBLE_NOCOWS(/etc/ansible/ansible-amer.cfg) = True
ANSIBLE_PIPELINING(/etc/ansible/ansible-amer.cfg) = True
CACHE_PLUGIN(/etc/ansible/ansible-amer.cfg) = redis
CACHE_PLUGIN_CONNECTION(/etc/ansible/ansible-amer.cfg) = production-a-us-west-2-ansible-redis.shared.xxxx.cloud:6379:0
CACHE_PLUGIN_TIMEOUT(/etc/ansible/ansible-amer.cfg) = 86400
CALLBACKS_ENABLED(/etc/ansible/ansible-amer.cfg) = ['datadog_callback']
COLLECTIONS_PATHS(/etc/ansible/ansible-amer.cfg) = ['/usr/share/ansible/collections']
DEFAULT_FORKS(/etc/ansible/ansible-amer.cfg) = 50
DEFAULT_GATHER_TIMEOUT(/etc/ansible/ansible-amer.cfg) = 20
DEFAULT_POLL_INTERVAL(/etc/ansible/ansible-amer.cfg) = 5
DEFAULT_REMOTE_USER(/etc/ansible/ansible-amer.cfg) = ansible
DEFAULT_ROLES_PATH(/etc/ansible/ansible-amer.cfg) = ['/etc/ansible/roles']
DEFAULT_STRATEGY_PLUGIN_PATH(/etc/ansible/ansible-amer.cfg) = ['/usr/local/lib/python3.10/dist-packages/ansible_mitogen/plugins/strategy']
DEFAULT_TIMEOUT(/etc/ansible/ansible-amer.cfg) = 20
DISPLAY_SKIPPED_HOSTS(/etc/ansible/ansible-amer.cfg) = False
HOST_KEY_CHECKING(/etc/ansible/ansible-amer.cfg) = False
PERSISTENT_COMMAND_TIMEOUT(/etc/ansible/ansible-amer.cfg) = 30
PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible-amer.cfg) = 40
RETRY_FILES_ENABLED(/etc/ansible/ansible-amer.cfg) = False

OS / Environment

+ uname -a
Linux 565837f08633 5.11.0-1028-aws #31~20.04.1-Ubuntu SMP Fri Jan 14 14:37:50 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
+ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Steps to Reproduce

We can reproduce this issue by using version 4.1.0 of the collection in requirements.yml

collections:
  - name: amazon.aws
    version: 4.1.0

Expected Results

How it looked in 4.0.0:

TASK [Gathering Facts] *********************************************************
ok: [production-a-activedirectory-aws2-us-west-2-10.2x2.1xx.x4]
ok: [production-a-activedirectory-aws1-us-west-2-10.xx.xx.xx]

Actual Results

How it looks in 4.1.0:

TASK [Gathering Facts] *********************************************************
ok: [production-a-activedirectory-aws2-us-west-2-10.2x2.1xx.x4]
ok: [ec2-35-x65-x0-xx3.us-west-2.compute.amazonaws.com]
ok: [ip-10-2x2-1xx-x4.us-west-2.compute.internal]
ok: [ec2-34-x16-x5-9x.us-west-2.compute.amazonaws.com]
ok: [ip-10-2x2-1xx-x45.us-west-2.compute.internal]
ok: [production-a-activedirectory-aws1-us-west-2-10.xx.xx.xx]

This results in process locks as each task is being run on the host 3 times. Certain tasks require dedicated locks, those that do fail similarly to the example below:

 FAILED! => {"changed": true, "cmd": "Export-NpsConfiguration -Path C:\\NPS\\NPSConfig_2022-08-02.xml", "delta": "0:00:03.209783", "end": "2022-08-02 18:32:04.252882", "msg": "non-zero return code", "rc": 1, "start": "2022-08-02 18:32:01.043099", "stderr": "Export-NpsConfiguration : The process cannot access the file because it is being used by another process. (Exception \r\nfrom HRESULT: 0x80070020)\r\nAt line:1 char:65\r\n+ ... ing $false; Export-NpsConfiguration -Path C:\\NPS\\NPSConfig_2022-08-02 ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : NotSpecified: (Microsoft.Nps.C...gurationCommand:ExportNpsConfigurationCommand) [Export- \r\n   NpsConfiguration], FileLoadException\r\n    + FullyQualifiedErrorId : Export-NpsConfiguration.FileLoadException,Microsoft.Nps.Commands.ExportNpsConfigurationC \r\n   ommand", "stderr_lines": ["Export-NpsConfiguration : The process cannot access the file because it is being used by another process. (Exception ", "from HRESULT: 0x80070020)", "At line:1 char:65", "+ ... ing $false; Export-NpsConfiguration -Path C:\\NPS\\NPSConfig_2022-08-02 ...", "+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~", "    + CategoryInfo          : NotSpecified: (Microsoft.Nps.C...gurationCommand:ExportNpsConfigurationCommand) [Export- ", "   NpsConfiguration], FileLoadException", "    + FullyQualifiedErrorId : Export-NpsConfiguration.FileLoadException,Microsoft.Nps.Commands.ExportNpsConfigurationC ", "   ommand"], "stdout": "", "stdout_lines": []}

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[yamjoepobuda]

Possibly relevant, our aws_ec2.yml:

---
plugin: amazon.aws.aws_ec2
regions:
  - us-west-2
  - us-east-1
  - us-east-2
filters:
  instance-state-name: running
hostnames:
  - tag:ExtendedName
  - dns-name
  - private-dns-name
strict: False
keyed_groups:
  - key: tags.region
    prefix: tag_region
  - key: tags.Business
    prefix: tag_Business
  - key: tags.Environment
    prefix: tag_Environment
  - key: tags.ExtendedName
    prefix: tag_extendedname
  - key: tags.Service
    prefix: tag_Service
  - key: tags.Component
    prefix: tag_Component
  - key: tags.ADDomain
    prefix: tag_ADDomain
  - key: tags.DNS
    prefix: tag_DNS
compose:
  ansible_host: private_ip_address
...

[gregharvey]

We are also seeing this exact issue.

[goneri]

This PR should resolve the problem: #1026
It introduces a new configuration key called allow_duplicated_hosts that default to False. False is the old behaviour, True, the new one.
Can you please give it a try?

The PR is still marked a WIP (WorkInProgress) because I need to refresh the functional tests.