Dependencies are re-downloaded on every run

[calvinbui]

Issue Type

• Bug report

Molecule and Ansible details

ansible 2.7.5
  config file = None
  configured module search path = ['/home/calvin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/calvin/Desktop/ansible/lib/python3.6/site-packages/ansible
  executable location = /home/calvin/Desktop/ansible/bin/ansible
  python version = 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]
molecule, version 2.19.0

Molecule installation method (one of):

• pip

Ansible installation method (one of):

• pip

Detail any linters or test runners used: Ubuntu 18.04 in Docker

platforms:
  - name: ansible-awx
    image: ubuntu:18.04

Desired Behavior

Galaxy should download only once instead of every run, especially if it already exists and the version matches

Actual Behaviour

First run

(ansible) ➜  ansible-awx molecule converge                      
--> Validating schema /mnt/hgfs/repos/ansible-awx/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix
    
└── default
    ├── dependency
    ├── create
    ├── prepare
    └── converge
    
--> Scenario: 'default'
--> Action: 'dependency'
    - downloading role 'ansible_git', owned by calvinbui
    - downloading role from https://github.com/calvinbui/ansible-git/archive/1.0.tar.gz
    - extracting calvinbui.ansible_git to /tmp/molecule/ansible-awx/default/roles/calvinbui.ansible_git
    - calvinbui.ansible_git (1.0) was installed successfully
    - downloading role 'ansible_pip', owned by calvinbui
    - downloading role from https://github.com/calvinbui/ansible-pip/archive/1.0.tar.gz
    - extracting calvinbui.ansible_pip to /tmp/molecule/ansible-awx/default/roles/calvinbui.ansible_pip
    - calvinbui.ansible_pip (1.0) was installed successfully
    - downloading role 'ansible_nodejs', owned by calvinbui
    - downloading role from https://github.com/calvinbui/ansible-nodejs/archive/1.0.tar.gz
    - extracting calvinbui.ansible_nodejs to /tmp/molecule/ansible-awx/default/roles/calvinbui.ansible_nodejs
    - calvinbui.ansible_nodejs (1.0) was installed successfully
    - downloading role 'ansible_docker', owned by calvinbui
    - downloading role from https://github.com/calvinbui/ansible-docker/archive/1.0.tar.gz
    - extracting calvinbui.ansible_docker to /tmp/molecule/ansible-awx/default/roles/calvinbui.ansible_docker
    - calvinbui.ansible_docker (1.0) was installed successfully
Dependency completed successfully.

Second run

(ansible) ➜  ansible-awx molecule converge
--> Validating schema /mnt/hgfs/repos/ansible-awx/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix
    
└── default
    ├── dependency
    ├── create
    ├── prepare
    └── converge
    
--> Scenario: 'default'
--> Action: 'dependency'
    - changing role calvinbui.ansible_git from 1.0 to 1.0
    - downloading role 'ansible_git', owned by calvinbui
    - downloading role from https://github.com/calvinbui/ansible-git/archive/1.0.tar.gz
    - extracting calvinbui.ansible_git to /tmp/molecule/ansible-awx/default/roles/calvinbui.ansible_git
    - calvinbui.ansible_git (1.0) was installed successfully
    - changing role calvinbui.ansible_pip from 1.0 to 1.0
    - downloading role 'ansible_pip', owned by calvinbui
    - downloading role from https://github.com/calvinbui/ansible-pip/archive/1.0.tar.gz
    - extracting calvinbui.ansible_pip to /tmp/molecule/ansible-awx/default/roles/calvinbui.ansible_pip
    - calvinbui.ansible_pip (1.0) was installed successfully
    - changing role calvinbui.ansible_nodejs from 1.0 to 1.0
    - downloading role 'ansible_nodejs', owned by calvinbui
    - downloading role from https://github.com/calvinbui/ansible-nodejs/archive/1.0.tar.gz
    - extracting calvinbui.ansible_nodejs to /tmp/molecule/ansible-awx/default/roles/calvinbui.ansible_nodejs
    - calvinbui.ansible_nodejs (1.0) was installed successfully
    - changing role calvinbui.ansible_docker from 1.0 to 1.0
    - downloading role 'ansible_docker', owned by calvinbui
    - downloading role from https://github.com/calvinbui/ansible-docker/archive/1.0.tar.gz
    - extracting calvinbui.ansible_docker to /tmp/molecule/ansible-awx/default/roles/calvinbui.ansible_docker
    - calvinbui.ansible_docker (1.0) was installed successfully
Dependency completed successfully.

molecule/default/requirements.yml

---

- src: calvinbui.ansible_git
  version: 1.0
- src: calvinbui.ansible_pip
  version: 1.0
- src: calvinbui.ansible_nodejs
  version: 1.0
- src: calvinbui.ansible_docker
  version: 1.0

molecule/default/mocule.yml (snippet)

---
dependency:
  name: galaxy

[decentral1se]

Galaxy should download only once instead of every run, especially if it already exists and the version matches

Yes please!!! This should be milestoned for the next release IMHO.

[ragingpastry]

This is actually built in, but the default is to download every time. Here is the specific line which configures the default: https://github.com/ansible/molecule/blob/master/molecule/dependency/ansible_galaxy.py#L84-L85

Check out this example molecule.yml file which shows this option.

---
dependency:
  name: galaxy
  options:
    force: False
driver:
  name: docker
lint:
  name: yamllint
platforms:
  - name: instance
    image: centos:7
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

The question becomes whether we want to change the default from forcing a download. I can see both ways, but if anyone else wants to chime in that would be great! Either way we should change the documentation to reflect this option. This can be found here: https://github.com/ansible/molecule/blob/master/molecule/dependency/ansible_galaxy.py#L47-L52

[calvinbui]

it should work the way ansible-playbook and ansible-galaxy works imo

[manics]

The problem with force: False is it will only install missing dependencies but won't update existing ones even if the version is different. Ideally there would be an option to only update out-of-date dependencies.

[decentral1se]

The question becomes whether we want to change the default from forcing a download.
it should work the way ansible-playbook and ansible-galaxy works imo

Then it should be force:false by default and we should document how to configure force:true. No?

EDIT: When using ansible-galaxy, you have to pass -f to force (not default)

[ragingpastry]

Sticking with the ansible-galaxy default makes sense to me. I think.

@manics I think for that use case you should specify force: True in your molecule config. Since the dependency check is just a wrapper around ansible-galaxy I think the sane default would mirror ansible-galaxy's sane default

[manics]

@ragingpastry The problem with --force is it always downloads the role, even if it's at the correct version:

$ ansible-galaxy --version
ansible-galaxy 2.7.5

Install role version 1.0.0:

$ ansible-galaxy install -p roles openmicroscopy.nginx,1.0.0
- downloading role 'nginx', owned by openmicroscopy
- downloading role from https://github.com/openmicroscopy/ansible-role-nginx/archive/1.0.0.tar.gz
- extracting openmicroscopy.nginx to /tmp/roles/openmicroscopy.nginx
- openmicroscopy.nginx (1.0.0) was installed successfully

Update to version 1.0.1, it fails without --force:

$ ansible-galaxy install -p roles openmicroscopy.nginx,1.0.1
 [WARNING]: - openmicroscopy.nginx (1.0.0) is already installed - use --force
to change version to 1.0.1

$ ansible-galaxy install -p roles openmicroscopy.nginx,1.0.1 --force
- changing role openmicroscopy.nginx from 1.0.0 to 1.0.1
- downloading role 'nginx', owned by openmicroscopy
- downloading role from https://github.com/openmicroscopy/ansible-role-nginx/archive/1.0.1.tar.gz
- extracting openmicroscopy.nginx to /tmp/roles/openmicroscopy.nginx
- openmicroscopy.nginx (1.0.1) was installed successfully

But rerunning with --force downloads the role again:

$ ansible-galaxy install -p roles openmicroscopy.nginx,1.0.1 --force
- changing role openmicroscopy.nginx from 1.0.1 to 1.0.1
- downloading role 'nginx', owned by openmicroscopy
- downloading role from https://github.com/openmicroscopy/ansible-role-nginx/archive/1.0.1.tar.gz
- extracting openmicroscopy.nginx to /tmp/roles/openmicroscopy.nginx
- openmicroscopy.nginx (1.0.1) was installed successfully

[ragingpastry]

I understand why force could be problematic. I don't see how Molecule should be solving this problem though. It already provides an interface for modifying the options passed to ansible-galaxy. Maybe I am misunderstanding what you are saying.

I think that changing the default ansible-galaxy option in Molecule to align with ansible-galaxy's own defaults make sense.

[manics]

I'm not sure about removing --force from molecule despite its disadvantages. Molecule's primary purpose is for testing, and --force ensures it's as close to your intended test setup as possible.

Without it you might bump a required version (either intentionally, or from a git pull) without realising the downloaded role is out of date. Since there's no ideal solution at the moment could molecule check the status from ansible-galaxy and perhaps fail if there's a role mismatch?

[rockandska]

Just my 2cts, but i had the same concern even without molecule and this behaviour should be fix on the ansible-galaxy/mazer side.
This is ansible-galaxy/mazer who need to choose to download the role again and not molecule and many discussions already exist about this :
ansible/proposals#23
ansible/ansible#6466
ansible/galaxy-issues#49
ansible/mazer#173

Regards,

[decentral1se]

Closing out based on two points:

• Multiple people said this shouldn't be solved here (see Dependencies are re-downloaded on every run #1641 (comment) and Dependencies are re-downloaded on every run #1641 (comment))
• Setting force:false is maybe bad (see Dependencies are re-downloaded on every run #1641 (comment))

Let's put some pressure on galaxy/mazer 👍

[decentral1se]

@rgarrigue has come to chat about this on IRC and I'm re-opening.

[iainvm]

--force should still be called into question. If we take Comment #1641 and say "molecule is a testing tool and --force protects us from wrong dependencies being used in a test then we should be using --force-with-deps as this ensures all sub deps are also re-download.

This can be seen when using roles from tar files as they aren't perceived to have "versions" even if the URL changes if the rolename is the same it assumes it's the correct role.

[ssbarnea]

I am closing this because this is more of a limitation of galaxy and there is not much molecule can do about it. Still, this does not mean that a PR related to this ticket would not be appreciated.

[jmartinspt]

I recently started using the molecule to integrate my project's Ansible roles into the CI/CD process and quickly ran into two problems:

• Duplication of the dependency file, for the use of the molecule (project/requirements.yml) and the role itself (project/meta/requirements.yml)
• Inability to force the download of roles if they had already been installed previously, being cached in the folder ~/.cache.

After much research and many attempts, I found the following solution:

.env.yml

---
ANSIBLE_ROLES_PATH: galaxy_roles

.gitignore

(...)
galaxy_roles/
(...)

molecule/default/molecule.yml

---
dependency:
  name: shell
  command: ansible-galaxy install -r meta/requirements.yml -p galaxy_roles/ --force
driver:
  name: docker
platforms:
  - name: molecule-ubuntu
    image: ubuntu
    pre_build_image: true
provisioner:
  name: ansible
verifier:
  name: ansible
lint: |
  set -e
  yamllint .
  ansible-lint

This way, just by changing dependency to shell and executing the command to install roles to a different folder (always forcing the installation) I can use the same dependency file for the molecule and the role itself.
So I always have the most up-to-date dependencies and avoid code duplication.

I hope it's useful to someone.

[ssbarnea]

In fact if you are using latest version of molecule, it will install repository revel requirements.yml itself without you having to add anything to molecule.yml file. You are better of removing dependency block from molecule config.

In fact molecule own logic is smarter than galaxy itself and installs dependencies only when they are missing (galaxy from 2.9 does not support that, not even ability to upgrade them).

[jmartinspt]

In fact if you are using latest version of molecule, it will install repository revel requirements.yml itself without you having to add anything to molecule.yml file. You are better of removing dependency block from molecule config.

In fact molecule own logic is smarter than galaxy itself and installs dependencies only when they are missing (galaxy from 2.9 does not support that, not even ability to upgrade them).

Thanks for your reply ssbarnea, but in may case I using my own dependencies from my git repository, and when I changed something and doesn't change the tag, the molecule not will force the dependence update, it executes the command without "--force" flag, and I am using the latest version of molecule. And with the solution that I proposed, I didn't need add a an additional "dependencies.yml" to the root project folder, it will consume the meta/dependencies.yml directly from the role.