ansible-lint should support installing pre-release versions of collection dependencies

[alinabuzachis]

Summary

The ansible-lint action should support installation of pre-release versions of collections using the --pre flag. For example, when using the ansible/ansible-lint@v6.18.0 action in this PR redhat-cop/cloud.aws_ops#84, it should successfully install the amazon.aws and community.aws dependencies, but it fails as showed below.

Issue Type

• Bug Report

OS / ENVIRONMENT

ansible-lint --version

• ansible installation method: one of source, pip, OS package
• ansible-lint installation method: one of source, pip, OS package

STEPS TO REPRODUCE

Desired Behavior

Possible security bugs should be reported via email to security@ansible.com

Actual Behavior

Full log https://github.com/redhat-cop/cloud.aws_ops/actions/runs/5965427292/job/16182852719?pr=84#step:3:759

Run ansible-lint 
  ansible-lint 
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.4/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib:/opt/hostedtoolcache/Python/3.9.17/x64/lib
ERROR    Your branch is up to date with 'origin/main'.
Your branch is up to date with 'origin/main'.
Starting galaxy collection install process
Process install dependency map
to see the full traceback, use -vvv

ERROR    No config file found; using defaults
Cloning into '/home/runner/.ansible/tmp/ansible-local-21886m5kpzje/tmp0oi_l78v/amazon.aws7vk4rz4r'...
Already on 'main'
Cloning into '/home/runner/.ansible/tmp/ansible-local-21886m5kpzje/tmp0oi_l78v/community.aws_gc6_6xk'...
Already on 'main'
ERROR! Unexpected Exception, this is probably a bug: Virtual collections do not have a namespace

Got 250 exit code while running: ansible-galaxy collection install -v -r tests/integration/requirements.yml
Error: Process completed with exit code 1.

[ssbarnea]

I do think that this might be an ansible-core bug as I am able to reproduce it with ansible-galaxy itself:

# ansible-galaxy collection install -v -r tests/integration/requirements.yml
---
collections:
  - name: https://github.com/ansible-collections/amazon.aws.git
    type: git
    version: main
  - name: https://github.com/ansible-collections/community.aws.git
    type: git
    version: main

@sivel @bcoca Any hints about this?

ERROR! Unexpected Exception, this is probably a bug: Virtual collections do not have a namespace

I also observed ansible/ansible#79109 on core but I am not sure if that is the cause. I added the debug commands and I seen:

Your branch is up to date with 'origin/main'.
Initial connection to galaxy_server: https://galaxy.ansible.com
Found API version 'v1, v2' with Galaxy server default (https://galaxy.ansible.com/api/)
Opened /Users/ssbarnea/.ansible/galaxy_token
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/community/crypto/
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/community/general/
  9324 1693401016.07419: Received end signal for display_progress display thread
Traceback (most recent call last):
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/galaxy/collection/__init__.py", line 1826, in _resolve_depenency_map
    return collection_dep_resolver.resolve(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/resolvelib/resolvers.py", line 254, in _attempt_to_pin_criterion
    raise InconsistentCandidate(candidate, criterion)
resolvelib.resolvers.InconsistentCandidate: Provided candidate <amazon.aws:7.0.0-dev0 of type 'dir' from /Users/ssbarnea/.ansible/tmp/ansible-local-93245x3zj5p0/tmpp55u4i70/amazon.awsuxf8d2e5> does not satisfy <amazon.aws:7.0.0-dev0 of type 'dir' from /Users/ssbarnea/.ansible/tmp/ansible-local-93245x3zj5p0/tmpp55u4i70/amazon.awsuxf8d2e5>, <amazon.aws:>=7.0.0-dev0 of type 'galaxy' from Galaxy>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/bin/ansible-galaxy", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/cli/galaxy.py", line 1853, in main
    GalaxyCLI.cli_executor(args)
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/cli/__init__.py", line 659, in cli_executor
    exit_code = cli.run()
                ^^^^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/cli/galaxy.py", line 719, in run
    return context.CLIARGS['func']()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/cli/galaxy.py", line 119, in method_wrapper
    return wrapped_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/cli/galaxy.py", line 1373, in execute_install
    self._execute_install_collection(
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/cli/galaxy.py", line 1422, in _execute_install_collection
    install_collections(
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/galaxy/collection/__init__.py", line 727, in install_collections
    dependency_map = _resolve_depenency_map(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/galaxy/collection/__init__.py", line 1851, in _resolve_depenency_map
    parents = [
              ^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/galaxy/collection/__init__.py", line 1852, in <listcomp>
    "%s.%s:%s" % (p.namespace, p.name, p.ver)
                  ^^^^^^^^^^^
  File "/Users/ssbarnea/.pyenv/versions/3.11.4/lib/python3.11/site-packages/ansible/galaxy/dependency_resolution/dataclasses.py", line 507, in namespace
    raise TypeError('Virtual collections do not have a namespace')
TypeError: Virtual collections do not have a namespace
ansible-galaxy collection install -v -r tests/integration/requirements.yml  9.68s user 1.93s system 73% cpu 15.747 total

[sivel]

Arguably a bug, because it results in a traceback, but ultimately the problem is that installing collections from git causing issues by not meeting the requirements of dependency resolution. Although --pre does seem to work here, the more appropriate solution in this instance is --no-deps.

Feel free to file an issue regarding the traceback.

[ssbarnea]

That is a problem from devtools point of view because:

• --pre there is no reason to add pre flag to the command. It would require a lot of code to pre-process requirements.yml file to see that it has some git dependencies and add --pre to the command for installing it.
• --no-deps does not make sense either, for similar reasons.

So while I see someone using ansible-galaxy cli as being able to use these workarounds, I do not see how we can change devtools to decide when to add either of these flag when installing test dependencies.

Even if we add configurable options for adding these arguments, we will have to add them in both ansible-lint and molecule and we will be facing users that keep getting installation errors due to not knowing about these hacks.

Anyone having better ideas on how to address this issue?

[bcoca]

So i have code that currently fixes the current bug, but still returns an error about unmet dependencies, current bug raises an error in middle of raising that error. At this point you would have to either fix the dependency issue or enable --pre or --no-deps to go forward. I'm not sure it makes much sense defaulting to those options in galaxy cli itself.

[s-hertel]

I have a PR that would with the general install issue (negating the need for --pre or --no-deps): https://github.com/ansible/ansible/pull/79112/files (At least, I think, testing with the example given here #3686 (comment)). Let me try to dust that off and simplify it.

As far as I remember, that doesn't fix the buggy error handling though.