Ansible: cannot connect to the LXD server (due to python version) #79473

[omani]

Summary

the lxd inventory plugin (and also molecule with lxd but that is another topic) does not work with python > 3.9.

python 3.10 breaks the LXD module due to SSL verification errors.

Issue Type

Bug Report

Component Name

lxd inventory plugin (and also molecule_lxd, but that is another topic).

Ansible Version

$ ansible --version
ansible [core 2.11.11]
  config file = None
  configured module search path = ['/home/sun/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/sun/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/sun/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/sun/.local/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general

Collection        Version
----------------- -------
community.general 6.0.1


Collection        Version
----------------- -------
community.general 3.8.1

Configuration

$ ansible-config dump --only-changed
ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR) = True

OS / Environment

any OS with python 3.10

Steps to Reproduce

start a new LXD container with alpine 3.16.

lxc launch images:alpine/3.16 a1

install python3 (will install python 3.10):

lxc shell a1
apk update
apk add python3

run a simple lxd inventory plugin or molecule, or anything that uses lxd connection plugin and it will throw an error that it cannot connect to the remote LXD server.

but now do the above steps with alpine 3.15 (which only has python 3.9.15) and everything will work.

the inventory plugin can be used like this. create a lxd.yml in your inventory folder:

plugin: community.general.lxd
url: https://LXDREMOTEIP_HERE:8443
type_filter: both

Expected Results

no error

Actual Results

LXDRaiseException("cannot connect to LXD server").

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[omani]

unfortunately, it's been months I debugged this. otherwise I would smash you guys with code blocks and the solution to this. I know this is a poor bug report.

anyway, iirc it had something to do with the change of the ssl context in python 3.10. and you had to use ssl._create_default_https_context = ssl._create_unverified_context to make it work or something related to that. cant remember anymore.

but it was a really easy fix for me back then. basically, python 3.10 wants to sslverify the remote LXD's cert, and can't (since it is a self-signed cert), then gives up and throws an exception.

anyway, Im using alpine/3.15 for my ansible environments since then and have no problem with this bug anymore.

but since I didnt see anyone mention this bug that it is actually due to python 3.10, I thought I should mention it for future people coming from google etc.

so, you have any "cannot connect to LXD server", no matter if you're using the lxd inventory plugin or molecule with the molecule_lxd module? just get rid of python 3.10 and install python 3.9.15 and that will fix the problem.

[felixfontein]

!component =plugins/inventory/lxd.py

[ansibullbot]

Files identified in the description:

• plugins/inventory/lxd.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @conloos
click here for bot help

[felixfontein]

I don't see any code that disables certificate validation in the existing code. So why do you need to disable certificate validation (which is something really dangerous!) to get this working? It sounds to me like there's another underlying problem you are working around by simply disabling certificate validation.

[omani]

I don't see any code that disables certificate validation in the existing code. So why do you need to disable certificate validation (which is something really dangerous!) to get this working? It sounds to me like there's another underlying problem you are working around by simply disabling certificate validation.

because - like I said - it is a private cert. especially when working with LXD. lxd generates a self-signed cert. and the python lib (whatever is used, Im not into python really), I guess it is the ssl lib, doesnt like self-signed certs.

I disabled the ssl_verify back then, because of the self-signed cert. not because I didnt want to verify a cert in general. it is the very special case of a self-signed cert verification that fails. I didnt know any other way around back then, so I just disabled it completely.

[felixfontein]

I don't see why this should change between Python 3.9 and Python 3.10. Self-signed certificates should never have been accepted.

[omani]

I dont know. see this bug for example: https://bugs.gentoo.org/show_bug.cgi?id=835498

and see the commit that fixed that issue. it has something to do with ssl.create_default_context().

[omani]

This works just fine with Python3.9 and below. But with Python3.10 I get...

he discovered the same thing like me.

[omani]

here is another example, with a similar solution:
https://stackoverflow.com/questions/35569042/ssl-certificate-verify-failed-with-python3/69724616#69724616

the web is full with questions regarding this. and all relates to python 3.10. everything < 3.10 works just fine.

[conloos]

Hello @ALL,

first thanks to: @omani for sharing that informations.

I took a look and I used code/utils at the inventory Plugin, which probably goes back to the early days of lxd implementations.
The code for opening the lxd-connections and certificate management is under:
community.general/module_utils/lxd.py

If I see it correctly these functions are used in a lot of modules:

• inventory/lxd.py
• modules/lxd_container.py
• modules/lxd_profile.py
• modules/lxd_project.py

So i will go deeper to fix that problem but i have to build a testlab first.

Thanks Frank

[fkuep]

Self-signed certificates should never have been accepted.

The default location of the client certificate implies the authentication flow described in lxd docu .
Following that method , all certificates in .config/lxc/servercerts/ could be trusted.

IMHO a lot better than explaining users, that lxd default authentication just is not secure and they should acknowledge that by setting validate_certs=false and feel guilty about it.

This approach to fixing the bug suggests the enhancement server_cert (in a 2nd PR perhaps?).

• defaults to url if https .

@heino You probably invested significant effort in the pyLXD / validate_certs=false route. Do You see a place for that approach in Your work ? From a first sight on read-the-docs it looks like manageable effort:

client = Client(
endpoint='http://10.0.0.1:8443',
cert=('/path/to/client.crt', '/path/to/client.key'),
verify='/path/to/server.crt')

[Spunge]

I just ran into a connection issue with a LXD using a SSL certificate issued by letsencrypt. Got the same output as reported in this issue:

LXDRaiseException("cannot connect to LXD server").

When i looked at the actual error it was:

Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:801)

After some digging i found python/cpython#96972 and this line.

After changing

            ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)

to

            ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)

My connection is working just fine.

I'm using Python 3.10.9

[felixfontein]

Yes, it sounds like Purpose.SERVER_AUTH (the actual default value) is the correct value.

[felixfontein]

I created a PR for @Spunge's solution: #6034.

[lictw]

Hello! What's the future of this? LXD uses self-signed server certificates by default and trusts them via password or token, this plugin must support by-pass certificate validation or work with .config/lxc/servercerts to be complied with that default behaviour.

And even better - ability to connect to LXD just by specifying its remote name from local LXC, often it already has all required information (url, certificate, the server also already know our certs).

[felixfontein]

I don't think anyone so far worked on adding support for a validate_certs parameter, or a way to use the server certificates respectively to allow to point to them in another parameter.