[docker] docker-py naming conflict since 2.0.2 release

[dminca]

Since the release of docker v2.0.2 both docker-py and docker libs cannot reside together as it'll throw a naming conflict error.

This change has already been applied in docker/docker-py#1399 in order to fix docker/docker-py#1395 docker/docker-py#1370 and last but not least docker/compose#4344 , thus docker-compose v1.10.1+ will require the docker lib instead of the docker-py one.

ISSUE TYPE

• Bug Report
• Documentation Report

COMPONENT NAME

• docker_container
• docker_image
• docker_image_facts
• docker_login
• docker_network

ANSIBLE VERSION

ansible 2.2.1.0                                        
  config file = /etc/ansible/ansible.cfg               
  configured module search path = Default w/o overrides

CONFIGURATION

OS / ENVIRONMENT

"N/A"

SUMMARY

Possible naming conflict on the docker-py package since version 2.0.2 will prevent user from keeping both docker-py and the new docker libs.

STEPS TO REPRODUCE

• install the docker suite:
  • docker-engine v1.13.1
  • docker-compose v1.11.2
• install docker via pip module
• via the shell or command module, run a docker-compose up on a docker-compose.yml
• run a task via docker_container module

- name: deploy master node via docker-compose                              
  environment:                                                             
    RANCHER_SRV_MASTER: "{{ rancher_server.master }}"                      
  shell: "docker-compose -f docker-compose.yaml -f master.yaml up -d"
  args:                                                                    
    chdir: /opt/rancher/rancher-stack                                      
  when: "inventory_hostname == rancher_server.master"

- name: return registration tokens of Rancher servers                                               
  uri:                                                                                              
    method: POST                                                                                    
    status_code: 201                                                                                
    url: "http://{{ rancher_server.master }}:{{ rancher_server.port }}/v1/registrationtokens?projectId={{ project_ids.stdout_lines[0] }}"                                                               
    return_content: true                                                                            
  register: rancher_token_url                                                                       
                                                                                                    
- name: get registration URLs from Rancher servers                                                  
  uri:                                                                                              
    method: GET                                                                                     
    url: "{{ rancher_token_url.json['links']['self'] }}"                                            
    return_content: true                                                                            
  register: rancher_token                                                                           


- name: register Rancher Agents with Rancher Servers      
  become: true                                            
  become_method: sudo                                     
  docker_container:                                       
    name: "{{ rancher_agent.name }}"                      
    image: "rancher/agent:{{ rancher_agent.version }}"    
    state: started                                        
    privileged: true                                      
    detach: true                                          
    cleanup: true                                         
    env:                                                  
      CATTLE_AGENT_IP: "{{ rancher_server.master }}"      
    volumes:·                                             
      - /var/run/docker.sock:/var/run/docker.sock         
      - /var/lib/rancher:/var/lib/rancher                 
    command: "{{ rancher_token.json['registrationUrl'] }}"

EXPECTED RESULTS

• run the docker-compose.yml via the command or shell module to lift up infrastructure
• successfully start the rancher-agent container via the docker_container module

ACTUAL RESULTS

• ansible will throw: Failed to import docker-py - No module named docker. Try "pip install docker-py"
  • docker and docker-py cannot reside together since docker v2.0.2
  • docker-compose will not function with docker-py since compose v1.10.1

failed: [web-node001.org] (item=4a4f54604e9d) => {
    "failed": true, 
    "invocation": {
        "module_args": {
            "api_version": null, 
            "blkio_weight": null, 
            "cacert_path": null, 
            "capabilities": null, 
            "cert_path": null, 
            "cleanup": false, 
            "command": null, 
            "cpu_period": null, 
            "cpu_quota": null, 
            "cpu_shares": null, 
            "cpuset_cpus": null, 
            "cpuset_mems": null, 
            "debug": false, 
            "detach": true, 
            "devices": null, 
            "dns_opts": null, 
            "dns_search_domains": null, 
            "dns_servers": null, 
            "docker_host": null, 
            "entrypoint": null, 
            "env": null, 
            "env_file": null, 
            "etc_hosts": null, 
            "exposed_ports": null, 
            "filter_logger": false, 
            "force_kill": true, 
            "groups": null, 
            "hostname": null, 
            "ignore_image": false, 
            "image": null, 
            "interactive": false, 
            "ipc_mode": null, 
            "keep_volumes": false, 
            "kernel_memory": null, 
            "key_path": null, 
            "kill_signal": null, 
            "labels": null, 
            "links": null, 
            "log_driver": null, 
            "log_options": null, 
            "mac_address": null, 
            "memory": "0", 
            "memory_reservation": null, 
            "memory_swap": null, 
            "memory_swappiness": null, 
            "name": "4a4f54604e9d", 
            "network_mode": null, 
            "networks": null, 
            "oom_killer": null, 
            "oom_score_adj": null, 
            "paused": false, 
            "pid_mode": null, 
            "privileged": false, 
            "published_ports": null, 
            "pull": false, 
            "purge_networks": false, 
            "read_only": false, 
            "recreate": false, 
            "restart": false, 
            "restart_policy": null, 
            "restart_retries": null, 
            "security_opts": null, 
            "shm_size": null, 
            "ssl_version": null, 
            "state": "absent", 
            "stop_signal": null, 
            "stop_timeout": null, 
            "timeout": null, 
            "tls": null, 
            "tls_hostname": null, 
            "tls_verify": null, 
            "trust_image_content": false, 
            "tty": false, 
            "ulimits": null, 
            "user": null, 
            "uts": null, 
            "volume_driver": null, 
            "volumes": null, 
            "volumes_from": null
        }, 
        "module_name": "docker_container"
    }, 
    "item": "4a4f54604e9d", 
    "msg": "Failed to import docker-py - No module named docker. Try `pip install docker-py`"

[alpeshspatel]

Hi,
I have issue in run() as well as pull():

  client = docker.DockerClient(base_url='tcp://172.17.0.1:2375', version='auto')

  container = client.containers.run(image='alpine:latest', command='echo hello world!')
  #client.images.pull('alpine:latest')

    container = client.containers.run(image='alpine:latest', command='echo hello world!')
  File "/usr/lib/python2.7/site-packages/docker/models/containers.py", line 655, in run
    self.client.images.pull(image)
  File "/usr/lib/python2.7/site-packages/docker/models/images.py", line 256, in pull
    self.client.api.pull(name, **kwargs)
  File "/usr/lib/python2.7/site-packages/docker/api/image.py", line 333, in pull
    header = auth.get_config_header(self, registry)
AttributeError: 'module' object has no attribute 'get_config_header'

I have "pip install docker". There is no docker-py
How to go about fixing this?
thx
-a

[dminca]

@alpeshspatel if you have previously installed docker-py, just :

• pip uninstall docker-py then
• pip install docker

The code needs to be updated to reference the new Python Docker library name.

[dminca]

Aborting. No time.

[jcberthon]

Can someone explain why this issue has been closed?

I'm running Ansible 2.3.1.0 on Ubuntu 16.04. I have installed the package python-docker (version 1.9), but when running a playbook with Docker, I get the error: Failed to import docker-py - No module named requests.exceptions. Try 'pip install docker-py'.

I tried to remove the python-docker package and do pip install docker-py. it installed version 1.10.6. But I still get the same error. Then I did uninstall that package (using uninstall instead of install) and installed docker with pip install docker. I got version 2.3.0. Bit I still have the same problem.

So at the moment, I have no ways to use the Docker modules in Ansible!

[ansibot]

cc @ThomasSteinbach @cove @dusdanig @jctanner @joshuaconner @kassiansun @softzilla @zfil
click here for bot help

[kassiansun]

@jcberthon Are you still encountering this problem? ansible 2.3.0 added detection with docker-py >= 2, make sure you have the correct python installation with correct version of docker-py and python-docker.

[sigmavirus24]

@kassiansun I just ran into this using docker 2.0.1 (and docker-py 1.10.6) on python 3.5 and ansible 2.3.1.0

[kassiansun]

@sigmavirus24 Well, from docker-compose's requirements file, now it depends on docker-py 2.4.2, and this should work well with docker_container(on my pc it runs smoothly).
So I didn't get why this is happening, what are the versions of you guys' docker-compse, docker-py, python-docker? I'm working with all latest version, with no error.

[sigmavirus24]

So I don't even have docker-compose installed. Literally did pip install ansible docker in a virtualenv on macOS (and homebrew python3) and I'm getting the

 AttributeError: 'module' object has no attribute 'get_config_header'

exception with

- name: "Repro for 22993"
  hosts: localhost
  tasks:
  - name: "Pull hello-world"
    docker_image:
       name: "hello-world"

[kassiansun]

@sigmavirus24 What's the version of docker-py?

[kassiansun]

Actually I don't think ansible can run with pip(3) install, ansible 2.3.0 is the offical version but it can't run with python3(at least not officially supported)

[sigmavirus24]

Yeah, I run ansible on Python 3 enough that I don't think it's that.

I started off with docker-py 1.10.6 (as above) and when I encountered the same AttributeError I tried docker instead but got the same error. I even recreated the virtualenv from scratch and still get this error.

[kassiansun]

If you want to run ansible with python3, the git version is better(v2.4 is working on python3 support), and I'm running it on my python3 environment.
docker-py runs on your target machine, not local, I have not much experience with virtualenv, this should be setup before you run ansible on remote.
On you local machine, pip install ansible is all you need to do. Then install docker-py or any other dependencies on target machine. ansible only need itself to run on local machine, it copies the modules to remote, and the modules runs on remote might need some other dependencies(like docker-py).

[jcberthon]

Hi @kassiansun

I forgot to update my comment. I found the problem. When I read the document it was written to install docker-py for dependency on the host that executes the module. That sentence was not clear to me and I installed docker-py on the host that executes the playbook (where I execute ansible-playbook ...). Of course that did not work.

Now I understand that "executes the module" refer to the machine where the ansible module is actually executed, so those are the hosts defined in the playbook. After I added a pre-step which install on the remote hosts the docker-py dependency, then it worked.

I guess the documentation could be less ambiguous. When you read it twice, you think it is on the host where you execute your playbook. After banging your head against a wall (it does help putting back some neurone in place ;-) ) and reading it a 3rd time, then you understand that it could be meant differently. So you try and you realise your mistake.

I would suggest updating the documentation as follow:

Requirements

Those are applicable for all hosts (remote or local) where this module will be executed.
<list of req>

[kassiansun]

@jcberthon Actually I think this is the source of this kind of problems, many people don't know how ansible works - copy the module to remote and execute it. And the docker-py/python-docker problem distracts people, make it even harder to locate what's happening.

Hope when people search for missing module problem, they will be directed to here.

[szarlatan]

I found this problem today, in a bit weird way, with ansible 2.3.1.0

With given tasks:

# Install Docker
- name: Install Docker
  apt: name={{ docker_package }} state=latest update-cache=yes

# Install docker-py
- name: Install docker-py
  pip: name={{ docker_py_package }} state=latest

the results are:

kbiszta@ubuntu-xenial:~$ docker --version
Docker version 17.05.0-ce, build 89658be

kbiszta@ubuntu-xenial:~$ pip show docker-py
Name: docker-py
Version: 1.10.6
Summary: Python client for Docker.
Home-page: https://github.com/docker/docker-py/
Author: Joffrey F
Author-email: joffrey@docker.com
License: UNKNOWN
Location: /usr/local/lib/python2.7/dist-packages
Requires: backports.ssl-match-hostname, six, websocket-client, ipaddress, docker-pycreds, requests

So, everything seems to be fine, but when next I run the task:

# Get gitlab-ci-runner images

- name: Get the gitlab-ci-runner docker images
  docker_image:
    name: 'gitlab/gitlab-runner'
    tag: '{{ item }}'
    state: present
  with_items:
    - 'latest'
    - 'alpine'

I get the "msg": "Error pulling image gitlab/gitlab-runner:latest - 'module' object has no attribute 'get_config_header'". I read on web that uninstalling docker-pyhelps to solve this issue, but after removing it I got "msg": "Failed to import docker-py - No module named docker. Try 'pip install docker-py'". So, after installing it back again, everything went fine, what made me confused.

Any ideas why reinstalling the docker-py with exactly the same version of the module made it work properly?

Best,
Kris.

[maxux]

@szarlatan I had the same issue, comparing another environment I have, doing:

pip3 uninstall docker
pip3 install docker==2.1.0

fixes the issue. I can now use again the module. Maybe a newer version works, I don't know.

[ansibot]

cc @chouseknecht
click here for bot help

[ansibot]

cc @keitwb @olsaki
click here for bot help

[abadger]

Has this been fixed for Ansible 2.5.0? I'm not too familiar with the docker APIs but I think that this PR, released in 2.5.0 may have fixed it: #36973 at least for many of the docker modules.

needs_info

[sivel]

@abadger no, it hasn't. Unfortunately docker and docker-py occupy the same namespace (docker), and depending on install order they will overwrite each others files.

I was just thinking about this earlier if there was a way we could detect it, and not sure that is possible, as it can manifest as an import error in some situations.

Ultimately, you cannot have both docker and docker-py installed together if you want a functional library.

Also, at this point docker-py is deprecated, and no one should be using it going forward.

[abadger]

<nod> but I thought the pr I linked to made it so docker-py did not need to be installed anymore.

…

[sivel]

@abadger yes, that is correct. You can have either docker or docker-py installed to use our modules. docker is preferred at this point.

However, you cannot have both python libraries installed at the same time.

[ansibot]

@dminca This issue is waiting for your response. Please respond or the issue will be closed.

click here for bot help

[dminca]

Since docker-py is now legacy, no issues are reported so far, although there are still traces of docker-py in the Ansible codebase. Should we replace them with the latest docker lib?

[sivel]

@dminca there is really no need. On another note, and I failed to come back to this issue, 2.6 (devel) and 2.5.3 will include extra protection around preventing docker and docker-py from being installed side by side.

#38884

If you have further questions please stop by IRC or the mailing list:

• IRC: #ansible on irc.freenode.net
• mailing list: https://groups.google.com/forum/#!forum/ansible-project

[huiminzeng]

Hi！@jcberthon i got the same problem as yours. But i just have merely one machine (ubuntu 16.04), so i don't know how to solve it. Do you have any idea?
Thank you!
Huimin

[jcberthon]

Hi @huiminzeng

Sorry for the long delay, I was away most of May and did not catch with all the emails notifications.

If you have only one machine, then do you something like delegate_to: 127.0.0.1 or equivalent? Or do you connect to a VM on the same machine?

Basically, you need to install docker-py on the "destination" machine where you want your want your Docker commands to be executed.

[tallandtree]

I ran into this same issue as well and tried to remove the docker python modules (with pip uninstall), but it still failed. Eventually I found out that ansible was using the pip2 environment, where both docker and docker-py was installed and I had to do pip2 uninstall and then re-install again.

So, make sure to check both pip and pip2 if you run into this issue.