Error mounting /tmp/airflowtmp... with remote docker

[christianbrugger]

Apache Airflow version: v2.1.0

Environment:

• Cloud provider or hardware configuration: ec2 t3a.medium
• OS (e.g. from /etc/os-release): Ubuntu 18.04.5 LTS
• Kernel (e.g. uname -a): 5.4.0-1051-aws
• Install tools: sudo pip3 install apache-airflow[mysql,ssh,docker,amazon]
• Others: python 3.6.9

What happened:

Task fails with error:

docker.errors.APIError: 400 Client Error for http://192.168.1.50:2375/v1.41/containers/create: 
Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp7naq_r53")

How to reproduce it:

Create an separate EC2 instance and forward the docker daemon:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo touch /etc/systemd/system/docker.service.d/options.conf
echo -e """
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2375
""" >> /etc/systemd/system/docker.service.d/options.conf
sudo systemctl daemon-reload
sudo systemctl restart docker

Create dag with DockerOperator

DockerOperator(
        task_id="run_image",
        docker_url="tcp://192.168.1.50:2375",
        image="ubuntu:latest",
        dag=dag,
    )

Run the DAG.

Anything else we need to know:

To me it looks like the DockerOperator is creating a temporary directory locally and tries to bind it to the container. However as this is a remote container the directory doesn't exist. here is the code part:

class DockerOperator(BaseOperator):
    ...

    def _run_image(self) -> Optional[str]:
        """Run a Docker container with the provided image"""
        self.log.info('Starting docker container from image %s', self.image)

        with TemporaryDirectory(prefix='airflowtmp', dir=self.host_tmp_dir) as host_tmp_dir:
            if not self.cli:
                raise Exception("The 'cli' should be initialized before!")
            tmp_mount = Mount(self.tmp_dir, host_tmp_dir, "bind")
            self.container = self.cli.create_container(
                command=self.format_command(self.command),
                name=self.container_name,
                environment={**self.environment, **self._private_environment},
                host_config=self.cli.create_host_config(
                    auto_remove=False,
                    mounts=self.mounts + [tmp_mount],
                    network_mode=self.network_mode,
                    shm_size=self.shm_size,
                    dns=self.dns,
                    dns_search=self.dns_search,
                    cpu_shares=int(round(self.cpus * 1024)),
                    mem_limit=self.mem_limit,
                    cap_add=self.cap_add,
                    extra_hosts=self.extra_hosts,
                    privileged=self.privileged,
                ),
                image=self.image,
                user=self.user,
                entrypoint=self.format_command(self.entrypoint),
                working_dir=self.working_dir,
                tty=self.tty,
            )

I see no way of disabling this behavior without some major patching.

How are you guys using remote docker daemons? Is this a use case? Would it be possible to implement something to allow that?

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Seems the same error as #16803 - we will have to take a look at fixing it for remote engine

[christianbrugger]

Yes, the guy in #16803 tries to run a container from within a container, running obviously into the same issue, as the directory created in the container he is calling from is not available on the host running the docker daemon.

[sudohainguyen]

I'm facing the same issue too, actually It can access the mounting point for docker socket, but thrown error when trying to mount tmp dir

[sudohainguyen]

Well, downgrade apache-airflow-providers-docker to 1.1.0 would help.

[potiuk]

I am not sure if downgrade would help. This behaviour was there like forever.

I looked at this and turned the temp file mounting into optional feature that you might disable: see #16932

[sudohainguyen]

actually, I fixed it yesterday by downgrading, you should try

[potiuk]

Ah. OK. I see Indeed. I thin kit was because we have changed how "Mount" works and while it was not failing before, it did not work properly either (the tmp volume was not mounted). I see then how it was not detected before. Good call @sudohainguyen !

[akki]

I don't have a complete proof for this yet (I might have to look into Docker's code for that), but from the docs (and the error message), it seems that we might be using the wrong type.

Is it possible for anyone here to change this line to tmp_mount = Mount(self.tmp_dir, host_tmp_dir, "volume") and see if that fixes the issue?

More details in this comment - #16932 (comment)