Runner ignores DOCKER_HOST variable when starting container

[ghost]

Describe the bug
I tried to run a self hosted github runner with rootless docker. I added

XDG_RUNTIME_DIR=/home/runner/.docker/run
DOCKER_HOST=unix:///home/runner/.docker/run/docker.sock

to .env and /home/runner/bin to .path. In general github actions and rootless docker work fine.

When the container is initialized it runs

/home/runner/bin/docker create --name 395a5f555e924b28bf52524b85bb3eee_docker1903dind_710b8c --label d94681 --workdir /__w/actionstest/actionstest --network github_network_1c96635676ce470c8b9cb5f86ec01f5c
  -e "HOME=/github/home" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/actions-runner/_work":"/__w" -v "/home/runner/actions-runner/externals":"/__e":ro -v "/home/runner/actions-runner/_work/_temp":"/__w/_temp" -v "/home/runner/actions-runner/_work/_actions":"/__w/_actions"
 -v "/home/runner/actions-runner/_work/_tool":"/__w/_tool" -v "/home/runner/actions-runner/_work/_temp/_github_home":"/github/home" -v "/home/runner/actions-runner/_work/_temp/_github_workflow":"/github/workflow" --entrypoint "tail" docker:19.03-dind "-f" "/dev/null"

As you can see it tries to volume mount the non existing /var/run/docker.sock from the host, which prevents any dind image from working. It seems this value is hardcoded here and DOCKER_HOST is ignored.

To Reproduce
Steps to reproduce the behavior:

1. Create a self hosted runner using rootless docker
2. Start a build with a step running inside a container

Expected behavior
/var/run/docker.sock should not be hard coded. DOCKER_HOST should be used if set.

Runner Version and Platform

Version of your runner? 2.274.2

OS of the machine running the runner? Linux, Ubuntu 18.04

Job Log Output

Runner starts the build step with

/home/runner/bin/docker create -[...] -v "/var/run/docker.sock":"/var/run/docker.sock" [...]

which makes this job

  dind-test:
    container: docker:19.03-dind
    runs-on: self-hosted
    steps:
      - run: docker ps

fail:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[npalm]

I have the runner working with rootless docker, check the example / ubuntu dir https://github.com/philips-labs/terraform-aws-github-runner

[ghost]

I have the runner working with rootless docker, check the example / ubuntu dir https://github.com/philips-labs/terraform-aws-github-runner

That terraform module is pure gold. 🥇

But im utterly confused. You did the very same I did. Essentially this line from your exmaple

echo DOCKER_HOST=unix:///run/user/$USER_ID/docker.sock >>.env

is what I think is completly ignored by the runner when passing the sock file into the container. Yes, github does start builds in docker containers, but I believe you still can not run builds using the dind images, which rely on the proper mounting of the sock file.

Can you confirm that your setup is actually running the dind-test-job above without problems?

[Frederik-Baetens]

I also tried following @npalm's example, I added the path & docker_host exports to my .bashrc, my .profile and my .env and .path in the actions-runner directory. None of this seems to get the actions runner to respect those env vars.

[Frederik-Baetens]

Turns out the problem was that I put "export DOCKER_HOST" (the notation in .profile and .bashrc) into the .env file of my actions-runner. Removing the export fixed the docker error.

For clarity, my .env now looks like this:

LANG=C.UTF-8
DOCKER_HOST=unix:///run/user/1000/docker.sock

(The lang was added by the runner setup itself)

my .profile and .bashrc ends with:

export PATH=/home/frederik/bin:$PATH
export DOCKER_HOST=unix:///run/user/1000/docker.sock

So I think that the runner does respect the DOCKER_HOST variable.

[ghost]

@Frederik-Baetens Can you successfully run this job

name: dindtestjob
on: [push]
jobs:
  dind-test:
    container: docker:19.03-dind
    runs-on: self-hosted
    steps:
      - run: docker ps

anf if so, could you provide the output of the docker create command during startup (from the github workflow output, step "Initialize cotainers -> Starting job container")?

[Frederik-Baetens]

Run docker ps
  docker ps
  shell: sh -e {0}
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Error: Process completed with exit code 1.

So it doesn't work, but what do you need dind for when you can access docker on the host? Dind is useful for when you need to run docker inside of another docker container because of the isolation/security model such as with gitlab's runner model where everything is forced to run in a container. But why do you need that for github actions? If you absolutely want to build in a docker container for some reason, just use kaniko.

I wrote a more in depth stackoverflow post about how I set this up: https://stackoverflow.com/questions/66137419/how-to-enable-non-docker-actions-to-access-docker-created-files-on-my-self-hoste/66137420#66137420

This is probably also relevant to rootless dind: https://docs.docker.com/engine/security/rootless/#rootless-docker-in-docker

Using that container version seems to get dind to use the right socket address for docker, but it gets a permission denied because of the lack of privileged, and I don't know if it's easy to add that privileged flag somehow. But again, I don't see why you need dind.