default: Warning: Remote connection disconnect. Retrying... #5

junoteam · 2021-12-09T16:15:07Z

Hey, in some reason, it doesn't work anymore.
It fails with this error:

vagrant up
Bringing machine 'default' up with 'docker' provider...
==> default: Creating and configuring docker networks...
==> default: Fixed port collision for 22 => 2222. Now on port 2200.
==> default: Creating the container...
    default:   Name: vagrant-docker-provider-original_default_1639066115
    default:  Image: rofrano/vagrant-provider:ubuntu
    default: Volume: /sys/fs/cgroup:/sys/fs/cgroup:ro
    default: Volume: /Users/user/vagrant-docker-provider-original:/vagrant
    default:   Port: 127.0.0.1:2200:22
    default:
    default: Container created: fb5e8f8218b7cc31
==> default: Enabling network interfaces...
==> default: Starting container...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2200
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
^C==> default: Waiting for cleanup before exiting...

Laptop: MacBook Pro (14-inch, 2021)
OS: macOS Monterey 12.0.1
Docker version: 4.3.0 (71786)

When I log in with exec to container, I see that systemd isn't running.

docker exec -ti vagrant-docker-provider-original_default_1639066406 bash
root@ubuntu:/# ps auxwf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           7  2.0  0.0   3880  3044 pts/0    Ss   16:13   0:00 bash
root          16  0.0  0.0   5472  2312 pts/0    R+   16:13   0:00  \_ ps auxwf
root           1  0.4  0.0  17444  6816 ?        Ss   16:13   0:00 /usr/sbin/init
root@ubuntu:/# systemctl status
Failed to connect to bus: No such file or directory

I am looking for the solution but didn't find yet.
On Linux it works fine:

docker run --tmpfs /tmp --tmpfs /run -it -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8081:80 -p 8080:8080 --name my_image <my_image>

@rofrano

The text was updated successfully, but these errors were encountered:

poolski · 2021-12-10T09:21:34Z

@junoteam You will need to change your invocation for starting the box slightly. It's discussed in this issue.

You'll need to add

docker.create_args = ["--cgroupns=host"]

to your provider config.

rofrano · 2021-12-11T14:53:01Z

@junoteam Did @poolski suggestion work for you? Because it didn't work for me. I'm still on Big Sur but as soon as I upgraded to Docker 4.3 everything broke. Adding --cgroupns=host didn't help.

I'm going back to 4.2 for now. I'll try and investigate more as time permits. This isn't the first time Docker has broken M1 support so I am very reluctant to upgrade to newer versions.

junoteam · 2021-12-11T15:07:28Z

@rofrano @poolski Nope, this solution doesn't work.
Here 's provider block from my Vagrantfile :

  # Provider for Docker
  config.vm.provider :docker do |docker, override|
    override.vm.box = nil
    docker.image = "rofrano/vagrant-provider:ubuntu"
    docker.remains_running = true
    docker.has_ssh = true
    docker.privileged = true
    docker.create_args = ["--cgroupns=host"]
    docker.volumes = ["/sys/fs/cgroup:/sys/fs/cgroup:ro"]
  end

Error is the same.

    default:   Port: 127.0.0.1:2200:22
    default:
    default: Container created: 167ec70eaf213446
==> default: Enabling network interfaces...
==> default: Starting container...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2200
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
^C==> default: Waiting for cleanup before exiting...
Vagrant exited after cleanup due to external interrupt.

The problem is, somewhere in the version of Docker.
I'll continue an investigation too, I need this working on my M1 Mac :)

junoteam · 2021-12-11T15:42:36Z

The same issue open in official docker for-mac repository:
docker/for-mac#6073

junoteam · 2021-12-11T16:11:48Z

@rofrano @poolski

I have found a workaround.
The root of the problem in the Docker Desktop 4.3.0 changelog:

Docker Desktop now uses cgroupv2. If you need to run systemd in a container then:
* Ensure your version of systemd supports cgroupv2. It must be at least systemd 247.
* Containers running systemd need the following options: --privileged --cgroupns=host -v /sys/fs/cgroup:/sys/fs/cgroup:rw.

To make it works, we need to run Ubuntu with at least this version 247 of systemd
Ubuntu 22:04 Jammy Jellyfish has installed this version of systemd:

root@ubuntu:~# systemctl --version
systemd 249 (249.5-2ubuntu1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

I fixed Dockerfile

FROM ubuntu:jammy
LABEL MAINTAINER="John Rofrano <rofrano@gmail.com>"

and Vagrantfile

  # Provider for Docker
  config.vm.provider :docker do |docker, override|
    override.vm.box = nil
    docker.image = "rofrano/vagrant-provider:ubuntu-jammy"
    docker.remains_running = true
    docker.has_ssh = true
    docker.privileged = true
    docker.volumes = ["/sys/fs/cgroup:/sys/fs/cgroup"]
  end

Now, if you wish to run Ubuntu 22:04 with Systemd installed without Vagrant, you need run it like that:

docker run -ti --privileged -v /sys/fs/cgroup:/sys/fs/cgroup rofrano/vagrant-provider:ubuntu-jammy

Now it works.

rofrano · 2021-12-11T17:31:44Z

@junoteam This is not working with Vagrant for me. I updated my Docker image as you suggested and this is what I get when I execute vagrant up:

% vagrant up                                                                                                                (master)
Bringing machine 'default' up with 'docker' provider...
==> default: Creating and configuring docker networks...
==> default: Creating the container...
    default:   Name: vagrant-docker-provider_default_1639243346
    default:  Image: rofrano/vagrant-provider:ubuntu-jammy
    default: Volume: /sys/fs/cgroup:/sys/fs/cgroup:ro
    default: Volume: /Users/rofrano/GitHub/rofrano/vagrant-docker-provider:/vagrant
    default:   Port: 127.0.0.1:2222:22
    default:  
    default: Container created: 381b80e2cf63b122
==> default: Waiting for container to enter "running" state...
The container started either never left the "stopped" state or
very quickly reverted to the "stopped" state. This is usually
because the container didn't execute a command that kept it running,
and usually indicates a misconfiguration.

If you meant for this container to not remain running, please
set the Docker provider configuration "remains_running" to "false":

  config.vm.provider "docker" do |d|
    d.remains_running = false
  end

If I try and run the container detached it does not keep running:

% docker run -it --privileged -v "/sys/fs/cgroup:/sys/fs/cgroup:ro" -d rofrano/vagrant-provider:ubuntu-jammy
b7834a4f1e17e274dbacb6dd4274d14d679e5da31b53b5b5ba80f004a63ef4ac

% docker ps                                                                                                                 (master)
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

However. if I run my debian image the same way it does keep running. Vagrant just can't ssh into it:

% docker run -it --privileged -v "/sys/fs/cgroup:/sys/fs/cgroup:ro" -d rofrano/vagrant-provider:debian
88a2a5a69f48bec382681760f5a6e74bf2e1050853fb4115ed89caced224695b

% docker ps
CONTAINER ID   IMAGE                             COMMAND            CREATED         STATUS        PORTS     NAMES
88a2a5a69f48   rofrano/vagrant-provider:debian   "/usr/sbin/init"   2 seconds ago   Up 1 second   22/tcp    quizzical_khayyam

When you start the ubuntu-jammy image with Vagrant does it keep running for you?

junoteam · 2021-12-12T14:17:09Z

@rofrano

I added a lot of small fixes and changes.
100% working version (on my setup) is here: https://github.com/junoteam/vagrant-docker-m1-macosx/tree/main/vm2-ubuntu22.04-debian

It's actually your repo, but I did customizations for my needs.
Everything is working for me just perfect

junoteam · 2021-12-13T22:05:32Z

BTW, docker just released version 4.3.1 with these notes in the changelog:

Docker Desktop now uses cgroupv2. If you need to run systemd in a container then:
Ensure your version of systemd supports cgroupv2. It must be at least systemd 247.

Consider upgrading any centos:7 images to centos:8.
Containers running systemd need the following options: --privileged --cgroupns=host -v /sys/fs/cgroup:/sys/fs/cgroup:rw

https://docs.docker.com/desktop/mac/release-notes/

valentin-mille · 2021-12-22T07:18:33Z

Hello,
I managed to run the ubuntu:jammy image with docker and vagrant on my m1 MacBook Pro with the x86_64 architecture.
Unfortunately, it runs in 32 bits only, which is frustrating because some packages are not compatible.
Did you successfully run amd64 in 64 bits?

Welcome to Ubuntu Jammy Jellyfish (development branch) (GNU/Linux 5.10.76-linuxkit x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
vagrant@ubuntu:~$ uname -a
Linux ubuntu 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
vagrant@ubuntu:~$ lscpu
Architecture:          x86_64
  CPU op-mode(s):      32-bit
  Byte Order:          Little Endian
CPU(s):                5
  On-line CPU(s) list: 0-4
Vendor ID:             0x00
  Model:               0
  Thread(s) per core:  1
  Core(s) per socket:  5
  Socket(s):           1
  Stepping:            0x0
  BogoMIPS:            48.00
  Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint
Vulnerabilities:
  Itlb multihit:       Not affected
  L1tf:                Not affected
  Mds:                 Not affected
  Meltdown:            Not affected
  Spec store bypass:   Vulnerable
  Spectre v1:          Mitigation; __user pointer sanitization
  Spectre v2:          Not affected
  Srbds:               Not affected
  Tsx async abort:     Not affected

rofrano · 2021-12-26T17:23:23Z

@junoteam Thanks for that update. Your files worked for me on my Intel Mac as well. I need to test on my M1 Mac and incorporate the changes that make ssh work again into my repo.

rofrano · 2021-12-27T22:45:01Z

@junoteam & @poolski , I have it all working now. I had to make the following changes to be compatible with updates that Docker Desktop 4.3 made to systemd support which now requires systemd 247 or greater:

Updated the Ubuntu image to 21.10 for systemd version 249
Updated the Debian image to 11 for systemd version 247
Changed Vagrantfile to mount /sys/fs/cgroup read-write
Added --cgroupns=host docker argument to run Docker in Docker

A key thing that I missed at first was changing the /sys/fs/cgroup mount to read write in my Vagrantfile. I couldn't understand why yours worked and mine didn't but when I re-read your posts I finally picked up on it.

If this fixes all of your problems let me know and I'll close this issue. It's working great on my M1 Mac.

A HUGE THANKS for your support in helping me figure this out!

junoteam · 2021-12-28T16:12:07Z

Thank you @rofrano
let's close the issue. It was solved for me too!

Small suggestion from my side.
I think it's better to use lts version of ubuntu, like 22.04 and not 21.10

but this is only small suggestion :)

rofrano · 2021-12-28T18:52:03Z

@junoteam I tend to agree with you on using an LTS as the default ubuntu image. I've changed it to 21.04 because 22.04 doesn't install Docker automatically with vagrant yet (it fails with a Release not found error). Thanks for your input.

rofrano · 2021-12-29T00:00:55Z

Issue seems to be resolved with the latest changes. Closing.

See rofrano/vagrant-docker-provider#5 (comment)

junoteam mentioned this issue Dec 11, 2021

Unable to run systemd services on Docker Desktop 4.3.0 docker/for-mac#6073

Closed

3 tasks

rofrano closed this as completed Dec 29, 2021

RomanShestakov mentioned this issue Aug 6, 2023

once again :default: Warning: Remote connection disconnect. Retrying... with with Ubuntu 23.04 and latest Docker 4.22 #20

Open

Ndpnt added a commit to openfisca/openfisca-ops that referenced this issue May 9, 2024

Update Vagrantfile for Docker Desktop >= 4.3.0 compatibility

2399bba

See rofrano/vagrant-docker-provider#5 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

default: Warning: Remote connection disconnect. Retrying... #5

default: Warning: Remote connection disconnect. Retrying... #5

junoteam commented Dec 9, 2021 •

edited

Loading

poolski commented Dec 10, 2021

rofrano commented Dec 11, 2021

junoteam commented Dec 11, 2021 •

edited

Loading

junoteam commented Dec 11, 2021

junoteam commented Dec 11, 2021

rofrano commented Dec 11, 2021

junoteam commented Dec 12, 2021

junoteam commented Dec 13, 2021 •

edited

Loading

valentin-mille commented Dec 22, 2021

rofrano commented Dec 26, 2021

rofrano commented Dec 27, 2021

junoteam commented Dec 28, 2021 •

edited

Loading

rofrano commented Dec 28, 2021

rofrano commented Dec 29, 2021

default: Warning: Remote connection disconnect. Retrying... #5

default: Warning: Remote connection disconnect. Retrying... #5

Comments

junoteam commented Dec 9, 2021 • edited Loading

poolski commented Dec 10, 2021

rofrano commented Dec 11, 2021

junoteam commented Dec 11, 2021 • edited Loading

junoteam commented Dec 11, 2021

junoteam commented Dec 11, 2021

rofrano commented Dec 11, 2021

junoteam commented Dec 12, 2021

junoteam commented Dec 13, 2021 • edited Loading

valentin-mille commented Dec 22, 2021

rofrano commented Dec 26, 2021

rofrano commented Dec 27, 2021

junoteam commented Dec 28, 2021 • edited Loading

rofrano commented Dec 28, 2021

rofrano commented Dec 29, 2021

junoteam commented Dec 9, 2021 •

edited

Loading

junoteam commented Dec 11, 2021 •

edited

Loading

junoteam commented Dec 13, 2021 •

edited

Loading

junoteam commented Dec 28, 2021 •

edited

Loading