Ubuntu Wily (15.10) vagrant ssh times out

[lukehoersten]

Summary

Ubuntu Wily vagrant box times out when trying to vagrant ssh into it. Works fine with ubuntu/vivid64 and before.

Versions

• Host: OS X 10.11.2
• Guest: Ubuntu Wily 15.10
• VirtualBox: 5.0.10 r104061
• Vagrant: 1.7.4

Vagrantfile

Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/wily64"
end

vagrant up --debug output available in this pastebin.

Investigation

My suspicion is that in Wily, Ubuntu started defaulting to a network bridge to linux container and this is overriding the necessary vagrant routing.

I've confirmed the VirtualBox port forwarding from Host port 2222 to Guest port 22 is configured correctly.

Vivid routing tables

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    0      0        0 eth0
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0

Wily routing tables

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.3.0        0.0.0.0         255.255.255.0   U     0      0        0 lxcbr0

This prevents the sshd form even seeing the connection attempt.

Other issues with ssh timeout symptoms

This is different than another issue people were seeing with the same symptom of ssh connection timeouts. That issue was specific to 64bit virtualization being disabled at the BIOS level. This is not the issue here because ubuntu/vivid64 box works just fine.

Thanks to ada on IRC for helping debug this issue.

[adamancini]

I can duplicate this issue on OS X 10.11.2, Vagrant 1.7.4, vbox 5.0.10 r104061

[Perni1984]

no time to check this right now, but could it be connected to the following issue on ubuntu/wily64:
https://gist.github.com/brbsix/b70413dec907906ef659

apparently the enumeration of network devices was changed drastically in wily64

[lukehoersten]

@Perni1984 you're brilliant. That was the problem. The fix worked at the bottom of the gist you posted:

To configure the new interfaces

First, determine the name(s) of the new interface(s). You may run ifconfig -a or check dmesg (i.e. dmesg | grep renamed) for a line like this:

[2.671805] e1000 0000:00:03.0 enp0s3: renamed from eth0

In this example, the new interface name is enp0s3.

Once you have determined the new interface name(s), go to /etc/network/interfaces.d. There should be a file named eth0.cfg with the following contents:

# The primary network interface
auto eth0
iface eth0 inet dhcp

Now copy eth0.cfg to enp0s3.cfg (or whatever the name of your interface is). Edit the new file and replace all use of eth0 with enp0s3. Do this for each interface you need configured. Remove eth0.cfg when you are finished.

Finally, restart networking with sudo systemctl restart networking.service.

Any ideas about how to permanently fix this in the Vagrant Wily box?

[Perni1984]

@lukehoersten: glad I could help. Apparently this should be fixed in Vagrant 1.8, see #6653

[Perni1984]

@lukehoersten https://github.com/LukeHoersten: How did you access
your Vagrant machine? I'm having the same issue, but can't get in to
make these changes.

—
Reply to this email directly or view it on GitHub
#6683 (comment).

boot up a vagrant machine with gui:

config.vm.provider "virtualbox" do |v|
v.gui = true
end

[lukehoersten]

Yes that's right. Logged in via the virtualbox GUI console.
On Fri, Dec 18, 2015 at 5:38 PM Christoph Pernsteiner <
notifications@github.com> wrote:

@lukehoersten https://github.com/LukeHoersten: How did you access
your Vagrant machine? I'm having the same issue, but can't get in to
make these changes.

—
Reply to this email directly or view it on GitHub
<#6683 (comment)
.

boot up a vagrant machine with gui:

|config.vm.provider "virtualbox" do |v| v.gui = true end|

—
Reply to this email directly or view it on GitHub
#6683 (comment).

[sgallen]

@lukehoersten and @Perni1984 Thanks!

For some reason only the first option posted by @Perni1984 worked for me. I'm not sure why. My attempt at the second option looked like the following:

Running dmesg | grep renamed I get enp0s3: renamed from eth0. I created /etc/network/interfaces.d/enp0s3.cfg, filling it with:

auto enp0s3
iface enp0s3 inet dhcp

I then ran, sudo ifconfig enp0s3 up. At this point I tried vagrant ssh and received ssh_exchange_identification: read: Connection reset by peer.

[sgallen]

I've created this box using the first suggested fix: https://atlas.hashicorp.com/sgallen/boxes/wily64/

[defreez]

I am being affected by this also. I still have the problem with 1.8. The workaround of adding the correct file to interfaces.d does work.

[james-bright-helix]

I can confirm this issue still occurs with vagrant 1.8 & virtual box 5.0.10.
More specifically this issue only started happening with wily64 v20151127.0.0.

We were previously using v20151120.0.0 with no problems, we use vagrant box update as part of our dev environment refresh script and started getting this issue last month.

We've been running on the version from the 20th since then but would prefer to run up to date.

I notice the issue is considered fixed via #4468 (according to release notes of 1.8) and #6653 but this doesn't appear to be the case for my teammates and I. Perhaps when @sethvargo looked into #6653 he had v20151120.0.0 or earlier an update would reproduce the issue.

Reference to versions: https://atlas.hashicorp.com/ubuntu/boxes/wily64

EDIT: Tried with @sgallen version and it's working ok but would prefer to run on official versions (no offence)

[briandant]

I can further confirm that the issue remains (I'm also using Vagrant 1.8 and Virtualbox 5.0.10). Unlike @james-bright-helix, I can't get this to work, even when using the wily64 v20151120.0.0 box. I did successfully update a running v20151127.0.0 box using the second solution in @brbsix's gist.

EDIT: I can vagrant ssh into this box, but vagrant provision still hangs on the ssh connection (using Ansible). I am, however, using private networking, so @gMagicScott's answer below might explain why I'm having issues and others aren't.

[brbsix]

@sgallen After creating /etc/network/interfaces.d/enp0s3.cfg you have to fully restart networking via a reboot or sudo systemctl restart networking.service. I'm not sure why, but sudo ifconfig enp0s3 up is not sufficient. This might explain your trouble with the second method.

Just a cursory look would seem to indicate the trouble is within https://github.com/mitchellh/vagrant/blob/master/plugins/guests/debian/cap/configure_networks.rb

Unlike the Arch-specific networking code here https://github.com/mitchellh/vagrant/blob/master/plugins/guests/arch/cap/configure_networks.rb, you'll notice interface operations for Debian/Ubuntu boxes are hardcoded eth#{interface}. This would appear incompatible with Predictable Network Interface Names. In summary, #4468 only added Predictable Network Interface Names support for Arch boxes. #6653 appears to have been closed erroneously.

[gMagicScott]

@brbsix #6683 (comment)
Just a cursory look would seem to indicate the trouble is within https://github.com/mitchellh/vagrant/blob/master/plugins/guests/debian/cap/configure_networks.rb

I just submitted a pull request about that (#6724). That setup is for when you add custom networking like config.vm.network "private_network", ip: "192.168.50.4" to your Vagrantfile. The setup of that networking is executed over an internal vagrant ssh. Fixing this initial connection either needs to happen when the box is built initially, or somewhere between Vagrant booting the box and trying to connect by SSH.

[james-bright-helix]

@briandant Someone here had a similar issue to you with that version and when we checked the packages had a different hash. I'm not sure if something changed on the central repo or if someone here fiddled with our cache and it's actually a version earlier that was working.... as it's been Christmas it's not so easy to figure out with everyone off.

[wasadigi]

@sgallen: I have seen your box on atlas and I am trying to apply the suggested fix with this:

GRUB_CMDLINE_LINUX="debian-installer=en_US net.ifnames=0" in my packer script.

Unfortunately, I still get stuck on "Waiting for ssh...". Would you mind sharing your packer template and supporting files?

[lukehoersten]

This is still an problem affecting latest wily images. Any plans on fixing this?

edit: so the ubuntu/wily64 on atlas is fixed but not the vagrant image ubuntu ships from:
https://cloud-images.ubuntu.com/vagrant/

Not sure if that's still a Vagrant or Ubuntu problem at this point then. Maybe we can close the issue.

[james-bright-helix]

@lukehoersten during some combination of upgrading vagrant, virtual box and getting the latest from atlas it started working for us so I don't believe it's a vagrant issue any more.

I haven't tried with the cloud-images version.

[geerlingguy]

Here's a fix I used in building my own minimal Xenial (16.04) base box using Packer: geerlingguy/packer-boxes#1 (comment) — still testing, but it looks like there's a PR and an already-committed-but-not-in-stable-release bugfix needed before this hacky workaround will not be needed.

[sethvargo]

Hi there,

I would recommend y'all try the bento boxes. A common misconception is that a namespace like "ubuntu" represents the canonical space for Ubuntu boxes. This is untrue. Namespaces on Atlas behave very similarly to namespaces on GitHub, for example. Just as GitHub's support team is unable to assist with issues in someone's repository, HashiCorp's support team is unable to assist with third-party published boxes.