Add support for serial consoles #436

Bischoff · 2018-10-10T15:30:47Z

(extracted from #432 from @MalloZup )

This PR adds a serial console to our VMs. It enables:

to use virsh console to connect to a VM, especially when network is unavailable
to avoid Ubuntu to run at 100% CPU and become unusable

MalloZup

as i did it i am not against it 😄 this should be ok for all host afaik

moio

Please test at least once with CentOS, fix the travis build and merge.

Thanks a lot, I can't count the number of times this can save us time!

Bischoff · 2018-10-10T16:16:32Z

There are two ways to do this, through a kernel parameter or through a systemd service.

For the moment i'm doing it the easy way, through the service, but on the long term the kernel parameter approach could give us access to the VM earlier during the boot process.

Yes, I will test CentOS as well and consider what Travis has to say.

MalloZup · 2018-10-10T16:22:51Z

https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/website/docs/r/domain.html.markdown#kernel-and-boot-arguments

Bischoff · 2018-10-10T16:38:45Z

To anyone interested, the magic kernel parameter is console=ttyS0,9600.

Bischoff · 2018-10-10T17:10:56Z

I don't know why, the deployment fails with:

* libvirt_domain.domain: timeout while waiting for state to become
  'all-addresses-obtained' (last state: 'waiting-addresses', timeout: 5m0s)
* module.client.module.client.libvirt_domain.domain: 1 error(s) occurred:

is this linked to some other change of you, or do I miss something, @MalloZup? Does it ring a bell?

MalloZup · 2018-10-11T07:15:55Z

That failure is because sumaform has the variabile wait_for_lease enabled true. This variable basicaly make a for loop over 5 min and wait for ip of the domain(in bridge or not bridge network ) .this is the failure. So in your case the client had pb getting the ip. Can you reproduce it always ? In maybe your client, the os. Image has pb with that kernel arg.it can be the diff img

Bischoff · 2018-10-11T10:57:45Z

@MalloZup I'm not even using the kernel arg, I'm trying with systemd.

I'll try to run without this patch and see if I get an IP. It could be sumaform is trying to use the system console that is now on another tty. @moio, does that hypothesis sound realistic?

I suppose I just have the standard images. I use a fresh sumaform and my usual main.tf.

moio · 2018-10-11T11:02:02Z

terraform-provider-libvirt gets the VM's IP address(es) via the QEMU guest agent (typically after they have been assigned by the DHCP). I have just checked and yes, the communication with the guest agent uses a serial console, so that sounds like the culprit

https://wiki.qemu.org/Features/GuestAgent

Bischoff · 2018-10-11T11:13:57Z

Thanks @moio, that's probably the explanation.

Did you get same problem in your own PR @MalloZup ? You define more "terminal"s but I tried to do the same, and it does not solve the problem. Are you doing some other magic I missed?

Bischoff · 2018-10-11T11:16:21Z

I might have the solution. The problem might be that qemu-agent is detecting the serial console and tries to use it, but there's no login on it yet.

I will use the kernel parameter approach. With that one, you get a longin prompt very early.

MalloZup · 2018-10-11T11:18:18Z

@Bischoff i think i tested only with the ubuntu and there was no problem i could log in with the qemu-agent
It might be specific issues with SLES images used by sumaform or SLES maybe generic

MalloZup · 2018-10-11T11:19:43Z

but take care that the ubuntu tf has other consoles, this would be your problem imho ( i would try it ou)
https://github.com/moio/sumaform/pull/432/files#diff-299bef565123dc0baa9415daf859349fR72

Bischoff · 2018-10-11T11:24:51Z

As i said, I already tried it out with the other consoles, and it did not help.

In this PR I address only the serial console feature, but feel free to add other consoles to solve ubuntu problems.

Bischoff · 2018-10-11T14:01:10Z

There is some progress: I can now see the kernel booting in my virsh console.

However the boot is broken, I probably need some more kernel arguments 😸 .

Bischoff · 2018-10-11T16:10:04Z

It does not work, I was not able to make "Direct kernel boot" feature work.

However, I don't think that's the good strategy. I realized that the good strategy is to add console=ttyS0,115200 inside grub configuration, on the image.

Which means image rebuilding. Given my past experience with that, I will stop at this point.

MalloZup · 2018-10-11T16:19:47Z

@Bischoff what is the 2nd parameter mean?

Bischoff · 2018-10-12T09:23:16Z

@MalloZup The baud rate. This was at the origin on serial telephone lines. As this is now virtualized, you can take the highest speed.

Any will to make images, Dario ⛑️ ?

MalloZup · 2018-10-12T09:28:12Z

@Bischoff i am still on Ubunt patch waiting on review there, i might help once i have finised.

Is really making images the only solution ? 📛

moio · 2018-10-12T09:36:41Z

@Bischoff JFYI kernel parameters can be specified in Terraform, as terraform-provider-libvirt supports them IIUC

https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/website/docs/r/domain.html.markdown

search for "params"

Bischoff · 2018-10-12T09:51:09Z

@moio as you might see above, dario already gave the pointer, it's what I tried and it does not work. You will see the corresponding code in this PR.

Furthermore, it's bad because:

it uses the "external kernel" feature of libvirt that I kind of dislike
(I'm not comfortable bypassing grub)
you can't just add a parameter, you have to specify all of kernel, initrd, and all boot parameters
(as a side effect, you might need to distinguish SLES / CentOS / etc.)

The correct solution is really to put it into the image.

moio · 2018-10-16T05:50:30Z

The correct solution is really to put it into the image.

OK. Do we want to keep this PR open then?

This sounds like we want to change all images, and at that point, I wonder if it would make sense to switch to JeOS/upstream + cloud init for most of them.

MalloZup · 2018-10-16T06:18:39Z

The switch to jeos will take more time. E.g JeOs need to be resized root partition is to small. I would do first ubuntu the jeos

Bischoff · 2018-10-16T11:46:34Z

OK. Do we want to keep this PR open then?

This sounds like we want to change all images, and at that point, I wonder if it would make sense
to switch to JeOS/upstream + cloud init for most of them.

It's independant problems, no? I would keep it open.

moio · 2018-10-17T05:04:50Z

The second sentence was a remark, but the question was relative to the quote. Quoting again:

The correct solution is really to put it into the image.
OK. Do we want to keep this PR open then?

Bischoff · 2018-10-25T13:25:51Z

@moio we still need the consoleblock for the virtualization, even if we move the "listening" part to the image.

If you create an image that listens on the console, I will refactor this by pruning the unneeded parts.

So yes, please keep it open.

moio · 2018-10-26T05:56:26Z

What changes are needed exactly on which images?

Bischoff · 2018-10-26T09:33:57Z

@moio Add console=ttyS0,115200 to kernel command line parameters on all images.

moio · 2018-11-06T08:24:21Z

OpenSUSE images rebuilt, SLE images are building (to be ready at 10am), CentOS images within the day. Please rebase, test and merge - let me know if something does not work as expected.

Bischoff · 2018-11-06T09:07:18Z

Thanks so much @moio !!!

Bischoff · 2018-11-07T11:25:48Z

apparently, there is nothing to merge anymore, since @MalloZup's code for cloudinit has been merged, so we have a serial console already...
but apparently you do not pass correct options on kernel's command line :

ebi3-minion:~ # cat /proc/cmdline
BOOT_IMAGE=/boot/linux.vmx root=/dev/vda1 disk=/dev/vda nomodeset elevator=noop showopts
ebi3-minion:~ # grep PRETTY /etc/os-release
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP3"

so it does not work

moio · 2018-11-09T07:57:14Z

Right, there was a problem with OBS publishing, now fixed. I tested both SLES and openSUSE and they do the right thing now.

New CentOS images will be provided via the sumaform-images project and they are ready, including console support.

Only problem is a slight incompatibility of GitHub Releases with terraform-provider-libvirt, solved in this PR currently pending:

dmacvicar/terraform-provider-libvirt#472

Plan:

I am closing this PR now as no code change is needed anyway
as soon as Support GET-only HTTP servers for image sources dmacvicar/terraform-provider-libvirt#472 is accepted, I will update sumaform's build of terraform-provider-libvirt
as soon as that is ready, I will be committing a change in sumaform to switch over to sumaform-images built images

In future, we will not have hosting problems any more as GitHub releases are publicly available. Moreover, maintenance of Packer images will be easier as all needed software is containerized, so reproducing the build is not machine dependent any more.

Bischoff · 2018-11-12T10:53:58Z

All clear, thanks Silvio for your help on this long-awaited feature of using virsh console 😃 .

Bischoff requested a review from MalloZup October 10, 2018 15:30

MalloZup approved these changes Oct 10, 2018

View reviewed changes

moio reviewed Oct 10, 2018

View reviewed changes

Bischoff self-assigned this Oct 10, 2018

MalloZup mentioned this pull request Oct 15, 2018

[WIP] Use upstream DOMAIN_INTERFACE_ADDRESSES_SRC_AGENT dmacvicar/terraform-provider-libvirt#432

Closed

Add support for serial consoles

c1a43cf

moio closed this Nov 9, 2018

Bischoff deleted the console branch November 30, 2018 14:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for serial consoles #436

Add support for serial consoles #436

Bischoff commented Oct 10, 2018

MalloZup left a comment

moio left a comment

Bischoff commented Oct 10, 2018 •

edited

Loading

MalloZup commented Oct 10, 2018 •

edited

Loading

Bischoff commented Oct 10, 2018 •

edited

Loading

Bischoff commented Oct 10, 2018 •

edited

Loading

MalloZup commented Oct 11, 2018 •

edited

Loading

Bischoff commented Oct 11, 2018 •

edited

Loading

moio commented Oct 11, 2018

Bischoff commented Oct 11, 2018

Bischoff commented Oct 11, 2018

MalloZup commented Oct 11, 2018

MalloZup commented Oct 11, 2018 •

edited

Loading

Bischoff commented Oct 11, 2018 •

edited

Loading

Bischoff commented Oct 11, 2018

Bischoff commented Oct 11, 2018

MalloZup commented Oct 11, 2018

Bischoff commented Oct 12, 2018

MalloZup commented Oct 12, 2018

moio commented Oct 12, 2018

Bischoff commented Oct 12, 2018 •

edited

Loading

moio commented Oct 16, 2018

MalloZup commented Oct 16, 2018

Bischoff commented Oct 16, 2018

moio commented Oct 17, 2018

Bischoff commented Oct 25, 2018

moio commented Oct 26, 2018

Bischoff commented Oct 26, 2018

moio commented Nov 6, 2018

Bischoff commented Nov 6, 2018

Bischoff commented Nov 7, 2018 •

edited

Loading

moio commented Nov 9, 2018

Bischoff commented Nov 12, 2018

Add support for serial consoles #436

Add support for serial consoles #436

Conversation

Bischoff commented Oct 10, 2018

MalloZup left a comment

Choose a reason for hiding this comment

moio left a comment

Choose a reason for hiding this comment

Bischoff commented Oct 10, 2018 • edited Loading

MalloZup commented Oct 10, 2018 • edited Loading

Bischoff commented Oct 10, 2018 • edited Loading

Bischoff commented Oct 10, 2018 • edited Loading

MalloZup commented Oct 11, 2018 • edited Loading

Bischoff commented Oct 11, 2018 • edited Loading

moio commented Oct 11, 2018

Bischoff commented Oct 11, 2018

Bischoff commented Oct 11, 2018

MalloZup commented Oct 11, 2018

MalloZup commented Oct 11, 2018 • edited Loading

Bischoff commented Oct 11, 2018 • edited Loading

Bischoff commented Oct 11, 2018

Bischoff commented Oct 11, 2018

MalloZup commented Oct 11, 2018

Bischoff commented Oct 12, 2018

MalloZup commented Oct 12, 2018

moio commented Oct 12, 2018

Bischoff commented Oct 12, 2018 • edited Loading

moio commented Oct 16, 2018

MalloZup commented Oct 16, 2018

Bischoff commented Oct 16, 2018

moio commented Oct 17, 2018

Bischoff commented Oct 25, 2018

moio commented Oct 26, 2018

Bischoff commented Oct 26, 2018

moio commented Nov 6, 2018

Bischoff commented Nov 6, 2018

Bischoff commented Nov 7, 2018 • edited Loading

moio commented Nov 9, 2018

Bischoff commented Nov 12, 2018

Bischoff commented Oct 10, 2018 •

edited

Loading

MalloZup commented Oct 10, 2018 •

edited

Loading

Bischoff commented Oct 10, 2018 •

edited

Loading

Bischoff commented Oct 10, 2018 •

edited

Loading

MalloZup commented Oct 11, 2018 •

edited

Loading

Bischoff commented Oct 11, 2018 •

edited

Loading

MalloZup commented Oct 11, 2018 •

edited

Loading

Bischoff commented Oct 11, 2018 •

edited

Loading

Bischoff commented Oct 12, 2018 •

edited

Loading

Bischoff commented Nov 7, 2018 •

edited

Loading