-
-
Notifications
You must be signed in to change notification settings - Fork 604
Live migration of OSv guests using qemu
Ingredients:
- 1 Linux host (Ubuntu 14.10 in my case; Fedora should work too)
- 1 virtual bridge (osvbr0; see below) running DHCP
- 2 instances of qemu (one active and one stand-by, that we'll be migrating to) attached to the virtual bridge
- 1 API client that connects to the active qemu instance via the virtual bridge
Diagrams:
+------------------------------------------------------------+
| |
| +-----------------+ +-----------------+ |
| | | | | |
| | OSv1 | | OSv2 | |
| | (runs cli) | --> | (standby) | |
| | | | listens on 4444 | |
| +-----------------+ +-----------------+ |
| | 192.168.123.10 | |
| | | |
| +----------------------------------------------------+ | before
| | osvbridge - 192.168.123.0/24 | | migration
| | | |
| +----------------------------------------------------+ |
| | |
| | 192.168.123.1 |
| +-----------------------+ |
| | API client | |
| +-----------------------+ |
| Ubuntu 14.10 (fvbrick) |
+------------------------------------------------------------+
+------------------------------------------------------------+
| |
| +-----------------+ +-----------------+ |
| | | | | |
| | OSv1 | | OSv2 | |
| | (idle) | | (runs cli) | |
| | | | | |
| +-----------------+ +-----------------+ |
| | | 192.168.123.10
| | | |
| +----------------------------------------------------+ | after
| | osvbridge - 192.168.123.0/24 | | migration
| | | |
| +----------------------------------------------------+ |
| | |
| | 192.168.123.1 |
| +-----------------------+ |
| | API client | |
| +-----------------------+ |
| Ubuntu 14.10 (fvbrick) |
+------------------------------------------------------------+
Preparation (all the paths are relative to the root of the OSv install directory):
-
Add the following lua script as 'modules/cli/commands/loop.lua':
local OptionParser = require('std.optparse') local socket = require('socket') local cmd = {} cmd.desc = [[infinite loop]] cmd.help = [[Usage: loop [-p period] Infinte loop.]] cmd.parser = OptionParser [[ loop Usage: loop infinite loop Options: -p, --period=[SECONDS] Loop interval (in seconds) ]] cmd.main = function(args) local args, opts = cmd.parser:parse(args) local period = 2 if opts.period then period = opts.period end local counter = 0 local prev_time = socket.gettime() while not cli_interrupted() do local date, status = osv_request({"os", "date"}, "GET") osv_resp_assert(status, 200) counter = counter + 1 local time = socket.gettime() io.write(date, ' - ', counter, ' - ', time, ' - ', (time - prev_time), '\n') prev_time = time socket.sleep(period) end end return cmd -
Build the standard OSv image as usual; at the same time also build the API cli:
make cli cd modules/cli && make cd - -
As 'root' (or using 'sudo'), set up (and check) the virtual bridge as follows (network, IP addresses and MAC addresses are arbitrary; if you change them, make sure you change them consistently in all the steps below):
brctl addbr osvbr0 ip addr add dev osvbr0 192.168.123.1/24 ip link set up dev osvbr0 ip addr show osvbr0 ip route show 192.168.123.0/24to allow 'qemu-bridge-helper' (used inside 'scripts/run.py' below) access to this new virtual bridge, add this line to '/etc/qemu/bridge.conf' - this step needs to be done only once, since this is a permanent file:
allow osvbr0 -
As 'root' (or using 'sudo') start a DHCP server just on the virtual bridge interface serving only IP address (192.168.123.10) to a specific MAC address (52:54:30:30:30:30), which will be the MAC address that qemu will use for the OSV instances:
/usr/sbin/dnsmasq --bind-interfaces --interface=osvbr0 --listen-address=192.168.123.1 --port=0 --no-resolv --dhcp-range=192.168.123.100,192.168.123.254 --dhcp-host=52:54:30:30:30:30,192.168.123.10,infinite -
In one terminal (terminal #1) start the first instance of qemu (the active one):
scripts/run.py -p kvm -c 1 -m 100 -n -b osvbr0 --mac 52:54:30:30:30:30 --novnc --nogdb(if everything is right, you should see that instance receive the IP address 192.168.123.10)
-
In another terminal (terminal #2) start the 'loop' script using the cli:
cd modules/cli && ./cli --api=http://192.168.123.10:8000 -- loop -p 1(if everything is right, you should see a line every second with the current time, a counter that starts from 1 and keeps increasing, the number of seconds returned by 'socket.gettime()', and the delta between two successive values)
-
In yet another terminal (terminal #3) start the second instance of qemu (the stand-by one) - please note that the space before ' -incoming' in this command is not a typo; the command will fail without that space due to the way Python argparse interprets its arguments:
scripts/run.py -p kvm -c 1 -m 100 -n -b osvbr0 --mac 52:54:30:30:30:30 --novnc --nogdb --pass-args ' -incoming tcp:0:4444'(nothing should happen here since this instance is currently in stand-by mode)
-
(Finally!) migrate the qemu active instance to the stand-by as follows:
-
in terminal #1 type: Control-A c (lowercase c) to be at the qemu command prompt (you should see the prompt '(qemu)')
-
at the '(qemu)' prompt type the actual migration command:
migrate "tcp:127.0.0.1:4444" -
within a second or so, the migration will complete; the cli terminal should continue running without interruption; you may see an abrupt change in the timestamp value, but the value returned by 'socket.gettime()' and the counter by one should work as expected, showing that the OSv guest state is correctly preserved during this live migration.
-
you can terminate the no longer used qemu instance (the one in terminal #1) by typing the command:
quitat the '(qemu)' prompt
-
-
(optional) you can play live migration ping pong and further migrate this OSv instance to another instance of qemu.
-
in terminal #1 start again qemu in stand-by mode:
scripts/run.py -p kvm -c 1 -m 100 -n -b osvbr0 --mac 52:54:30:30:30:30 --novnc --nogdb --pass-args ' -incoming tcp:0:4444' -
in terminal #3 (which after the previous step is now the active instance), go to the '(qemu)' prompt (by typing ^A c) and enter the migrate command:
migrate "tcp:127.0.0.1:4444" -
and then terminate that qemu instance with the command 'quit' (or ^A x)
-
the cli command (in terminal #2) should still continue running; at any moment you can also validate which one is the active instance by typing return in that terminal and you should see the standard '/#' prompt where you can type cli commands. You'll also see that if you terminate the active instance (with ^A x, for instance) the time/sequence progression in the cli will stop immediately.
-
Enjoy!
References:
- KVM Migration - http://www.linux-kvm.org/page/Migration
This part assumes you are already familiar with the inter host (same host) case, since it borrows many of the concepts and setups from that simpler scenario.
I used two qemu virtual machines for this case, since I only have my computer; it should work also using Vagrant or with two different physical servers.
Ingredients:
- 1 Linux host (Ubuntu 14.10 in my case; Fedora should work too)
- 2 virtual machines to simulate the two hosts used for the qemu OSv migration (two Fedora server 21 instances run under qemu in my case - see below)
- 1 virtual bridge to run commands (via ssh) on the two VMs above - for this purpose I used the standard 'virbr0' that comes with the 'libvirt' package
- 1 virtual bridge (osvbr0; see below) running DHCP
- 2 instances of qemu (one active and one stand-by, each of them running on one of the VMs that act as the hosts to migrate from and to) attached to the 'osvbr0' virtual bridge
- 1 API client that connects to the active qemu instance via the 'osvbr0' virtual bridge
Diagrams:
Please note that to make the diagrams not too complicated, I don't show below the 'virbr0' bridge used to send commands to the two Fedora 21 VMs, and the additional Ethernet interface that is used on the two VMs to connect to the 'osvbr0' bridge on the Ubuntu host.
+--------------------------------------------------------------+
| |
| +---------------------+ +-----------------------+ |
| | Fedora 21 Server | | Fedora 21 Server | |
| | hostA | | hostB | |
| | +----------+ | --> | +------------+ | |
| | | OSv1 | | | | OSv2 | | |
| | |(runs cli)| | | |(stdby/4444)| | |
| | +----------+ | | +------------+ | |
| +---------- | ------+ +---------- | --------+ |
| | 192.168.123.10 | |
| | | |
| +------------------------------------------------------+ | before
| | osvbridge - 192.168.123.0/24 | | migration
| +------------------------------------------------------+ |
| | |
| | 192.168.123.1 |
| +-----------------------+ |
| | API client | |
| +-----------------------+ |
| Ubuntu 14.10 (fvbrick) |
+--------------------------------------------------------------+
+--------------------------------------------------------------+
| |
| +---------------------+ +-----------------------+ |
| | Fedora 21 Server | | Fedora 21 Server | |
| | hostA | | hostB | |
| | +----------+ | | +------------+ | |
| | | OSv1 | | | | OSv2 | | |
| | | (idle) | | | | (runs cli) | | |
| | +----------+ | | +------------+ | |
| +---------- | ------+ +---------- | --------+ |
| | | 192.168.123.10
| | | |
| +------------------------------------------------------+ | after
| | osvbridge - 192.168.123.0/24 | | migration
| +------------------------------------------------------+ |
| | |
| | 192.168.123.1 |
| +-----------------------+ |
| | API client | |
| +-----------------------+ |
| Ubuntu 14.10 (fvbrick) |
+--------------------------------------------------------------+