lxc-create fails mysteriously if /var/lxc is BTRFS and ~/.vagrant/boxes is not

[sport-clan]

Hi,

I'm following the README and failing at the vagrant up stage. I'm running Ubuntu 13.04. Here's some sample output:

james@simba:~/projects/hq/hq-mongodb-installer$ vagrant up --provider=lxc
Bringing machine 'default' up with 'lxc' provider...
[default] Importing base box 'quantal64'...
There was an error executing ["sudo", "lxc-create", "--template", "vagrant-tmp-hq-mongodb-installer-1369893590", "--name", "hq-mongodb-installer-1369893590", "--", "--arch", "amd64", "--release", "quantal", "--tarball", "/home/james/.vagrant.d/boxes/quantal64/lxc/rootfs.tar.gz", "--auth-key", "/opt/vagrant/embedded/gems/gems/vagrant-1.2.2/keys/vagrant.pub"]

For more information on the failure, enable detailed logging by setting
the environment variable VAGRANT_LOG to DEBUG.

If I set the debug mode I can see the following output from lxc-create:

DEBUG subprocess: stdout: 
DEBUG subprocess: stdout: lxc-create: No config file specified, using the default config /etc/lxc/default.conf
DEBUG subprocess: stdout: Extracting /home/james/.vagrant.d/boxes/quantal64/lxc/rootfs.tar.gz ...
DEBUG subprocess: stderr: rm: cannot remove '/var/lib/lxc/hq-mongodb-installer-1369893761/rootfs': Operation not permitted
DEBUG subprocess: stderr: lxc-create: failed to execute template 'vagrant-tmp-hq-mongodb-installer-1369893761'
DEBUG subprocess: stdout: Delete subvolume '/two/lxc/hq-mongodb-installer-1369893761/rootfs'
DEBUG subprocess: stderr: lxc-create: aborted
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 1

This turns out to be a problem with the way my filesystem is set up... I hade made /var/lxc a symlink to /two/lxc, where /two is a BTRFS filesystem. I guess lxc-create is trying to use some kind of BTRFS magic to clone the filesystem from my vagrant boxes dir ~/.vagrant/boxes to the /var/lxc location.

I managed to fix this by creating a new subvolume and making ~/.vagrant/boxes to that. Follow the instructions again and the box comes up fine. I wonder if you should make this part of the documentation, or even better detect it and warn the user beforehand.

I imagine that using BTRFS to drive /var/lxc will be a common requirement, and if you already have a root partition with another filesystem then this is an easy way to achieve it.

[fgrehm]

@sport-clan tks for reporting this, I haven't used BTRFS before but after some debugging looks like the issue is this line on creation scripts. could you please try commenting out that rm -rf from your ~/.vagrant.d/boxes/quantal64/lxc/lxc-template and recreating the box to see if it works?
TBH I'm not really sure why I added that, I'm gonna have to look at the git history to try to find out :-P

[jamespharaoh]

Sorry, I reported under the wrong user id, this is my personal account.

As for the bug, I can't reproduce it now :-(

I've set my boxes dir back to a plain old directory (on my ext3 root partition as opposed to a symlink to the same btrfs partition where /var/lib/lxc points). But it seems to work fine now!

I wonder if I've installed an update that has fixed the underlying behaviour since yesterday...

[fgrehm]

@jamespharaoh I was able to reproduce it and looks like removing the rm -rf did the trick. Since I've never used BTRFS before, I'll put up a Vagrant + VBox example setting things up when I have a chance for you to try out and check if I'm missing something else

[fgrehm]

@jamespharaoh would you be able to help me understand if I got it right? I've used an Ubuntu Raring VBox machine that you can find on https://gist.github.com/fgrehm/b07c6370a710be622807 and I followed instructions from http://www.funtoo.org/BTRFS_Fun to set things up and reproduce the problem.

Once you vagrant up that VBox machine, you can ssh into it and run:

dd if=/dev/zero of=/tmp/btrfs-vol0.img bs=1G count=1
sudo losetup /dev/loop0 /tmp/btrfs-vol0.img
sudo mkfs.btrfs -m single /dev/loop0
sudo mkdir -p /var/lib/lxc
sudo mount /dev/loop0 /var/lib/lxc

That will make /var/lib/lxc a BTRFS filesystem, once that is set, you can run:

mkdir ~/btrfs-test
cd ~/btrfs-test
vagrant init quantal64 http://dl.dropbox.com/u/13510779/lxc-quantal-amd64-2013-05-08.box
VAGRANT_LOG=debug vagrant up

And you'll get the same error again. If you remove the line 169 from ~/.vagrant.d/boxes/quantal64/lxc/lxc-template and try bringing up that machine again it will just work.

Does that represent the setup you had when you've hit this?

[jamespharaoh]

@fgrehm Yes i think that is basically the same as what I had when I saw the error.

[fgrehm]

@jamespharaoh cool, I'll release new boxes with the updated lxc template as soon as I have the time. tks for reporting :-)

[fgrehm]

@jamespharaoh this has been released ;)