Windows 7 x64 HVM on R4.0

[ghost]

EDIT - the instructions below are outdated ; updated instructions are in the official docs at https://www.qubes-os.org/doc/windows/

---

[the purpose of this issue is to gather user experience and fine tune the instructions before submitting a PR to the official documentation]

Qubes OS version:

R4.0

Affected TemplateVMs:

Windows 7 x64 HVM

Related issues:

#3585
#3495

To be discussed:

• feedback/problems with the current set of instructions
• link to how to use LVM snapshots ? If that works it's likely the best way to save the installation state between reboots instead of qvm-clone (qvm-backup is far too slow).

---

Edits

• 18 feb. / reflect comments by @dmoerner in Qubes Windows Tools (QWT) on R4 #3585
  • qrexec_timeout 300
  • windows update + Xen VBD driver note
• 17 feb. / reflect comments by @dmoerner + added tweaks section.
  • re-added qvm-prefs win7new virt_mode hvm since Qubes 4 defaults to PVH
  • fixed missing vm name in qvm-start
  • extend the root volume to 25g (which might still not be enough). @dmoerner reports a total install size of 19.8g (my install is ~11g but I've set a fixed pagefile size and QWT moved the user profiles to the private volume).

---

Importing a Windows VM from R3.2

Importing should work, just make sure that you're not using Xen's newer R4 linux stubdomain and that the VM is in HVM mode (these steps should be done automatically when importing the VM):

qvm-features VMNAME linux-stubdom ''
qvm-prefs VMNAME virt_mode hvm

Note however that you're better off creating a new Windows VM to benefit from the more recent emulated hardware: R3.2 used a MiniOS based stubdomain with an old and mostly unmaintained 'qemu-traditional' while R4.0 uses a Linux based stubdomain with a recent version of upstream qemu (see this post).

Windows 7 x64 install on R4.0

tl;dr;

Most of the commands below can be combined into only a few commands but they're split for the sake of clarity; each command is explained in the next section.

qvm-create --class StandaloneVM --label red win7new
qvm-prefs win7new virt_mode hvm
qvm-prefs win7new memory 4000
qvm-prefs win7new maxmem 4000
qvm-prefs win7new kernel ''
qvm-volume extend win7new:root 25g
qvm-prefs win7new debug True
qvm-features win7new video-model cirrus
qvm-start --cdrom=untrusted:/home/user/windows_install.iso win7new
# restart after the first part of the windows installation process ends
qvm-start win7new
# once Windows is installed and working
qvm-features --unset win7new video-model
qvm-prefs win7new memory 2000
qvm-prefs win7new maxmem 2000
# with Qubes Windows Tools installed:
qvm-prefs win7new debug False
qvm-prefs win7new qrexec_timeout 300

Detailed instructions

Create a VM named win7new in HVM mode (limitations in xen precludes from using PVH):

qvm-create --class StandaloneVM --label red win7new
qvm-prefs win7new virt_mode hvm

Windows' installer requires a lot of memory ; 2000m is not enough, the VM will crash and you'll get out of memory errors like this in /var/log/xen/console/hypervisor.log:

p2m_pod_demand_populate: Dom120 out of PoD memory! (tot=102411 ents=921600 dom120)
(XEN) domain_crash called from p2m-pod.c:1218
(XEN) Domain 120 (vcpu#0) crashed on cpu#3:

Increase to 4000m temporarily (note that memory = maxmen because there's no memory balancing):

qvm-prefs win7new memory 4000
qvm-prefs win7new maxmem 4000

We don't want to start a linux kernel so that the VM can boot from the CD:

qvm-prefs win7new kernel ''

A Windows installation needs more than 10g (usually between 15g up to 19g by default depending on the version) and Windows updates usually end up using significant space so
extend the root volume from the default 10g to 25g (note: if 25g aren't enough once Windows is installed, simply extend the volume again in dom0, and then extend the system partition in Windows's disk manager).

qvm-volume extend win7new:root 25g

Set the debug flag to have a graphical console:

qvm-prefs win7new debug True

The second part of the installation process will crash with the standard vga adapter and will stay in "transient" mode with the following error in guest-win7new-dm.log:

qemu: /home/user/qubes-src/vmm-xen-stubdom-linux/build/qemu/exec.c:1187: cpu_physical_memory_snapshot_get_dirty: Assertion `start + length <= snap->end' failed.

Workaround: temporarily switch to the cirrus video adapter:

qvm-features win7new video-model cirrus

The VM is now configured and can be started; the best practice is to use an installation ISO located in a VM:

qvm-start --cdrom=untrusted:/home/user/windows_install.iso win7new

At this point you may open a few tabs in dom0 for debugging should something go amiss:

tail -f /var/log/qubes/vm-win7new.log
tail -f /var/log/xen/console/hypervisor.log
tail -f /var/log/xen/console/guest-win7new-dm.log

The VM will shutdown after the installer finishes extracting the files. It's a good idea to clone, backup or snapshot the VM now. Then, (re)start the VM with qvm-start win7new.

The second part of Windows' installer should then be able to complete successfully. It is then OK to revert to the standard VGA adapter and decrease the VM's memory to a more reasonable value:

qvm-features --unset win7new video-model
qvm-prefs win7new memory 2000
qvm-prefs win7new maxmem 2000

At that point you should have a functional windows 7 installation, although without updates, Xen's PV drivers nor Qubes integration. It's a good time to clone the VM again.

Windows update

There is nothing specific to Xen or Qubes when updating the VM with Windows Update.

Depending on how old your installation media is, fully updating your Windows VM may take hours so make sure you clone your VM between the mandatory reboots in case something goes wrong. This comment provides useful links on updating a Windows 7 SP1 VM.

Note: if you already have Qubes Windows Tools installed the VM's video adapter will be set to "Qubes video driver" in which case you won't be able to see the Windows Update process when the VM is being powered off because Qubes services would have been stopped by then. Depending on the size of the Windows update packs it may take a bit of time until the VM shutdowns by itself, leaving one wondering if the VM is crashed or still finalizing the updates (in dom0 a changing CPU usage - eg. shown with xentop - usually indicates that the VM hasn't crashed).
To avoid guessing the VM's state make sure debugging is enabled (qvm-prefs win7new debug true), temporarily re-enable the standard VGA adapter, and disable Qubes' "Qubes video driver" (My computer -> Manage / Device manager / Display adapters).

Xen PV drivers + Qubes integration

If you plan to update your newly isntalled Windows VM it is recommended that you do so before installing Qubes Windows Tools (QWT). If QWT are installed, you should temporarily re-enable the standard VGA adapter in Windows (see the section above).

Installing QWT on R4.0 requires using QWT for Qubes R3.2. See issue #3585 for instructions and known issues.

Qubes Windows Tools install Xen's PV drivers by default so there is no need to install the official ones. However, Xen's VBD (storage) PV driver is disabled by default during QWT's setup and enabling it on a Windows 7 VM that isn't fully updated results in a BSOD and broken VM. Updating Windows takes hours and for casual usage there isn't much of a performance difference with or without the storage PV driver; so there is likely no need to go through the lengthy Windows Update process if your VM is isolated from the network and you don't use I/O intensive apps.

Once QWT is installed you can now turn off debugging (early graphical console):

qvm-prefs win7new debug False

And increase qrexec_timeout:

• QWT needs time to move the user profiles to the private volume the first time it's run.
• In case you happen to get a BSOD or a similar crash in the VM, chkdsk won't complete on restart before qrexec_timeout automatically halts the VM. That can really put the VM in a totally unrecoverable state, whereas with higher qrexec_timeout, chkdsk or the appropriate utility has plenty of time to fix the VM.

qvm-prefs win7new qrexec_timeout 300

Useful tweaks

• Set a fixed size page file size (system properties / advanced tab / Performance settings / advanced tab / Virtual memory / Change...)
• Adjust for best performance (system properties / advanced tab / Performance settings).

[dmoerner]

Thanks for this, a few comments:

1. I needed to also run the following to avoid Starting VM with ISO Fails: libxenlight failed to create new domain:  #3339:
   qvm-prefs win7new virt_mode hvm
2. Your qvm-start command is missing the name of the VM.
3. In my experience, 20 gb is not enough; a fresh install of Windows 7 for me, after install of all updates, comes to 19.8 gb.

[ghost]

Thanks for the feedback. I've edited the doc to reflect your comments and I also added a few 'tweaks'.

Interestingly the original doc had the qvm-prefs win7new virt_mode hvm line but I've removed it when I opened this issue because the doc says using --StandaloneVM makes hvms by default.

As for the install size, mine comes at 11g for a Win Professional version, but QWT moved the user profiles to the private volume and I've set the pagefile size to 1Go. IIRC the install size before those steps was ~15g.

[edit1 - --standaloneclass -> --StandaloneVM]
[edit2 : the first section of the hvm doc page clearly states 'As of Qubes 4, every VM is PVH by default'; so setting virt_mode to hvm is indeed required].

[awokd]

I see what you mean,

It is unnecessary for Qubes 4 users to pass in the --hvm switch. To create a StandaloneVM in Qubes 4

isn't right (as of rc4) if you need it to be HVM instead of PVH. Might be worth playing with it a bit to see if HVM is really required for a Win7 VM without PCI attached pass-through. Is it just for the install step, and can be changed to PVH after?

[ghost]

doc: fyi @adubois removed the note in his PR.

PVH/HVM: you had me here - I always assumed Windows required HVM, pci devices attached or not. The xen wiki indeed shows that HVM is required, but a ML post seems to indicate PVH is possible (but then that's the only related post I could find).
If xen's wiki is up to date PVH only supports direct kernel boot, so I don't see how Windows would work - at least for now. I'm not knowledgeable enough with xen stuff so please correct me if I'm wrong.

When set to PVH my (otherwise working) windows VM doesn't reach the boot loader because the VM's kernel pref is '' and PVH requires a kernel, resulting in the following (expected) error message:

Start failed: internal error: libxenlight failed to create new domain 'win7-r4'

/var/log/libvirt/libxl/libxl-driver.log:

xc: panic: xc_dom_core.c:208: failed to open file 'None/vmlinuz': No such file or directory: Internal error
libxl: libxl_dom.c:994:libxl__domain_firmware: xc_dom_kernel_file failed: No such file or directory

[awokd]

@taradiddles Thank you for checking! It's hard to tell on some of those Xen docs how current they are...

[marmarek]

Yes, Windows require virt_mode hvm. At least for now.

[ghost]

Thanks for the clarification, I've added a comment in the doc about PVH.

[adubois]

Sorry coming back to life after far too much work to be able to contribute. Great work, I however have few personal comments:

• Reading the doc, it seemed that Windows is very hard to get working and a troubleshooting guide is provided, which it is not the case anymore with the right parameters thanks to all contributions it is now easy. We should keep the spirit of the HVM current documentation while using the information collected to provide detailed technical understanding and troubleshooting at the end (if this error, this is because you did not allocate enough memory, if that error, it is because you did not set the basic VGA, etc...).
• An initial note should be added clarifying that the Microsoft prerequisite (of 2GB RAM and 20GB disk) are no longer valid, in particular with the Win7 Pro x64 SP1 iso and that 4GB and 25GB are the minimum (are you able to run after install with 2GB?). We should reference other pages on how to change memory, increase disk, etc... but not go into too much details.
• You need only 3 commands to type before Windows initial install (qvm-create, qvm-volume and qvm-features), I feel normal user who do not want to get into the details, just want the minimum required.
  We obviously keep the more detailed explanations in the section after. I know a fair amount of effort was required to reach a working Windows install. But now that it works, I feel we should not show any un-required complexity and just enlighten the user in case he did not follow to the letter and need to get more understanding.
  Hope I don't sound too negative and that I managed to pass my message.

[ghost]

Nope you don't sound negative - I get the idea. The problem with Windows is that you end up loosing a lot of time because 99% of the time you have to begin from scratch when your VM crashes and is unfixable, so IMHO users ought to understand what each command does. For instance if they know that the VM will be totally broken when the windows installer crashes because there's not enough RAM, maybe they'll decide to add more RAM preemptively. But I agree that's debatable.

Re- RAM: I don't have any issue lowering it to 2Go on Win7 x64 pro. The page file is fixed at 1Go. But then I don't use heavy apps like photoshop. We could advise users to keep 4Go but without memory balancing windows VMs are a memory hog and not everyone has enough RAM (like issue #3618 - I added a comment in the doc about it).

For info I've submitted a PR a few days ago - see here. The PR covers both 3.2 and 4.0, and more stuff than the issue here.

• please add your comments to the PR if you have time; I see that Marek is assigned, it might give him some ideas about what to change.
• @awokd suggested to move the command summary to the end of the doc; that's a good idea IMO given the reasons I've stated above
• I don't have anything against grouping the first few commands of the commands summary into a single one, but given that the total number of commands is ~14 for R4.0, that won't change much.

BTW I only use Windows 10min every two weeks. Just went into that doc stuff because I lost a day installing it and thought I'd save the other people some time. So any feedback is welcome to improve the docs - I'm far from a specialist on that specific topic.

[adubois]

I have done some testing with QWT, and you are right, it is not even in a rc state. Apologies for assuming it was more stable than it is. I have put comments in PR#600.

I had written this before adding my comments. your version is better and I think needs to have the spirit of the following...

What about having 3 parts for the R4:
1 - Installation (no QWT) and just put:
qvm-create win7 --class StandaloneVM --property virt_mode=hvm --property kernel="" --property memory=4096 --property maxmem=4096 --label blue --property debug=True
qvm-volume extend win7:root 25g
qvm-features win7 video-model cirrus
qvm-start win7 --cdrom=...
Including all Windows side activities (Windows Updates, netplwiz) and that if tight you can shrink mem to 2GB.
qvm-features --unset win7 video-model
This is were a user will have a stable experience.
2 - QWT (unstable, use at your own risk): and all the associated instructions.
3 - Additional info: explanation on all the options of section1
I have started the process of building QWT to try to fix some of these problems (there are a lot).

[ghost]

Closing - the instructions are now in the official docs (https://www.qubes-os.org/doc/windows-vm/)