Test your setups virtually

A key part of most setup development is testing from a known state to see that your setup leaves the machine in another known state. Usually, it’s from a “clean” state to one with your app installed but I’m sure I’m not alone with needing to duplicate bugs that come from a “dirty” machine. In both cases, you need to be able to quickly get a test machine to the starting state.

One way to get consistent machine states is with imaging software: Take an image of the machine in a particular state using software like Acronis True Image and restore the image before you begin each test. The downside to machine images is the time needed to restore an image; a bare operating system image might be only a couple of gigabytes for a svelte Windows XP image but your setup or app might have other prerequisites that balloon the image size. For example, my testing for Flight Simulator X Service Pack 1 and Flight Simulator X: Acceleration both require an installation of Flight Simulator X, all 13GB and 42,000 files worth. Such an image takes about 20 minutes to restore, not counting the reboots and loading the imaging software: Call it a good half an hour. With that overhead, I avoid re-imaging between install tests; the overhead’s not worth it. And surely I never author any bad setup or make even the slightest bug in my deferred custom actions, much less the rollback custom actions. Oh wait…

But virtual machines have significantly less overhead. Instead of re-imaging, you shut down the VM and discard changes made during the test session. Then restart the VM to start clean: The only overhead is the time it takes the VM to “power on” and boot. Most VM software lets you create snapshots to record the state of the VM at key points. You can also create new VMs as clones or deltas of other VMs, to save the time of installing OSes over and over.

There are two potential drawbacks to cloning VMs:

  1. Don’t forget your OS licensing.
  2. If you need to join your VMs to a network domain, you’ll have to use a tool like SysPrep or NewSID to join cloned VMs to the domain.

Of course, both those things are true with hardware machine images too. I avoid the problem by not joining VMs to domains. Most VM software offers other ways of getting test files to VMs, like shared folders that simulate a network share to the host computer or drag-and-drop.

VMs aren’t as fast as real hardware but unless you have a huge setup or want to investigate timing-related issues, it’s rarely a serious problem. Though it gives you a great excuse for a powerful dev box: A quad-core CPU with 4GB RAM and RAID 0 storage works really well.<g> Fewer cores, less RAM, and non-RAID disks work too, of course, depending on how many simultaneous VMs you want to run.

VM software

Here’s a list of the VM software I’ve used.

  • Virtual PC, Microsoft’s desktop VM product. Free (as in beer). Easy to use, OK GUI. Biggest cons: It’s a single-threaded app, so multiple VMs don’t take advantage of multi-core CPUs and it has no automation API.
  • Virtual Server, Microsoft’s current server VM product. Free (as in beer). Multi-threaded with COM API. Biggest con: In the box, there’s only a Web administrative interface. VMRCplus, an unsupported WinForms UI, is available and while it has some rough edges, it’s a big improvement.
  • VMware Workstation, VMware’s desktop VM product. $189. Great multi-process GUI with automation APIs. Hardware support improves with each version; the current version, 6.0, supports USB 2.0. The only drawback is that because it’s not free software (beer or speech), you need to make sure you’re in compliance with license if you want to use it on multiple machines.
  • VirtualBox, innotek’s VM product. Free (beer and speech). VirtualBox comes in a closed-source edition that’s free of charge and a subset is available under the GPL. Multi-process and nicely automatable, via command line and XML configuration files. There’s a separate API that isn’t yet documented. The biggest downside is that virtual machines, hard drives, and CD/DVD images have to be registered so it’s slightly more complicated than the others to copy VMs among host machines. Secret setup geek benefit: The closed-source edition of VirtualBox comes with an installer — authored in WiX — that’s amazingly clean; it doesn’t quite pass validation but it’s very light on black-box custom actions.

Using machine images is vital if your setup requires particular hardware or if your app does and you need to test both the setup and the app. Otherwise, save yourself headaches, hassles, and horked dev machines and test your setups in virtual machines.

7 thoughts on “Test your setups virtually”

  1. Nice article. I use VMWare extensively to test and debug my installers.

    A couple tips for VMWare:
    I only have a single harddrive and I find find Disk IO to be the biggest performance bottleneck–especially when an installer is running in the VM and copying a lot of files. The disk churn makes the host OS very frustrating to use.
    I put my VM images on an external hard drive and it makes things run a lot more smoothly,

    A use a lot of “Snapshots” (VMWare-ese for saved states). Saving and restoring In Memory snapshots (while the guest OS is running) can take several minutes. Saving and restoring snapshots of when the guest OS is powered off are instantaneous. Of course, sometimes you need In Memory snapshots.

    -Ben

  2. Thanks — even if you don’t use RAID 0, keeping your VMs and host OS and apps on separate drives helps avoid contention if you want to do some work while a setup churns in the VM. External drives are great, especially on laptops. I’m not in the market now, but the next time I buy a laptop, it’ll have an eSATA port!

  3. I’ve recently begun preaching VM based testing and noticed you left out VMware’s free VMware Server products. Granted, it’s not as fully featured as VMware’s other products, but reverting your only snapshot to a fresh install of your test OS still gets the job done quicker than imaging a machine each time.

  4. Erik,
    I skipped it only because I haven’t used it enough to judge it. I like the desktop-oriented products because it’s easier to use VMs that aren’t connected to a network. (That helps avoid the problem I mentioned about domain networks.)

    But I’m waiting to see what parts of Workstation 6 end up in Server 2.

  5. I use Altiris Software Virtualization Solution (SVS). It’s like a sandbox: you can install any software using this application and if you don’t want the app anymore, don’t bother uninstalling it – just delete it. All registry settings and files – EVERYTHING is removed. This is very handy if your installation worked, but the uninstall fails.
    Its free for personal use…

  6. Nice writeup. It should also be noted that there are ways around the domain membership problems by disabling computer account password changes. This allows you to do tricks like reload the OS and rejoin the domain without havingto ask an administrator to join you. Also reload images without losing domain membership and even switch between different operating systems or different OS configs.

    http://support.microsoft.com/kb/154501

Comments are closed.