Two great tastes: Agile development and distributed, declarative setup

I’ve had the good fortune to work on several agile software-development teams over the years. If you’ve ever hated having software you’ve developed sit on the shelf for months or years until the next release, shipping several times a year feels good. Shipping almost every month is even better. Shipping almost every week is phenomenal.

Most of the principles of the agile manifesto reinforce the idea that shipping software often is how to best deliver customer value. To ship often, you need predictability:

  • Finish all the work associated with a feature.
  • Make sure it’s of ship quality.

Done versus done-done

The waterfall model of designing, then coding, then throwing over the wall to test, then fixing bugs, then throwing over the other wall to setup–yes, of course I’m going to talk about setup–is just procrastinating. Sure, it’s more fun to start working on the next new thing, but it’s just postponing the work required to get a feature really done and ready to ship.

Setup development, like testing, is often considered one of the “taxes” of software development that is best put off as long as possible. But if you want to deliver shippable software on a rapid schedule, paying taxes late doesn’t work; you might get to April 15th and discover that you have more bugs than you can fix by midnight. Instead, you pay your taxes as you go along so there are no surprises when you get to the last day and you want to ship.

Rather than developers considering themselves “done” when they’ve written some code and debugged it, developers need to get “done done” by providing code, unit tests, acceptance tests, and setup.

WiX supports “done done” by easily integrating into the development build cycle. Every member of an agile team can author setup. That prevents the lone setup developer, who probably supports multiple product teams, from being a bottleneck.

Plus, if developers are responsible for their own setup authoring, they’re less likely to push bad design decisions onto the setup developer’s shoulders.

Keep quality high and take the weekends off

Agile methodologies bring a whole set of tools and processes to help keep quality high and ready to “ship on a dime,” like:

  • Test-driven development to produce high-quality designs and code
  • Automated acceptance testing to reduce the need for armies of testers to validate the whole system at the end

Add some setup flavor:

  • Make sure that “done” for every feature includes its setup authoring.
  • Run your automated acceptance tests from official setup packages, not from local builds.
  • Build upgrades and patches and create automated tests to ensure you can service the product.
  • Use only fully declarative custom actions–they make it easier to handle patching, upgrades, and repair.

Frequent, automated testing tells you if your code (and setup) do what they’re supposed to (or not–at least as far as the tests are concerned). Adding setup into the testing mix helps ensure that your setup is ready too.

Test your setups virtually

A key part of most setup development is testing from a known state to see that your setup leaves the machine in another known state. Usually, it’s from a “clean” state to one with your app installed but I’m sure I’m not alone with needing to duplicate bugs that come from a “dirty” machine. In both cases, you need to be able to quickly get a test machine to the starting state.

One way to get consistent machine states is with imaging software: Take an image of the machine in a particular state using software like Acronis True Image and restore the image before you begin each test. The downside to machine images is the time needed to restore an image; a bare operating system image might be only a couple of gigabytes for a svelte Windows XP image but your setup or app might have other prerequisites that balloon the image size. For example, my testing for Flight Simulator X Service Pack 1 and Flight Simulator X: Acceleration both require an installation of Flight Simulator X, all 13GB and 42,000 files worth. Such an image takes about 20 minutes to restore, not counting the reboots and loading the imaging software: Call it a good half an hour. With that overhead, I avoid re-imaging between install tests; the overhead’s not worth it. And surely I never author any bad setup or make even the slightest bug in my deferred custom actions, much less the rollback custom actions. Oh wait…

But virtual machines have significantly less overhead. Instead of re-imaging, you shut down the VM and discard changes made during the test session. Then restart the VM to start clean: The only overhead is the time it takes the VM to “power on” and boot. Most VM software lets you create snapshots to record the state of the VM at key points. You can also create new VMs as clones or deltas of other VMs, to save the time of installing OSes over and over.

There are two potential drawbacks to cloning VMs:

  1. Don’t forget your OS licensing.
  2. If you need to join your VMs to a network domain, you’ll have to use a tool like SysPrep or NewSID to join cloned VMs to the domain.

Of course, both those things are true with hardware machine images too. I avoid the problem by not joining VMs to domains. Most VM software offers other ways of getting test files to VMs, like shared folders that simulate a network share to the host computer or drag-and-drop.

VMs aren’t as fast as real hardware but unless you have a huge setup or want to investigate timing-related issues, it’s rarely a serious problem. Though it gives you a great excuse for a powerful dev box: A quad-core CPU with 4GB RAM and RAID 0 storage works really well.<g> Fewer cores, less RAM, and non-RAID disks work too, of course, depending on how many simultaneous VMs you want to run.

VM software

Here’s a list of the VM software I’ve used.

  • Virtual PC, Microsoft’s desktop VM product. Free (as in beer). Easy to use, OK GUI. Biggest cons: It’s a single-threaded app, so multiple VMs don’t take advantage of multi-core CPUs and it has no automation API.
  • Virtual Server, Microsoft’s current server VM product. Free (as in beer). Multi-threaded with COM API. Biggest con: In the box, there’s only a Web administrative interface. VMRCplus, an unsupported WinForms UI, is available and while it has some rough edges, it’s a big improvement.
  • VMware Workstation, VMware’s desktop VM product. $189. Great multi-process GUI with automation APIs. Hardware support improves with each version; the current version, 6.0, supports USB 2.0. The only drawback is that because it’s not free software (beer or speech), you need to make sure you’re in compliance with license if you want to use it on multiple machines.
  • VirtualBox, innotek’s VM product. Free (beer and speech). VirtualBox comes in a closed-source edition that’s free of charge and a subset is available under the GPL. Multi-process and nicely automatable, via command line and XML configuration files. There’s a separate API that isn’t yet documented. The biggest downside is that virtual machines, hard drives, and CD/DVD images have to be registered so it’s slightly more complicated than the others to copy VMs among host machines. Secret setup geek benefit: The closed-source edition of VirtualBox comes with an installer — authored in WiX — that’s amazingly clean; it doesn’t quite pass validation but it’s very light on black-box custom actions.

Using machine images is vital if your setup requires particular hardware or if your app does and you need to test both the setup and the app. Otherwise, save yourself headaches, hassles, and horked dev machines and test your setups in virtual machines.

Apple Safari setup built with WiX

Apple’s Safari browser is now available in public beta on Windows. A little spelunking shows that it uses Windows Installer packages and that they’re built with WiX. Sadly, they didn’t use WixUI.<g>

On a more serious-but-sad note, the packages have ICE validation errors (other than the typical ones), contain VBScript custom actions, and the main Safari package uses a custom action to install the Apple Software Updater package (instead of using a chainer). Already there’s a forum report of a 2738 error with the VBScript CA. And I guess the only way to report bugs is via forum posts…? It’s not clear. The “Report Bugs to Apple” command on the Help menu seems focused on rendering problems. Too bad. After all, Setup Development Is Just Development.

When VBScript and JScript custom actions are even more evil than usual

As everyone knows, script custom actions are inherently evil. A security addition to Windows Installer 4.0 in Windows Vista means that script CAs are even more likely to fail; see Heath’s blog entry on the issue and Aaron’s follow-up.

But did you know that script CAs, evil that they are, nonetheless ship in Orca, MsiVal2, and even the WiX toolset? Shocking but true. The Internal Consistency Evaluators (ICEs) are implemented as custom actions in .cub files that are MSI databases with a vastly different schema you’re used to seeing. A couple of ICEs are written in VBScript so a misregistered VBScript engine will cause those ICEs to fail. As WiX v3 runs validation by default during linking (and on-demand using the Smoke tool), failed ICEs fail your setup build.

Of 98 ICEs in the version of Darice.cub in the Windows Vista SDK, only four are written in VBScript — yet that’s all it takes to fail your build.

If you run into the problem, check out the workaround Aaron offers before you take the drastic step of disabling validation.