Two great tastes: Agile development and distributed, declarative setup

I’ve had the good fortune to work on several agile software-development teams over the years. If you’ve ever hated having software you’ve developed sit on the shelf for months or years until the next release, shipping several times a year feels good. Shipping almost every month is even better. Shipping almost every week is phenomenal.

Most of the principles of the agile manifesto reinforce the idea that shipping software often is how to best deliver customer value. To ship often, you need predictability:

  • Finish all the work associated with a feature.
  • Make sure it’s of ship quality.

Done versus done-done

The waterfall model of designing, then coding, then throwing over the wall to test, then fixing bugs, then throwing over the other wall to setup–yes, of course I’m going to talk about setup–is just procrastinating. Sure, it’s more fun to start working on the next new thing, but it’s just postponing the work required to get a feature really done and ready to ship.

Setup development, like testing, is often considered one of the “taxes” of software development that is best put off as long as possible. But if you want to deliver shippable software on a rapid schedule, paying taxes late doesn’t work; you might get to April 15th and discover that you have more bugs than you can fix by midnight. Instead, you pay your taxes as you go along so there are no surprises when you get to the last day and you want to ship.

Rather than developers considering themselves “done” when they’ve written some code and debugged it, developers need to get “done done” by providing code, unit tests, acceptance tests, and setup.

WiX supports “done done” by easily integrating into the development build cycle. Every member of an agile team can author setup. That prevents the lone setup developer, who probably supports multiple product teams, from being a bottleneck.

Plus, if developers are responsible for their own setup authoring, they’re less likely to push bad design decisions onto the setup developer’s shoulders.

Keep quality high and take the weekends off

Agile methodologies bring a whole set of tools and processes to help keep quality high and ready to “ship on a dime,” like:

  • Test-driven development to produce high-quality designs and code
  • Automated acceptance testing to reduce the need for armies of testers to validate the whole system at the end

Add some setup flavor:

  • Make sure that “done” for every feature includes its setup authoring.
  • Run your automated acceptance tests from official setup packages, not from local builds.
  • Build upgrades and patches and create automated tests to ensure you can service the product.
  • Use only fully declarative custom actions–they make it easier to handle patching, upgrades, and repair.

Frequent, automated testing tells you if your code (and setup) do what they’re supposed to (or not–at least as far as the tests are concerned). Adding setup into the testing mix helps ensure that your setup is ready too.

Managed custom actions in MySQL Connector/NET

Reggie Burnett, creator and developer of MySQL’s Connector/NET ADO.NET provider, announced an alpha release of the MySQL Connector/NET. Among the new features:

Perfmon counters are now created using Wix code instead of a managed custom action. This seems to help the problem where some installation were failing becausing of being unable to register the counters.

Great job! Now, about the two remaining managed custom actions and the H2Reg CAs that don’t have rollback…

Semi-custom actions

I was working on my current double-secret project at work and got a requirement that I knew would need a custom action. The requirement is to add entries to a configuration file without overwriting existing entries.

The configuration file (scenery.cfg, for FlightSim fans) is actually in .ini format. Naturally, I looked into using the standard IniFile table but it wasn’t sufficient for my needs. Scenery.cfg has entries that point to scenery data; each entry is numbered to control the priority of the scenery in rendering.

So, for example, given the following tail entries:

[Area.114]
Title=Propeller Objects
Local=Scenery\Props
Layer=114
Active=TRUE
Required=TRUE

[Area.115]
Title=Addon Scenery
Local=Addon Scenery
Layer=115
Active=TRUE
Required=FALSE

New entries would start out like this:

[Area.116]
Title=Test1
Local=MyScenery\Test1
Layer=116
Active=TRUE
Required=FALSE

Unfortunately, scenery.cfg is different among the various editions and languages of FlightSim, so the entry numbers aren’t static. (Also, users can add third-party scenery and we can’t overwrite those entries.) Though the IniFile table supports formatted strings, adding scenery entries would require string manipulation and math operators. While those might make for interesting MSI features, they don’t exist today.

So I started planning to write a deferred custom action and its matching rollback CA. Neither is too difficult — Win32 still supports API functions to add and remove .ini file entries. (Amusingly, the SDK doc groups them together with Registry Functions.)

Still, I’m a strong believer in not doing more work than necessary. (I prefer to think of it as efficiency rather than laziness.) MSI already handles the deferred and rollback aspects of modifying .ini files via the IniFile table and WriteIniValues and RemoveIniValues standard actions. If I could plug in dynamic data, I’d be able to write a simpler immediate CA rather than two more complicated deferred/rollback CAs.

Efficiency through transience

As you probably suspect by now, MSI has just such support. Immediate CAs can add temporary tables, rows, and columns to the active database. By adding temporary rows to standard tables, immediate CAs can determine at install time what data should be installed.

It’s important to note that temporary rows are…well…temporary. There’s not a whole lot of doc in the MSI SDK about temporary rows in general but one mention points out their temporary nature:

A custom action can be used to add rows to the Registry table during an installation, uninstallation, or repair transaction. These rows do not persist in the Registry table and the information is only available during the current transaction. The custom action must therefore be run in every installation, uninstallation, or repair transaction that requires the information in these additional rows. The custom action must come before the RemoveRegistryValues and WriteRegistryValues actions in the action sequence.

So that’s the recipe: An immediate CA runs, reads whatever data it needs from a custom table, using component states to decide what data should be written to temporary rows, and writes the rows (to the Registry or IniFile tables, for example). No deferred or rollback CAs are required, because MSI handles that.

WiX’s wcautil library makes this kind of CA easy to write. The WcaOpenExecuteView function executes a query and returns a view handle. WcaFetchRecord fetches the next record from the view. The WcaGetRecordInteger, WcaGetRecordString, and WcaGetRecordFormattedString functions get column values from the record. The best is WcaAddTempRecord which does all the work of adding a temporary row. It’s a fairly straightforward function but as no other CA in WiX uses it, here’s an example:

hr = WcaAddTempRecord(&hIniTableView, &hIniColumns,
// the table
L”IniFile”,
// the column number of the key we want “uniquified”
1,
// the number of columns we’re adding
8,
// primary key
L”AcesSceneryConfig”,
// FileName — always scenery.cfg
L”scenery.cfg”,
// DirProperty — set by AppSearch in extension .wixlib
L”ACESSCENERYCFGDIR”,
// Section — [Area.<n>]
pwzArea,
// Key
L”Title”,
// Value
wzTitle,
// Action
msidbIniFileActionAddLine,
// Component_
wzComponent);

The first two parameters are pointers to MSIHANDLEs. You initialize them to NULL and the first time you call WcaAddTempRecord, it initializes them. The first handle is to a view on the table you’re inserting into. The second is a handle to information about the columns in the table. By passing them in and out as arguments, you can let WcaAddTempRecord initialize them once then re-use the same handles. WcaAddTempRecord doesn’t know when you’re done adding records, so you’re responsible for calling MsiCloseHandle on both handles.

The fourth parameter is named uiUniquifyColumn and is the column number of a column that must have a unique value, like the primary key of a table. Yes, somebody made uniquify a verb; as far as I can tell, Rob‘s to blame. Anyway, what uiUniquifyColumn does is add a semi-random value to the value you pass in as the value for the specified column. (Remember that column numbers start at one, not zero.) That ensures the temporary rows you’re adding won’t conflict with existing rows. As the rows are temporary, there’s no harm in using non-deterministic IDs.

The other parameters are straightforward: the table name, the number of columns, and the value of each column. Note that by default the number of columns you pass in must match the number of columns in the table. If you want to pass in fewer than the actual number of columns, you need to pass in your own view of the table, via a query that selects just the columns you’re interested in.

Risks

Because the data you’re adding is processed during the install transaction, then dropped, a bug in the CA could orphan the data those rows represent during uninstallation. The easiest way to mitigate this risk is to run the immediate CA during every transaction and use the install state and action state of a component to determine whether to write the temporary rows. You can write the temporary rows only when the component state is changing (i.e., being installed, being removed, being repaired). For example:

er = ::MsiGetComponentStateW(WcaGetInstallHandle(), wzComponent, &isInstalled, &isAction);
if (WcaIsInstalling(isInstalled, isAction) || WcaIsReInstalling(isInstalled, isAction) || WcaIsUninstalling(isInstalled, isAction))
{ … }

By triggering off any change in component state, the CA supports installation, uninstallation, repair, and also is smart enough to not do anything when the component isn’t being installed. That lets you avoid adding any conditions to the CA scheduling itself.

Extra credit

Naturally, a WiX compiler extension lets users easily author data into the custom table. Also, there’s no reason you have to limit yourself to standard tables: You can just easily add temporary rows to the XmlConfig table to have the WiX XmlConfig custom action modify XML files in ways you couldn’t with just XPath and formatted properties alone.

Apple Safari setup built with WiX

Apple’s Safari browser is now available in public beta on Windows. A little spelunking shows that it uses Windows Installer packages and that they’re built with WiX. Sadly, they didn’t use WixUI.<g>

On a more serious-but-sad note, the packages have ICE validation errors (other than the typical ones), contain VBScript custom actions, and the main Safari package uses a custom action to install the Apple Software Updater package (instead of using a chainer). Already there’s a forum report of a 2738 error with the VBScript CA. And I guess the only way to report bugs is via forum posts…? It’s not clear. The “Report Bugs to Apple” command on the Help menu seems focused on rendering problems. Too bad. After all, Setup Development Is Just Development.

When VBScript and JScript custom actions are even more evil than usual

As everyone knows, script custom actions are inherently evil. A security addition to Windows Installer 4.0 in Windows Vista means that script CAs are even more likely to fail; see Heath’s blog entry on the issue and Aaron’s follow-up.

But did you know that script CAs, evil that they are, nonetheless ship in Orca, MsiVal2, and even the WiX toolset? Shocking but true. The Internal Consistency Evaluators (ICEs) are implemented as custom actions in .cub files that are MSI databases with a vastly different schema you’re used to seeing. A couple of ICEs are written in VBScript so a misregistered VBScript engine will cause those ICEs to fail. As WiX v3 runs validation by default during linking (and on-demand using the Smoke tool), failed ICEs fail your setup build.

Of 98 ICEs in the version of Darice.cub in the Windows Vista SDK, only four are written in VBScript — yet that’s all it takes to fail your build.

If you run into the problem, check out the workaround Aaron offers before you take the drastic step of disabling validation.