Lux unit testing, now with extra mutations

I’m currently spending a lot of time working with custom actions, removing them where I can and improving them where they’re still necessary. The biggest change I’m making is to make the custom actions data driven using the immediate/deferred/rollback custom action triad. One of the advantages in doing so is that I can use Lux to help test my work. One of the advantages of that is that real-world experience is the best way to improve tools.

One of the challenges of developing data-driven custom actions is testing all the combinations of state that affect the custom action data immediate custom actions pass to deferred and rollback custom actions. Because Lux test packages don’t modify the machine, testing the full set of possible states is difficult. Providing known state is a strength of traditional unit-testing techniques, however; if your custom action code is well factored, it should be a simple task to provide a test double that arranges hard-coded state information that you can then test against.

Naturally, I wanted to figure out how I could provide the best of both worlds in Lux. Here’s what I came up with: test mutations. From the doc:

Test mutations let you author unit tests with different expected results. The mutation id is passed as the value of the WIXLUX_RUNNING_MUTATION property. Your custom action, typically in an ‘#ifdef DEBUG’ block, retrieves the WIXLUX_RUNNING_MUTATION property and mock different behavior based on the mutation. To author test mutations, use the Mutation element with UnitTest elements as children. For example:

<lux:Mutation Id="SimulateDiskFull">
  <lux:UnitTest … />
</lux:Mutation>

Nit runs the test package once for each mutation, setting the WIXLUX_RUNNING_MUTATION property to one mutation id at a time. Tests that aren’t children of a mutation are run every time.

Test mutations are a fairly simple change but one I hope simplifies combining traditional unit-testing techniques with declarative assertions.

Lux’s test mutations will ship in the next weekly release of WiX.

Introducing Lux: Declarative unit testing for custom actions

The Cambridge-based WiX East Virtual Team have is pleased to announce its first major contribution to WiX: The Lux unit-testing framework. Lux will be available in the next weekly release of WiX v3.5.

Unit-testing custom actions with Lux

Custom actions are a frequent cause of installation failures so it’s important to test them thoroughly. Custom actions themselves usually aren’t tested. The traditional testing approach is to run functional tests on an entire installer and to cover as many scenarios and platform combinations as possible.

Custom action patterns

WiX compiler extensions provide one way of improving custom action quality: Because compiler extensions run at build time instead of install time, they can perform all sorts of data validation and conversion on strongly-typed authoring before converting it to rows and columns of custom tables in the MSI package.

Immediate custom actions then read those custom tables, check current state (for example, component action state, the state of the machine itself), and serialize the resulting data into a custom action data property. Immediate custom actions are the place to do the logic that needs live state and cannot be determined at build time by a compiler extension. Because immediate custom actions run in the security context of the installing user and outside an installation transaction, they generally do not have permissions to modify the machine and if they fail, the installation simply ends without the need to do any cleanup or rollback.

Deferred custom actions read the custom action data property set by immediate custom actions to know what to do. One way to improve custom action reliability is to make as few decisions as possible in deferred custom actions; instead, implement all the logic in compiler extensions and immediate custom actions and have deferred custom actions simply read the custom action data property in a loop to modify the machine.

The WiX custom actions that modify the machine use this pattern. For example, XmlConfig authoring is validated by the WixUtilExtension compiler extension and translated to rows and columns in the XmlConfig table. The SchedXmlConfig immediate custom action reads the XmlConfig table, constructs a custom action data property based on the XmlConfig table and machine’s state (including checking component state and storing existing file data to support rollback), then schedules the ExecXmlConfig deferred custom action to execute the XML changes and the ExecXmlConfigRollback rollback custom action to roll back the changes.

Testing with Lux

Lux is a WiX extension (and associated tools) that let you write data-driven unit tests for your custom actions. The executive summary: Lux runs your immediate custom actions then validates they set properties to the values you expect.

While it’s a simple approach, if your custom actions are factored as discussed above, validating the properties set by immediate custom actions can validate all the interaction between your custom actions, the MSI package, and MSI itself.

If your custom actions aren’t factored as discussed–for example, if your deferred custom actions expect only an installation directory and have logic to construct file paths from it–then it’s likely that your immediate custom actions don’t have a lot of logic that’s useful to test.

Lux does not help you test the custom action code that actually modifies the machine; for that, continue to use other unit-test frameworks and automated tests. By working only with immediate custom actions, Lux can let MSI run the custom actions as-is, eliminating the need to write custom test doubles for the MSI API. Lux runs from a per-user package so unless you run the tests from an elevated command prompt, none of the custom actions get elevated privileges and therefore cannot modify the machine.

Here’s how Lux works:

  1. You write your unit tests using XML in WiX source files.
  2. The Lux extension converts the XML to a table in a test .msi package.
  3. The Lux custom action runs after all other immediate custom actions and evaluates your unit tests.

Authoring unit tests

Lux supports the following unit tests:

  • Property values
  • Expressions
  • Multi-value properties
  • Name/value-pair properties

Note that you should always author unit tests in fragments separate from your custom action authoring or any other product authoring. If you mix unit tests with other authoring, WiX includes the unit-test data in your "real" installers.

Property value tests

A simple test lets you specify a property to test, a value to test against, and the operator to compare with (which defaults to "equal").

<Fragment>
  <lux:UnitTest CustomAction="TestCustomActionSimple" Property="SIMPLE" Value="[INSTALLLOCATION]" Operator="equal" />
</Fragment>

When the test runs, Lux compares the value of the SIMPLE property against the (formatted) value [INSTALLLOCATION]. If the two match (because the operator is "equal"), the test passes. Legal values of the Operator attribute are:

equal
(Default) Compares Property to Value and succeeds if they are equal.
notEqual
Compares Property to Value and succeeds if they are NOT equal.
caseInsensitiveEqual
Compares Property to Value and succeeds if they are equal (ignoring case).
caseInsensitiveNotEqual
Compares Property to Value and succeeds if they are NOT equal (ignoring case).

Test conditions

Conditions let you validate code paths in your custom action. For example, if your custom action behaves differently on Windows XP than it does on Windows Vista and later, you can create two tests with mutually exclusive conditions:

<Fragment>
  <lux:UnitTest CustomAction="TestCustomActionSimple" Property="SIMPLE" Value="[INSTALLLOCATION]">
    <lux:Condition><![CDATA[VersionNT < 600]]></lux:Condition>
  </lux:UnitTest>
  <lux:UnitTest CustomAction="TestCustomActionSimple" Property="SIMPLE" Value="[INSTALLLOCATION]">
    <lux:Condition><![CDATA[VersionNT >= 600]]></lux:Condition>
  </lux:UnitTest>
</Fragment>

If a test has a condition, the test runs only if its condition is true.

Expression tests

Expression tests let you test any valid MSI expression. If the expression is true, the test passes. If the expression is false or invalid, the test fails.

<Fragment>
  <lux:UnitTest CustomAction="TestCustomActionSimple">
    <lux:Expression>NOT MsiSystemRebootPending AND SIMPLE</lux:Expression>
  </lux:UnitTest>
</Fragment>

Multi-value property tests

Because deferred custom actions can access only a single custom-action data property, custom actions that need more than one piece of data encode it in a single string. One way is to have the immediate custom action separate multiple elements with a known separator character, then have the deferred custom action split the string at those separate characters. Lux supports such separators using the ValueSeparator and Index attributes.

<Fragment>
  <lux:UnitTest CustomAction="TestCustomActionMultiValue" Property="MULTIVALUE" ValueSeparator="*">
    <lux:Condition>VersionNT</lux:Condition>
    <lux:UnitTest Index="0" Value="1" />
    <lux:UnitTest Index="1" Value="[INSTALLLOCATION]">
      <lux:Condition>NOT Installed</lux:Condition>
    </lux:UnitTest>
    <lux:UnitTest Index="2" Value="WIXEAST" />
  </lux:UnitTest>
</Fragment>

A condition under the parent UnitTest element applies to all individual unit tests. Override it with a Condition child element.

Name/value-pair property tests

Another way of providing multiple values to a deferred custom action is to combine name/value pairs into a single string. Lux supports name/value-pair properties using the NameValueSeparator and Index attributes.

<Fragment>
  <lux:UnitTest CustomAction="TestCustomActionNameValuePairs" Property="NAMEVALUEPAIRS" NameValueSeparator="#">
    <lux:UnitTest Index="InstallationRoot" Value="[INSTALLLOCATION]" />
    <lux:UnitTest Index="Developers" Operator="caseInsensitiveNotEqual" Value="WIXEAST" />
  </lux:UnitTest>
</Fragment>

Building test packages

Lux unit tests run from a minimal package that includes just your unit tests and the resources they need to run. Because Lux runs only immediate custom actions, it doesn’t need a full, per-machine package that includes all the files and other resources to be installed. Such a minimal package saves build time but does require that your WiX source code be well modularized with fragments. For example, you should always author unit tests in fragments separate from any other authoring. If you mix unit tests with other authoring, WiX includes the unit-test data in your "real" installers. Likewise, any other WiX authoring included in unit-test fragments is included in test packages.

Lux comes with a tool that simplifies the creation of test packages. Its name is lux.exe. To use lux.exe:

  1. Compile the source file containing your unit tests.
  2. Run lux.exe on the .wixobj file and specify a source file for the test package.
  3. Compile the test package source.
  4. Link the test package .wixobj with the unit tests .wixobj.

For example:

candle -ext WixLuxExtension CustomActions.wxs
lux CustomActions.wixobj -out LuxSample1_test.wxs
candle -ext WixLuxExtension LuxSample1_test.wxs
light -ext WixLuxExtension LuxSample1_test.wixobj CustomActions.wixobj -out LuxSample1_test.msi

Lux also includes an MSBuild task and .targets file to let you build test packages from the same .wixproj you use to build your installers. To build a test package, build the BuildTestPackage target using MSBuild 3.5:

%WINDIR%\Microsoft.NET\Framework\v3.5\MSBuild.exe /t:BuildTestPackage

Running unit tests

After building the test package, you can run it with logging enabled to capture test results:

msiexec /l test1.log /i bin\Debug\LuxSample1_test.msi

Search the log for WixRunImmediateUnitTests to see test results and other logging from the Lux custom action.

Nit: The Lux test runner

Lux also includes Nit, a console program that monitors the logging messages emitted by unit tests and reports success or failure. To use Nit on your test packages, just specify their filenames as arguments to nit.exe. For example:

nit LuxSample1_test.msi

Lux also lets you run Nit on your test packages from the same .wixproj you use to build your installers. To run a test package under Nit, build the Test target using MSBuild 3.5:

%WINDIR%\Microsoft.NET\Framework\v3.5\MSBuild.exe /t:Test

The test package will be built before the tests are run, if necessary. The output looks like the following, with failing tests highlighted in red as build errors:

Test:
  Microsoft (R) Windows Installer Xml Unit Test Runner version 3.5.1204.0
  Copyright (C) Microsoft Corporation. All rights reserved.
  
  Test luxB21F0D12E0701DBA30FFB92A532A5390 passed: Property 'SIMPLE' matched expected value '[INSTALLLOCATION]'.
  Test TestConditionBeforeVista passed: Property 'SIMPLE' matched expected value '[INSTALLLOCATION]'.
  Test TestConditionVistaOrLater passed: Property 'SIMPLE' matched expected value '[INSTALLLOCATION]'.
  Test TestExpressionTruth passed: Expression 'NOT MsiSystemRebootPending AND SIMPLE' evaluated to true.
nit.exe : error NIT8103: Test luxA6D27EC5903612D7F3786FF71952E314 failed: Property 'MULTIVALUE' expected value '2' but actual value was '1'.
  Test lux210257649C16AFA33793F1CDDF575505 passed: Property 'MULTIVALUE' matched expected value '[INSTALLLOCATION]'.
nit.exe : error NIT8103: Test lux402940A90D3ADAD181D599AB8C260FA0 failed: Property 'MULTIVALUE' expected value 'xxxWIXEAST' but actual value was 'WIXEAST'.
  Test lux453EC8DB458A8F66F0D22970CFF2AE99 passed: Property 'NAMEVALUEPAIRS' matched expected value '[INSTALLLOCATION]'.
  Test lux20CB4F88795F22D15631FD60BA03AFEB passed: Property 'NAMEVALUEPAIRS' matched expected value 'WIXWEST'.
nit.exe : error NIT8102: 2 tests failed. 7 tests passed.
Done Building Project "C:\Delivery\Dev\wix35\src\lux\samples\LuxSample1\LuxSample1.wixproj" (Test target(s)) -- FAILED.

Build FAILED.

"C:\Delivery\Dev\wix35\src\lux\samples\LuxSample1\LuxSample1.wixproj" (Test target) (1) ->
(Test target) -> 
  nit.exe : error NIT8103: Test luxA6D27EC5903612D7F3786FF71952E314 failed: Property 'MULTIVALUE' expected value '2' but actual value was '1'.
  nit.exe : error NIT8103: Test lux402940A90D3ADAD181D599AB8C260FA0 failed: Property 'MULTIVALUE' expected value 'xxxWIXEAST' but actual value was 'WIXEAST'.
  nit.exe : error NIT8102: 2 tests failed. 7 tests passed.

    0 Warning(s)
    3 Error(s)

Time Elapsed 00:00:07.87

FAQ

Are these really unit tests? They look a lot like Fit tests.
Fit tests are tabular and data-driven, so they have a lot in common with Lux’s unit tests. But fit tests are focused on high-level outputs, whereas unit tests are low-level developer tests.
Using the custom action code as-is sounds good, but are there any limitations with that approach?
Yes. Because you are running the actual custom action, any code paths that rely on machine state reflect the state of the machine you run the tests on. For example, code that has different behavior on different versions of Windows runs only one way, just like it does in a normal installer. You can add debug code that looks for the presence of the WIXLUXTESTPACKAGE property; it’s set to 1 in a test package.
I have unit tests that fail because directory properties are being returned as empty strings. Why?
The most likely cause is that your directories are defined as children of your installer’s Product element. Lux.exe builds its own Product element to product a minimal test package, so none of the resources defined in your Product are available to the unit tests. The simplest solution is to move those resources to their own Fragment.
Do I have to write my custom actions in C++?
No, Lux works with any immediate custom actions, regardless of the language they’re written in, including MSI type 51 property-setting custom actions.

Verbose logging from WcaUtil

WcaUtil is a static library of convenience functions for writing custom actions in native C++. One of the more useful functions is WcaLog, which writes messages into the Windows Installer log. The first argument to WcaLog is the level of the message:

  • LOGMSG_TRACEONLY: Written to the log only in debug builds for debugging custom actions.
  • LOGMSG_VERBOSE: Written to the log only when verbose logging is enabled.
  • LOGMSG_STANDARD: Always written to the log.

WcaLog considers verbose logging enabled whenever any of the following is true:

  • LOGVERBOSE property: There’s a property in your package named LOGVERBOSE, regardless of its value.
  • MsiLogging property: There’s a property in your package named MsiLogging that contains a V character.
  • Logging policy: The logging policy is set and contains a V character.

Otherwise, messages tagged with LOGMSG_VERBOSE will be ignored.

The second argument is a printf-style format string so there are a variable number of arguments (zero or more) after it which specify the values referred to in the format string. For example:

WcaLog(LOGMSG_VERBOSE, "App: %S found running, %d processes, setting ‘%S’ property.", wzApplication, cProcessIds, wzProperty);

Note that WcaLog uses ANSI strings for the format string and its arguments, so if you want to log a Unicode string, you need to use the %ls or %S field characters.

Testing your deferred and rollback custom actions

When you include deferred custom actions — that somehow modify the machine — in your setup, you have two big responsibilities:

  1. Provide rollback custom actions that "undo" what the deferred CAs do so that the installation transaction is actually transactional.
  2. Test.
  3. Test.
  4. Test.

OK, so numbers 2 through 4 are kinda the same but not really: Even a simple installation (say, without patching or upgrades) has three different scenarios you need to test when you have deferred/rollback custom actions:

  1. Installation rollback.
  2. Repair rollback.
  3. Uninstallation rollback.
  4. All of the above.

The right behavior for each kind of rollback is usually the opposite action. Rolling back installation is uninstallation. Rolling back uninstallation is installation. Rolling back repair is usually installation. Mixing installation, repair, and uninstallation is possible if your package has user-selectable features and users go into maintenance mode to turn on and off features. And, of course, it’s always an option from the msiexec.exe command line using the ADDLOCAL/ADDSOURCE/ADDDEFAULT, REMOVE, and REINSTALL properties.

Testing rollback means testing failure

Windows Installer initiates rollback when an action fails, so to test rollback you need to cause a failure. WiX includes an easy way to trigger failure: The WixFailWhenDeferred custom action, part of WixUtilExtension, triggers a failure when it’s executed. Include it in your package by referencing WixUtilExtension (in your Votive .wixproj or via the -ext switch to the light.exe command line) and adding a CustomActionRef to your package authoring:

<CustomActionRef Id="WixFailWhenDeferred" />

WixFailWhenDeferred automatically schedules itself in InstallExecuteSequence before InstallFinalize, with a condition of:

WIXFAILWHENDEFERRED=1

The condition means that you can have one package to test all the different possible combinations of "normal" installation and rollback. Just pass the WIXFAILWHENDEFERRED=1 property value on the msiexec.exe command line to trigger rollback. For example:

msiexec /qb- /i intermediate.msi /L*vx installfail.log WIXFAILWHENDEFERRED=1
msiexec /qb- /i intermediate.msi /L*vx install.log
msiexec /qb- /fvamus intermediate.msi /L*vx repairfail.log WIXFAILWHENDEFERRED=1
msiexec /qb- /fvamus intermediate.msi /L*vx repair.log
msiexec /qb- /x intermediate.msi /L*vx uninstallfail.log WIXFAILWHENDEFERRED=1
msiexec /qb- /x intermediate.msi /L*vx uninstall.log

WixFailWhenDeferred has been in WiX v3 weekly releases since April.

Google Earth setup experience

Google announced the release of Google Earth 4.3 today. Given the recent release of their WiX-based setup for the Google App Engine SDK, I had to give it a shot. (It helps that my day job also deals with 3-D terrain imagery.)

When you click the link to “Download Google Earth 4.3” (and accept the EULA), you download not Google Earth but “Google Updater.exe.” Run it and it starts downloading the Google Earth installer.

Personally, I much prefer to download the actual installers for the software I use. Some of it’s purely practical: I can stick it on a network and put it on multiple computers (EULA permitting, of course) without waiting for multiple downloads. Perhaps more importantly, for the paranoid among us, is the ability to virus-scan the installers. (Google Updater requests elevation, so it has admin rights to install multiple packages.)

Google Updater also runs as a startup app, optionally showing an icon in the increasingly-unusable system tray. (Yes, I know it’s technically the “notification area,” but come on, who calls it that?) I don’t think I need 24/7 instant access to software that doesn’t get updated that often. In fact, I’m sure I don’t.

Naturally, installing a packaging system with update capabilities is a boon to many users (and Google itself, of course, which has a nicely visible entry point to suggest additional apps for download). Apple does it with iTunes and Microsoft does it with Windows Live. Luckily, Google Updater has its own entry in Add/Remove Programs so you can remove it without impact to Google Earth.

Google Earth installer

The Google Earth installer is built with InstallShield, using its support for “dynamic file linking.” (If you haven’t used it, think of running Heat or Tallow with every build.) Interestingly, it uses the CAQuietExec custom action from WiX and has wixca.dll in the Binary table.

The .msi package fails ICE validation, with errors in ICE03, ICE15, ICE34, ICE38, ICE43, ICE44, ICE57, ICE64, and ICE99, and warnings in ICE45, ICE60, ICE82, ICE86, and ICE91.

There are 36 custom actions, some of which are mildly disturbing:

  • registerFlashSOL is a deferred, no-impersonate custom action that runs an included .exe with a “-install” command-line switch. Oh joy: self-reg.
  • InstallToolBarCA is an immediate custom action that runs an installed .exe. It’s not scheduled and is only available from the UI sequence – the one that’s suppressed because Google Updater runs the installation silently.
  • SetGEUserStats is another custom action run only from the suppressed – but still included – UI sequence.
  • SET_RES_READ_ONLY sets QtExecCmdLine to run the WiX CAQuietExec custom action. What is it hiding? It’s running attrib.exe to turn off the read-only attribute on a recursive set of files/directories. Doesn’t the File table let you control the read-only attribute? Yes, it does, but apparently not when you use InstallShield’s dynamic file linking to harvest a directory tree at build time. In a previous life, I used that functionality and turned off the read-only attributes in the build script before harvesting. Doing it as part of installation is bad karma on Google’s part.

Overall, it’s not a bad installer, but I hope Google cleans it up a bit before it loses the “beta” label.