MSTest command line code coverage (with NAnt)

Introduction

If you have an automated build system and you are using the .net + visual studio stack you probably know how much pain is to gather code coverage from command line, and moreover how painful it is to convert the final report to a usable format. In order to collect code coverage information using visual studio command line tools, we need a couple of utilities. Those utilities come only in the Team Edition, but are also available to download from microsoft's site.

Test execution

Let's start by executing our tests from command line using MSTest.exe. Let's see an example using NAnt:

           <exec program="MSTest.exe" verbose="true" failonerror="false">
                <arg line="/testmetadata:${vsmdi}" />
                <arg line="/testlist:${testlist}" />
                <arg line="/resultsfile:${outdir}${suitename}.trx" />
            </exec>

The previous code executes MSTest by passing it a vsmdi file which contains all the tests grouped in lists. This will generate a trx with the result of the test execution.

Instrumenting

Once our tests are running and passing al very nicely we can move to the code coverage thing. In order to collect code coverage information first we need to instrument our libraries/executables (See more at: http://msdn.microsoft.com/en-us/library/ms243141.aspx). If our assemblies have strong names, we'll also need to re-sign them or disable the signing verification.

Visual Studio give us some tools (poorly documented IMHO) called "Performance Tools" and you can find them at %PROGRAM FILES%\Visual Studio {n}\Team Tools\Performance Tools directory.

The process to instrument an assembly uses the tool called vsintr.exe. We need to tell vsinstr that we want to collect coverage info and the assembly to instrument (${assembly}).

            <!-- Instrument assembly -->
            <exec program="C:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\Performance Tools\VsInstr.exe">
                <arg line="/Coverage" />
                <arg line="${assembly}" />
            </exec>

If you have a lot of assemblies to instrument... may I recommend using a fileset and a foreach loop? :)

There is a special extra step we need to execute if our assemblies have a strong name, that is disabling signing verification. You can achieve that by using the sn.exe utility with the -Vr option.

            <!-- Register <assembly> for verification skipping -->
            <exec program="C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\sn.exe" failonerror="false">
                <arg line="-Vr" />
                <arg value="${assembly}" />
            </exec>

Now that all of our assemblies have been instrumented we can now execute tests and collect the data.

Data Collection

The data collection is performed by a process called vsperfmon.exe. This process should be running during the entire test session and shutted down at the end to generate the report.

To start collecting coverage data the process must be started with the Coverage and Output options as shown below.

    <!-- Start monitor in coverage mode -->
    <exec program="C:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\Performance Tools\VsPerfCmd.exe">
        <arg line="/Start:Coverage" />
        <arg line="/Output:${outdir}\Coverage\file.coverage" />
    </exec>

That will spawn a background process that will be listening for coverage data. Since the program will be running in background you cannot execute two coverage collect sessions with the same user, and a new session cannot be started until the previous have being stopped. Note: When the process spawns, it locks the output file, making "difficult" to delete it (take that in consideration if your test session suddenly aborts and you need to start a new one, you must always stop the VSPerfMon.exe process)

After we're done with the tests we need to stop the background process. To shut it down we run vsperfmon with the shutdown option.

    <!-- Shutdown monitor and save coverage file -->
    <exec program="C:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\Performance Tools\VsPerfCmd.exe">
        <arg line="/Shutdown" />
    </exec>

That will write and flush all the coverage information to the file we pass on the output option when we started it. The result, a something.coverage file with all the coverage information.

Report

The format of the coverage file is propietary and nearly no documented at all, so it's very difficul to read directly. Another problem is that this coverage file doesn't contain all the neccesary information, even after the vsperfmon is down, we still need the instrumented images to be able to read the report. This means that if you take the coverage file to a different computer, you won't be able to open it because it doesn't have the instrumented assemblies at reach. Awful design choice... but who am I to talk about it...

Fortunately, we have this (great?) visual-coverage tool which comes in handy. The tool will take as input the coverage file, and will output a clover (xml) or html report that is independent of the computer or assemblies).

    <!-- Transforms coverage report (from VsPerfCmd) to clover (bamboo). -->
    <exec program="C:\BuildTools\VisualCoverage\bin\VisualCoverage.exe">
        <arg line="--input ${outdir}\Coverage\file.coverage" />
        <arg line="--clover ${outdir}\Coverage\clover.xml" />
        <arg line="--html ${outdir}\Coverage\report.html" />
    </exec>

Now we have a nice report to show to our stakeholders!.