Executing visual studio unit tests without installing visual studio

UPDATE: I've written a new post for Visual Studio 2010.

The professional edition of visual studio 2008 has a nicely integrated unit testing framework. I suppose visual studio team server will run these for you whenever somebody checks in a source code change. Unfortunately, we don't have team server or the team edition of visual studio at work.

This lead me to look for a way to execute these unit tests on our build server. "build server" may be a bit of an euphemism; it is little more than a virtual machine with a collection of python and batch scripts and it doesn't have visual studio installed. In case you are wondering, compiling C# projects and solutions from the command line with msbuild works fine without installing visual studio. I wanted a similar solution for the unit tests.

It turns out that there is a command line utility called mstest.exe that can be used to run these tests. This utility comes with visual studio. It is not straightforward to get it working without installing visual studio. Here's what I did to get mstest.exe running on our build server:

1) Created a \tools\mstest folder on the buildserver
2) Copied %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe
3) Copied these files from %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies

Microsoft.VisualStudio.QualityTools.AgentObject.dll
Microsoft.VisualStudio.QualityTools.CommandLine.dll
Microsoft.VisualStudio.QualityTools.Common.dll
Microsoft.VisualStudio.QualityTools.ControllerObject.dll
Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.AssemblyResolver.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip.dll
Microsoft.VisualStudio.QualityTools.TMI.dll

4) Copied these files from c:\windows\assembly\gac_msil subfolders. This requires you to use the command line as windows explorer hides the actual files there.

Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
Microsoft.VisualStudio.QualityTools.Resource.dll

5) Added certain registry keys to the build server registry. You can save these in a .reg file and then simply double click it:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\HostAdapters\VS IDE\SupportedTestTypes]
"{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}"=""

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes]
@="TestTypes"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.SUnitTestService, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#212"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{00000000-0000-0000-0000-000000000000}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestTip, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\Extensions]
".dll"=dword:000000d7
".exe"=dword:000000d7

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\SupportedHostAdapters]
"Smart Device"="Smart Device Host Adapter"
"ASP.NET"="ASP.NET Host Adapter"
"VS IDE"=""

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.SOrderedTestService, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#313"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{700218d8-f6f1-4ec3-be76-a35f72260503}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.AutoSuiteTip, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}\Extensions]
".orderedtest"=dword:0000013c

After that, you can invoke mstest.exe on the command line like this:

mstest /noisolation /testcontainer:"path\to\testproject.dll"

mstest.exe will return an error code if any test fails.

I have not figured out how to get things working without the /noisolation switch. I believe that any problems caused by this switch are rare. In any case it will only cause problems if there are serious issues with the code under test which will need to be fixed anyway.

Using System.Diagnostics.StackTrace to leave debugging clues.

When I am debugging, I often find myself wishing I could step back to a certain point, in order to discover at which point some piece of data becomes corrupted. Most of the time, it is possible to set a conditional breakpoint which will break the program at the point where something unexpected happens, and reproduce the bug again.

However, this does not always work. Let's say for example that some piece of code is calling Dispose on an instance which it doesn't own, and therefore shouldn't dispose. Things will go haywire when the instance is used after being disposed. At that point, it may be hard to determine the culprit who called Dispose. And because of the number of instances being created and disposed, it may very well be impractical to set a breakpoint in Dispose and inspect each call to it!

What we need here is a way to see who called Dispose in the past.
The way I do this in .NET is by setting a field to a new System.Diagnostics.StackTrace in Dispose just for debugging. Then when things go haywire and the debugger breaks, I simply read the stacktrace to discover the culprit. This trick can be used in any programming language or environment which supports the runtime generation and inspection of stacktraces.

In visual studio 2005, you could also avoid modifying your code by inserting a tracepoint with the following expression:

Instance with hashcode {GetHashCode()} was disposed. Call stack: $CALLSTACK

Then when things go haywire, you can inspect the hashcode of the corrupted instance and search for it in the trace log.

Don't document that you throw ArgumentException or InvalidOperationException

When programming by contract (which should always be the case, at least implicitly), each method has pre-conditions that the caller should adhere to and post-conditions which the callee guarantees if the pre-conditions were satisfied; if they weren't, all bets are off.

Let's look at an example in the .NET framework: System.XML.XmlReader.ReadElementContentAsString. This method has the precondition that the xml reader should be positioned on an element. However, Microsoft has chosen to actually document this as a postcondition: if the xml reader is not positioned on an xml element, InvalidOperationException is thrown. This is the normal documentation style for the .NET framework, but it has never sat right with me for two reasons.

The first reason is that you shouldn't encourage programmers to rely on avoidable exceptions. I feel it would be better if this was simply documented as a precondition which must never be violated. What happens when the caller violates the contract doesn't need to be documented, so the guarantee of an InvalidOperationException could be omitted. It would of course still be thrown, but as an indication of a programming error. It shouldn't be relied upon.

Another reason is inheritance. One of the great things of programming by contract is that it provides rules for what you can do in an overriding method. Any method must implement the contract of the method it overrides, because the caller may not be aware of the exact type of the callee. However, the overriding method may loosen the pre-conditions. It must merely accept anything that the overridden method would have accepted. It may accept more. Unfortunately, if the preconditions are documented as exceptions, they are now post-conditions which restrict the overriding method without good reason.

Conclusion: don't document ArgumentException and InvalidOperationException if it is sufficient to document pre-conditions.