adrift in the sea of experience

Tuesday, November 18, 2008

Executing visual studio unit tests without installing visual studio

UPDATE: I've written a new post for Visual Studio 2010.

The professional edition of visual studio 2008 has a nicely integrated unit testing framework. I suppose visual studio team server will run these for you whenever somebody checks in a source code change. Unfortunately, we don't have team server or the team edition of visual studio at work.

This lead me to look for a way to execute these unit tests on our build server. "build server" may be a bit of an euphemism; it is little more than a virtual machine with a collection of python and batch scripts and it doesn't have visual studio installed. In case you are wondering, compiling C# projects and solutions from the command line with msbuild works fine without installing visual studio. I wanted a similar solution for the unit tests.

It turns out that there is a command line utility called mstest.exe that can be used to run these tests. This utility comes with visual studio. It is not straightforward to get it working without installing visual studio. Here's what I did to get mstest.exe running on our build server:

1) Created a \tools\mstest folder on the buildserver
2) Copied %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe
3) Copied these files from %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies

Microsoft.VisualStudio.QualityTools.AgentObject.dll
Microsoft.VisualStudio.QualityTools.CommandLine.dll
Microsoft.VisualStudio.QualityTools.Common.dll
Microsoft.VisualStudio.QualityTools.ControllerObject.dll
Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.AssemblyResolver.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip.dll
Microsoft.VisualStudio.QualityTools.TMI.dll


4) Copied these files from c:\windows\assembly\gac_msil subfolders. This requires you to use the command line as windows explorer hides the actual files there.

Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
Microsoft.VisualStudio.QualityTools.Resource.dll


5) Added certain registry keys to the build server registry. You can save these in a .reg file and then simply double click it:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\HostAdapters\VS IDE\SupportedTestTypes]
"{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}"=""

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes]
@="TestTypes"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.SUnitTestService, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#212"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{00000000-0000-0000-0000-000000000000}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestTip, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\Extensions]
".dll"=dword:000000d7
".exe"=dword:000000d7

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\SupportedHostAdapters]
"Smart Device"="Smart Device Host Adapter"
"ASP.NET"="ASP.NET Host Adapter"
"VS IDE"=""

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.SOrderedTestService, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#313"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{700218d8-f6f1-4ec3-be76-a35f72260503}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.AutoSuiteTip, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}\Extensions]
".orderedtest"=dword:0000013c


After that, you can invoke mstest.exe on the command line like this:

mstest /noisolation /testcontainer:"path\to\testproject.dll"


mstest.exe will return an error code if any test fails.

I have not figured out how to get things working without the /noisolation switch. I believe that any problems caused by this switch are rare. In any case it will only cause problems if there are serious issues with the code under test which will need to be fixed anyway.