The professional edition of visual studio 2008 has a nicely integrated unit testing framework. I suppose visual studio team server will run these for you whenever somebody checks in a source code change. Unfortunately, we don't have team server or the team edition of visual studio at work.
This lead me to look for a way to execute these unit tests on our build server. "build server" may be a bit of an euphemism; it is little more than a virtual machine with a collection of python and batch scripts and it doesn't have visual studio installed. In case you are wondering, compiling C# projects and solutions from the command line with msbuild works fine without installing visual studio. I wanted a similar solution for the unit tests.
It turns out that there is a command line utility called mstest.exe that can be used to run these tests. This utility comes with visual studio. It is not straightforward to get it working without installing visual studio. Here's what I did to get mstest.exe running on our build server:
1) Created a \tools\mstest folder on the buildserver
2) Copied %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe
3) Copied these files from %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies
Microsoft.VisualStudio.QualityTools.AgentObject.dll
Microsoft.VisualStudio.QualityTools.CommandLine.dll
Microsoft.VisualStudio.QualityTools.Common.dll
Microsoft.VisualStudio.QualityTools.ControllerObject.dll
Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.AssemblyResolver.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip.dll
Microsoft.VisualStudio.QualityTools.TMI.dll
4) Copied these files from c:\windows\assembly\gac_msil subfolders. This requires you to use the command line as windows explorer hides the actual files there.
Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
Microsoft.VisualStudio.QualityTools.Resource.dll
5) Added certain registry keys to the build server registry. You can save these in a .reg file and then simply double click it:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\HostAdapters\VS IDE\SupportedTestTypes]
"{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}"=""
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes]
@="TestTypes"
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.SUnitTestService, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#212"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{00000000-0000-0000-0000-000000000000}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestTip, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\Extensions]
".dll"=dword:000000d7
".exe"=dword:000000d7
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\SupportedHostAdapters]
"Smart Device"="Smart Device Host Adapter"
"ASP.NET"="ASP.NET Host Adapter"
"VS IDE"=""
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.SOrderedTestService, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#313"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{700218d8-f6f1-4ec3-be76-a35f72260503}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.AutoSuiteTip, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}\Extensions]
".orderedtest"=dword:0000013c
After that, you can invoke mstest.exe on the command line like this:
mstest /noisolation /testcontainer:"path\to\testproject.dll"
mstest.exe will return an error code if any test fails.
I have not figured out how to get things working without the /noisolation switch. I believe that any problems caused by this switch are rare. In any case it will only cause problems if there are serious issues with the code under test which will need to be fixed anyway.
9 comments:
Here are a couple of links that may be helpful:
MSBuild, NAnt, NUnit, MSTest, and frustration
Clean build server with MSTest == FAIL
Those posts aren't that useful. The authors are complaining that they couldn't get mstest.exe to work without installing visual studio.
The point of my post is exactly the opposite: it's possible to make it work.
So very helpful, thanks for the post.
If you're on Win7 (probably Vista too), you may want to mount the gac_msil directory with the `subst` command as the DLLs are in subdirectories.
Also, I should point out that in steps 2 and 3 in this post, the paths should read: `%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\[...]`
@Ammaar Limbada: Thanks for the feedback. I have fixed the paths where the "IDE" folder was missing.
I'd be careful of license implications.
How make this whith MSTest.exe from VisualStudio v10?
I maked all of your's steps, but there is marks:
(Where VS10 is installed):
****
Loading TESTS\WebTestM2M.dll...
Failed to initialize the unit test extension 'CodedUITestAttribute': Failed to create an instance of the TestClassExtensionAttribute ('Microsoft.VisualStudio.TestTools.UITesting.CodedUITestAttribute, Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a') for the unit test extension 'CodedUITestAttribute': Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. ЌҐ г¤ Ґвбп ©вЁ гЄ § л© д ©«.
TestTypes\37e36796-fb51-4610-8d5c-e00ceaa68b9f could not be loaded because the TIP could not be instantiated for the following reason(s): System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.Tips.WebLoadTest.Tip, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. ЌҐ г¤ Ґвбп ©вЁ гЄ § л© д ©«.
File name: 'Microsoft.VisualStudio.QualityTools.Tips.WebLoadTest.Tip, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMarkHandle stackMark, Boolean loadTypeFromPartialName, ObjectHandleOnStack type)
at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, Boolean loadTypeFromPartialName)
at System.RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark)
at System.Type.GetType(String typeName, Boolean throwOnError)
at Microsoft.VisualStudio.TestTools.TestManagement.TipDelayLoader.LoadTip()
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
Test SeeAllCoded cannot be found.
Test SeeAllCoded cannot be found.
Starting execution...
No tests to execute.
****
(where VS10 is not installed):
Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.Common, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference.
at Microsoft.VisualStudio.TestTools.RunnerCommandline.Runner.Main
****
@Pavel: this post is for Visual Studio 2008. For Visual Studio 2010, follow the link at the top of the post.
@Wim: Very informative blog. Instead of using MSTest.exe, can we somehow build the unit_test as unittest.exe instead of unittest.dll. That way we can simply copy to other systems which does not have Visual Studio installed. Then all we need is unittest.exe and code.exe and then we should be able to test our code. Is it possible ?
Post a Comment