adrift in the sea of experience

Friday, February 25, 2011

Executing visual studio 2010 unit tests without installing visual studio

In a previous post I already explained how to get your unit tests running on a build machine with mstest.exe, without having to install Visual Studio 2008. We recently upgraded to Visual Studio 2010 so I had to repeat the exercise.

This time I wrote a batch script which does most of the work. Just run the script on a machine where Visual Studio 2010 is installed, and it will create a "mstest" folder with the necessary binaries and registry entries. Copy the folder to your build machine, prepare the registry by executing mstest.reg and you're good to run mstest.exe like this:

mstest /noisolation /testcontainer:"path\to\testproject.dll"

This will return an error code if any of the tests fail.

As an anonymous commenter on my previous post pointed out, you should be careful of the license implications. In our case it is most likely OK because we have a site license. But even so it can be useful to avoid a full Visual Studio install: it saves a lot of disk space, and you don't have to spend time babysitting the installation. Or multiple installations, if you have a cluster of build machines!

update: thanks to Frederic Torres for pointing out that the script doesn't work on a 64-bit Windows, and suggesting a fix! It should work now. Apparently there are still problems with 64-bit machines. But I don't have a 64-bit machine without VS2010 to test with, so I'll have to leave it as an exercise for the reader...

19 comments:

Thomas Schreiner said...

Thanks for the Post. But I can't figure out where on the BuildServer I have to put the MSTest directory. That results in the inability to build our solution on the buildserver as msbuild does not find the MsTest-Assemblies and fails.

So where do I put the mstest folder or how do I tell MsBuild where the MSTest assemblies are located?

Thanks,
Thomas

Wim Coenen said...

@Thomas: in my case I have a batch script which invokes mstest.exe, logs the output and checks the return code. I don't use an msbuild task to execute the tests.

Wim Coenen said...

@Thomas: I'm not sure which msbuild task you are using, but with "Exec" you should be able to do the same thing as my batch script.

Sneha Saggam said...

This solution worked for me. But i am running mstest with testlist and testmetadata switches. When I run mstest without visual studio on some other box, I get the error that /testlist and /testmetadata are invalid switches.

Wim Coenen said...

@Sneha: to enable those switches, you can copy the registry entries at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\10.0\Licenses

You can use the export/import in regedit to copy them all at once.

Sneha Saggam said...

@Wim - I was able to run the fix on a 32-bit machine, but the fix is not working on a 64-bit machine.

Thanks,
Sneha

Wim Coenen said...

@Sneha: on a 64-bit machine, the registry entries for 32-bit applications are in another location:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\Licenses

So if you export the entries from a 32-bit machine, you have to edit the file to insert "Wow6432Node\" in the keys before importing on a 64-bit machine.

That being said, I don't have a 64-bit machine without Visual Studio so I can't test any of this.

Sneha Saggam said...

@Wim - Changing the registry keys made it work on a 64-bit machine.
Thanks for the information.

Sneha

Armando Jr. said...

I'm having the following problem:
Failed to initialize the unit test extension 'CodedUITestAttribute': Failed to create an instance of the TestClassExtensionAttribute ('Microsoft.VisualStudio.TestTools.UITesting.CodedUITestAttribute, Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a') for the unit test extension 'CodedUITestAttribute': Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
bin\release\prjcco.dll
Unable to load the test container 'bin\release\prjcco.dll' or one of its dependencies. Error details: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestTools.UITesting, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

can u help me?
i try set CopyLocal=true, but don't work!!! :/

Karl said...
This comment has been removed by the author.
Karl said...

Thanks for your share.

Arieh Faiga said...

Thanks for the batch file!

I found 3 problem with it:
1. export registry on 64 bit
2. should export the IDE\PublicAssembly directory to cater for Microsoft.VisualStudio.QualityTools.CodedUITestFramework.dll
3. Should export the QT*.exe files which are called by MSTest at the end of the test.


So my updated batch file is as follows:

@echo off
setlocal
set here=%~dp0

mkdir mstest
set targetfolder=%here%mstest

set programs=%programfiles%
if exist "%programfiles(x86)%" set programs=%programfiles(x86)%
set vs2010="%programs%\Microsoft Visual Studio 10.0"
set gac1="%windir%"\assembly
set gac2="%windir%"\Microsoft.NET\assembly

echo === Copying from Visual Studio 2010 install folder...
copy %vs2010%\Common7\IDE\mstest* "%targetfolder%"
copy %vs2010%\Common7\IDE\qt* "%targetfolder%"
copy %vs2010%\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.Quality* "%targetfolder%"
copy %vs2010%\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.Quality* "%targetfolder%"

echo === Copying from %gac1%...
pushd "%gac1%"
dir /s /b *.dll | findstr QualityTools | findstr 10.0.0.0 > %here%tmp.filelist
popd
for /F "tokens=*" %%f in (tmp.filelist) DO copy "%%f" "%targetfolder%"

echo === Copying from %gac2%...
pushd "%gac2%"
dir /s /b *.dll | findstr QualityTools | findstr 10.0.0.0 > %here%tmp.filelist
popd
for /F "tokens=*" %%f in (tmp.filelist) DO copy "%%f" "%targetfolder%"

del tmp.filelist

echo === Exporting registry keys if on 32 bit machine...
regedit.exe /e %targetfolder%\mstest.reg HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\10.0\EnterpriseTools\QualityTools

echo === Exporting registry keys if on 64 bit machine...
regedit.exe /e %targetfolder%\mstest32on64.reg HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\EnterpriseTools\QualityTools

echo === Done. Check output for errors!
exit /b 0

William Jernigan said...

When I didn't use the /noisolation option, mstest would create a folder that it couldn't use while running, then fail to complete. More specifically, it appeared that mstest tried to create a (workspace?) folder called USER@DOMAIN-USERNAME yyyy-mm-dd hh:mm:ss, but the folder is created without the '@' and ':' symbols, which makes mstest incapable of finding it and finishing its job. Do you know why this happens?

Joel: Adventure Awaits. said...
This comment has been removed by the author.
Cullen Johnson said...

Thanks! Modifications for VS 2012 on a 64-bit machine:

@echo off
setlocal
set here=%~dp0

mkdir mstest
set targetfolder=%here%mstest

set programs=%programfiles%
if exist "%programfiles(x86)%" set programs=%programfiles(x86)%
set vs2012="%programs%\Microsoft Visual Studio 11.0"
set gac1="%windir%"\assembly
set gac2="%windir%"\Microsoft.NET\assembly

echo === Copying from Visual Studio 2012 install folder...
copy %vs2012%\Common7\IDE\mstest* "%targetfolder%"
copy %vs2012%\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.Quality* "%targetfolder%"

echo === Copying from %gac1%...
pushd "%gac1%"
dir /s /b *.dll | findstr QualityTools | findstr 11.0.0.0 > %here%tmp.filelist
popd
for /F "tokens=*" %%f in (tmp.filelist) DO copy "%%f" "%targetfolder%"

echo === Copying from %gac2%...
pushd "%gac2%"
dir /s /b *.dll | findstr QualityTools | findstr 11.0.0.0 > %here%tmp.filelist
popd
for /F "tokens=*" %%f in (tmp.filelist) DO copy "%%f" "%targetfolder%"

del tmp.filelist

echo === Exporting registry keys...
regedit.exe /e %targetfolder%\mstest.reg HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\11.0\EnterpriseTools\QualityTools

echo === Done. Check output for errors!
exit /b 0

aparna john said...

dsHi, nice method for Executing visual studio 2010 unit tests without installing visual studio.I didn't use unit tests to execute.Thanks...

-Aparna
Theosoft

Poorvi said...

Any idea how can we modify this script for using vstest?

Alain said...

These are the changes I need to make for Windows 8 + VS 2012 + 64bit. (Some of the quotation marks were messed up or missing in previous comments):


@echo off
setlocal
set here=%~dp0
mkdir mstest
set targetfolder=%here%mstest
set programs=%programfiles%
if exist "%programfiles(x86)%" set programs=%programfiles(x86)%
set vs2012="%programs%\Microsoft Visual Studio 11.0"
set gac1= "%windir%\assembly"
set gac2= "%windir%\Microsoft.NET\assembly"

echo === Copying from Visual Studio 2012 install folder...
copy %vs2012%\Common7\IDE\mstest* "%targetfolder%"
copy %vs2012%\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.Quality* "%targetfolder%"

echo === Copying from %gac1%...
pushd %gac1%
dir /s /b *.dll | findstr QualityTools | findstr 11.0.0.0 > "%here%tmp.filelist"
popd
for /F "tokens=*" %%f in (tmp.filelist) DO copy "%%f" "%targetfolder%"

echo === Copying from %gac2%...
pushd %gac2%
dir /s /b *.dll | findstr QualityTools | findstr 11.0.0.0 > "%here%tmp.filelist"
popd
for /F "tokens=*" %%f in (tmp.filelist) DO copy "%%f" "%targetfolder%"

del tmp.filelist

echo === Exporting registry keys...
regedit.exe /e "%targetfolder%\mstest.reg" HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\11.0\EnterpriseTools\QualityTools

echo === Done. Check output for errors!
exit /b 0

Poorvi said...

I am getting the below exception with the bat code above:

"Exception has been thrown by the target of invocation"

I am using a simple console print test.