We're big fans of JetBrains' TeamCity product here at the HQ and use it to produce our nightly builds, all of the Contour and Courier builds. We're also using it to deploy our upcoming Concorde portal and the Our Umbraco sandbox site (and the live site in the future).
In the past few months we've worked hard to bring more quality into the core of Umbraco and our other products and have introduced unit tests using the NUnit framework. This is working out great and whenever we see new tests failing either locally or on TeamCity, we know that we messed something up without realizing it. Cool, that really helps making our releases of higher quality.
There's one problem however, that I noticed recently: deploys and nightlies appeared online while the tests for them were actually failing. So anybody downloading those releases got a version with known bugs, not good.
When I looked into this, the cause was evident: TeamCity runs NUnit, NUnit reports that it did it's job and found x failing tests and y succeeding test. Nothing failed in NUnit itself, so it reports back: "I did my job succesfully, nothing to see here". In other words: The errorlevel was 0.
Thing is.. TeamCity can cancel the next steps if the errorlevel is higher than 0, but NUnit tells TeamCity: "I just did my job there were no exceptions in my code. There might have been in the code I was testing, but I functioned just fine. Errorlevel: 0".
Apparently this issue has been known for a while and JetBrains don't currently have plans to make modifications.
I had a terribly difficult time finding a workaround, but I finally did.. and it's really pretty easy!
I've set up a TestProject on BitBucket and if you look in the tools folder I added the most minimal set of files needed to run NUnit.
Update: As Eugene points out in the comments another workaround is also possible, that will give you the test output while it is running. The workaround below will only give the output after the tests are done.
To make this happen, copy the TeamCity NUnit addin dlls into your NUnit folder and then run the NUnit console. It's basically the same what I'm doing below but with the added benefit of getting real-time test results.
See the Teamcity2.proj file in the TestProject for an example as to how to configure this.
In TeamCity I can then define a build step of type Command Line and have it run the tests, outputting a results XML file, in the case of the testproject the command looks like this:
tools\NUnit\nunit-console.exe TestProject.Tests\bin\Debug\TestProject.Tests.dll /result=TestProject.Tests\NUnitTestResults.xml
This results in a proper Errorlevel. 0 if there were no tests failing, something else if there are tests failing. Result: the next build step is not performed, awesome!
But.. TeamCity gives you a super handy overview of tests results and allows you to drill down into them when something fails, this was now missing, there's just a message "process exited with code 1". Not very helpful.
Then, I discovered: Additional Build Features. You can find this option in your TeamCity project configuration on config step 3, under the build steps. One of the build features you can add is XML report processing. And that's where our NUnitTestResults.xml file comes in.
Conveniently, there is a report type called NUnit and all you have to do next is tell the report processor where to find it, in the monitoring rules, in my case I had to enter: TestProject.Tests\NUnitTestResults.xml
Et voliá! The build result is once again complete, including the failing unit tests, and the next step will be cancelled because this step is correctly reporting an errorlevel that is not 0.
I hope this helps some people out there who are as frustrated about this as I was!
In the TestProject, I also have a TeamCity.proj file that uses the MSBuild Community tasks library to run NUnit and that produces the exact same result. So you don't have to do this using a command line, you can incorporate the same thing in an MSBuild file if you want.