Make TeamCity respect failing NUnit tests and stop the build

Wednesday, January 16, 2013 by Sebastiaan Janssen


We're big fans of JetBrains' TeamCity product here at the HQ and use it to produce our nightly builds, all of the Contour and Courier builds. We're also using it to deploy our upcoming Concorde portal and the Our Umbraco sandbox site (and the live site in the future).

In the past few months we've worked hard to bring more quality into the core of Umbraco and our other products and have introduced unit tests using the NUnit framework. This is working out great and whenever we see new tests failing either locally or on TeamCity, we know that we messed something up without realizing it. Cool, that really helps making our releases of higher quality.

Problem

There's one problem however, that I noticed recently: deploys and nightlies appeared online while the tests for them were actually failing. So anybody downloading those releases got a version with known bugs, not good.

When I looked into this, the cause was evident: TeamCity runs NUnit, NUnit reports that it did it's job and found x failing tests and y succeeding test. Nothing failed in NUnit itself, so it reports back: "I did my job succesfully, nothing to see here". In other words: The errorlevel was 0.

Thing is.. TeamCity can cancel the next steps if the errorlevel is higher than 0, but NUnit tells TeamCity: "I just did my job there were no exceptions in my code. There might have been in the code I was testing, but I functioned just fine. Errorlevel: 0".

Apparently this issue has been known for a while and JetBrains don't currently have plans to make modifications.

Workaround

I had a terribly difficult time finding a workaround, but I finally did.. and it's really pretty easy!

I've set up a TestProject on BitBucket and if you look in the tools folder I added the most minimal set of files needed to run NUnit. 

Update: As Eugene points out in the comments another workaround is also possible, that will give you the test output while it is running. The workaround below will only give the output after the tests are done.

To make this happen, copy the TeamCity NUnit addin dlls into your NUnit folder and then run the NUnit console. It's basically the same what I'm doing below but with the added benefit of getting real-time test results. 

See the Teamcity2.proj file in the TestProject for an example as to how to configure this.


In TeamCity I can then define a build step of type Command Line and have it run the tests, outputting a results XML file, in the case of the testproject the command looks like this:

tools\NUnit\nunit-console.exe TestProject.Tests\bin\Debug\TestProject.Tests.dll /result=TestProject.Tests\NUnitTestResults.xml

2013-01-16_161203

This results in a proper Errorlevel. 0 if there were no tests failing, something else if there are tests failing. Result: the next build step is not performed, awesome!

But.. TeamCity gives you a super handy overview of tests results and allows you to drill down into them when something fails, this was now missing, there's just a message "process exited with code 1". Not very helpful.

Then, I discovered: Additional Build Features. You can find this option in your TeamCity project configuration on config step 3, under the build steps. One of the build features you can add is XML report processing. And that's where our NUnitTestResults.xml file comes in.

Conveniently, there is a report type called NUnit and all you have to do next is tell the report processor where to find it, in the monitoring rules, in my case I had to enter: TestProject.Tests\NUnitTestResults.xml

2013-01-16_161648

Et voliá! The build result is once again complete, including the failing unit tests, and the next step will be cancelled because this step is correctly reporting an errorlevel that is not 0.

2013-01-16_155206

I hope this helps some people out there who are as frustrated about this as I was! 

In the TestProject, I also have a TeamCity.proj file that uses the MSBuild Community tasks library to run NUnit and that produces the exact same result. So you don't have to do this using a command line, you can incorporate the same thing in an MSBuild file if you want.

16 comment(s) for “Make TeamCity respect failing NUnit tests and stop the build”

  1. Gravatar ImageEugene Petrenko Says:

    Please also consider the approach that would also workaround the issue, put it would also provide on-the-fly tests reporting. http://confluence.jetbrains.net/display/TCD7/TeamCity+Addin+for+NUnit

  2. Gravatar ImageJohn Says:

    Or simpler, just use PowerShell and the REST-API

    http://stackoverflow.com/questions/13212748/how-to-integrate-mstest-in-your-teamcity-build-process/

  3. Gravatar ImageSebastiaan Janssen Says:

    @Eugene Nice, I thought this was an old addin though, therefore I disregarded it. I might have to check it out.

  4. Gravatar ImageSebastiaan Janssen Says:

    @Eugene actually, isn't that just the addin that got built in and is not working? There's no download link I can find for it anywhere..

    @John Yes, that might work, but it's pretty dirty adding credentials in a build step. Besides, if I understand Pavel's comment correctly that might add have it's own issues too:
    http://youtrack.jetbrains.com/issue/TW-17002#comment=27-416595

  5. Gravatar ImageMorgan Persson Says:

    @Eugene The built in nunit runner also have the bug that it doesn't stop the build on failure, se the link by Sebastiaan

  6. Gravatar ImageNuno Costa Says:

    you can also tell teamcity to import data from NUnitTestResults.xml using :



    after

  7. Gravatar ImageNuno Costa Says:

    who eat my html ??

    using the with the text property = ##teamcity[importData type='nunit' path='$(OutputPath)\NUnitReport.xml']

    afeter running the NUnit task

  8. Gravatar ImageMatt Says:

    I'm using TeamCity - to run builds using a build script crafted in F# using FAKE (https://github.com/fsharp/fake).

    Behind the scenes, NUnit test runs are fired off using the NUnit console - and kicks out instantly... and the XML output I read back into TeamCity to get detailed information as well (although I could just look at the FAKE log as well).

    Each of our CI build configurations have like 1 or 2 steps only.

    Obviously the beauty of using the build script is that all the developers can run the same process as the build server as well (well you can pass in a parameter to the build script to run a CI build... which does the same as the developer build, but also packages and publishes thing to Nuget etc - but could equally manage some other kind of integration or end-2-end testing etc).

  9. Gravatar ImageAndy Says:

    I know it doesn't have such a great reputation but would just note in our builds using MSTest, the build stops on failing tests just fine. Though you do need VS.Net on your CI server unfortunately.

  10. Gravatar ImageChi Chan Says:

    I ran into a similar issue today with TeamCity 8.1. But I was able to abort the build by selecting "Only if build status is successful" under Execute step.

    So if I have three steps:

    1) Build
    2) Run NUnit test
    3) Deploy files

    I would set Execute step = Only if build status is successful in step 3)

  11. Gravatar ImageSebastiaan Janssen Says:

    @Chi Yep, this is new in TeamCity 8, making this blog post obsolete for v8+ (yay!)

  12. Gravatar Imagecialis_dosage Says:

    Hello!

  13. Gravatar Imagecialis_dosage Says:

    Hello!

  14. Gravatar Imagecialis_dosage Says:

    Hello!

  15. Gravatar Imagecialis_sale Says:

    Hello!

  16. Gravatar Imagefree_viagra Says:

    Hello!

Leave a comment