At work there's a discussion going on whether we will choose NUnit or Team System Unit Test for our unit tests. Currently we have about 1000+ unit tests written in NUnit and we are most pleased about it. The reason that this discussion has come up is a company-wide adoption of unit testing, something that we are all very happy about.
In order to get a clear and objective view, I gathered all pros and cons for each of these unit test frameworks. Here goes ...
First I made a listing of all attributes:
| NUnit attributes | TS Unit Test attributes |
| [Category(“name”)] | [AssemblyCleanup] |
| [Description(“description”)] | [AssemblyInitialize] |
| [ExpectedException(“myException”)] | [ClassCleanup] |
| [Explicit] | [ClassInitialize] |
| [Ignore(“reason”)] | [CssIteration("name")] |
| [Platform(“name”)] | [TestProperty("name", "value")] |
| [Property(“name”, “value”)] | [CssProjectStructure("name")] |
| [SetUp] | [DataSource("name")] |
| [SetUpFixture] | [DeploymentItem("path")] |
| [Suite] | [Description("text")] |
| [TearDown] | [ExpectedException("myExcepton")] |
| [Test] | [HostType("type")] |
| [TestFixture] | [Ignore] |
| [TextFixtureSetUp] | [Owner("name")] |
| [TextFixtureTearDown] | [Priority(1)] |
| [TestClass] | |
| [TestCleanup] | |
| [TestInitialize] | |
| [TestMethod] | |
| [Timeout(30)] |
and all assertions:
| NUnit assertions | TS Unit Test assertions |
| Assert.AreEqual | Assert.AreEqual |
| Assert.AreNotEqual | Assert.AreNotEqual |
| Assert.AreNotSame | Assert.AreNotSame |
| Assert.AreSame | Assert.AreSame |
| Assert.Contains | Assert.Equals |
| Assert.Fail | Assert.Fail |
| Assert.Greater | Assert.Inconclusive |
| Assert.GreaterOrEqual | Assert.IsFalse |
| Assert.Ignore | Assert.IsInstanceOfType |
| Assert.IsAssignableFrom | Assert.IsNotInstanceOfType |
| Assert.IsEmpty | Assert.IsNotNull |
| Assert.IsFalse | Assert.IsNull |
| Assert.IsInstanceOfType | Assert.IsTrue |
| Assert.IsNaN | |
| Assert.IsNotAssignableFrom | |
| Assert.IsNotEmpty | |
| Assert.IsNotInstanceOfType | |
| Assert.IsNotNull | |
| Assert.IsNull | |
| Assert.IsTrue | |
| Assert.Less | |
| Assert.LessOrEqual | |
| StringAssert.Contains | |
| StringAssert.StartsWith | |
| StringAssert.EndsWith | |
| StringAssert.AreEqualIgnoringCase | |
| StringAssert.IsMatch | |
| CollectionAssert.AllItemsAreInstancesOfType | |
| CollectionAssert.AllItemsAreNotNull | |
| CollectionAssert.AllItemsAreUnique | |
| CollectionAssert.AreEqual | |
| CollectionAssert.AreEquivalent | |
| CollectionAssert.AreNotEqual | |
| CollectionAssert.AreNotEquivalent | |
| CollectionAssert.Contains | |
| CollectionAssert.DoesNotContain | |
| CollectionAssert.IsSubsetOf | |
| CollectionAssert.IsNotSubsetOf | |
| CollectionAssert.IsEmpty | |
| CollectionAssert.IsNotEmpty | |
| FileAssert.AreEqual | |
| FileAssert.AreNotEqual |
I have also made a listing of all pros and cons of both unit testing frameworks.
NUnit
- Its the de-facto standard for unit tests in .NET.
- Already exists for several years now and it's very stable.
- New releases and new features on a regular basis.
- It's very easy to extend and write your own assertions (constraint model).
- Lots of features and assertions with support for collections and files.
- Very good integration with all editions of Visual Studio thanks to the TestDriven.NET add-in.
- It's open-source (yes, I see this as an advantage).
- Has its own stand-alone test runner that can installed with a simple XCOPY deployment.
- It's available for both .NET 1.1, .NET 2.0 or higher.
- Very easy to use with Selenium or WatiN.
- No out-of-the-box integration with Team Foundation Server.
Team System Unit Test
- Out-of-the-box integration with Team Foundation Server.
- Less users
- Its not free and only available for the Team System editions of Visual Studio. There's also no stand-alone test runner available. When our project lead or our functional analysts like to run some integration tests, they have to install one of the Team System editions of Visual Studio in order to do so.
- It's has many bugs and is less stable than NUnit. One of these bugs has to do with the ExpectedException attribute. I've also experienced some unexpected behavior with Assert.AreEqual (tests that should fail but instead pass for some obscure reasons). I've also got some frequent crashes of test runs.
- Test runs are very slow. One of the most important characteristics of a good unit test is that it's fast.
- Less features than other unit test frameworks like NUnit or MbUnit. Less assertions, no inheritance of test fixtures and no hierarchical organization of unit tests to name a few.
- Only available for .NET 2.0 or higher.
- Web testing of Team System simply sucks compared to Selenium and WatiN. It's possible to use the latter web testing frameworks in favor of the first one, but they are not well integrated.
I value the out-of-the-box integration of Team System Unit Test with TFS, but in order to sacrifice all the NUnit goodness, Microsoft has to do a better job for their unit test framework. Based on this list, information I found and my own experiences, I think NUnit is a no-brainer. I guess that after some Googling around and some tweaking, it's possible to integrate NUnit with TFS as well but I can't prove that right now (if you can help me with that, please send me an e-mail).
Therefore I would like to get your opinion on this matter. Some feedback is highly appreciated. You can also have a look at this post from Roy Osherove.
