Blog

Most test frameworks out there have the capability to disable tests. This is usually done by adding some kind of annotation that instructs the test runner to ignore an individual test method.

[TestFixture]
public class SomeTests {

    [Test]
    public void RegularTest() 
    {
        ...
    }
    
    [Test]
    [Ignore]    // Ignore a single test
    public void IgnoredTest() 
    {
        ...
    }
}

It’s also possible to disable all tests of an entire test fixture by adding an annotation at the class level.

[TestFixture]
[Ignore]    // Ignore all tests
public class IgnoredTests {

    [Test]
    public void IgnoredTest() 
    {
        ...
    }
    
    [Test]
    public void AnotherIgnoredTest() 
    {
        ...
    }
}

A while back, I came to realise that I never use this kind of functionality to prevent the execution of tests. I became aware of this when I started observing other people’s development workflow. For some developers, being able to ignore tests seems to be a very important feature of a test framework. I’ve come to believe that the need for ignoring tests is more prevalent in a Test-After approach than with a Test-First approach. Let’s have a look at a very simple code example.

public class Task
{
    public Person Assignee { get; private set; }

    public void Assign(Person personToAssign)
    {
        Assignee = personToAssign;
    }
}

Suppose that we’ve built an application that provides an online Kanban board service to our customers. In the domain of this application there’s a class named Task. Currently, a task can be assigned to a particular person. This is how the test for the Assign method looks like.

[TestFixture]
public class TaskTests
{
    [Test]
    public void It_should_be_able_to_assign_a_person_to_a_task()
    {
        var person = new Person("Joe");
        
        var SUT = new Task();
        SUT.Assign(person);
        
        Assert.That(SUT.Assignee, Is.SameAs(person));
    }
}

These days, being able to assign only a single person to a task is a relic from the past. Therefore, in light of everything “ensemble”, the business requested whether it would be possible to assign multiple people to a single task.

Let’s see how a Test-First approach would look like. Obviously, we would start by writing a failing test. So we’ll add the following test method to the TaskTests test fixture.

[Test]
public void It_should_be_able_to_assign_multiple_people_to_a_task()
{
    var people = new[]
    {
        new Person("Joe"),
        new Person("Annie")
    };
    
    var SUT = new Task();
    SUT.Assign(people);
}

This new code doesn’t compile at this point as the Assign method currently accepts only a single Person object instead of a collection. This is a good thing because a test method that doesn’t compile qualifies as a failing test. Let’s fix this by adding a new overload of the Assign method.

public class Task
{
    public Person Assignee { get; private set; }

    public void Assign(Person personToAssign)
    {
        Assignee = personToAssign;
    }
    
    public void Assign(IEnumerable<Person> peopleToAssign)
    {
        
    }
}

We’re going to leave the implementation of this new method empty for the time being. Now the code of the application compiles again. Also, all the tests pass when we run them. We have working software again. However, it’s still not very useful as we didn’t completely finish the implementation of the new test method. An assert statement is still missing, so we’re going to add that as our next step.

[Test]
public void It_should_be_able_to_assign_multiple_people_to_a_task()
{
    var people = new[]
    {
        new Person("Joe"),
        new Person("Annie")
    };
    
    var SUT = new Task();
    SUT.Assign(people);

    Assert.That(SUT.Assignees, Is.SameAs(people));
}

By adding the assert statement we find out that the code doesn’t compile anymore. This is a good thing because a test method that doesn’t compile qualifies as a failing test. In order to fix this, we’re need to add an Assignees property to the Task class.

public class Task
{
    public Person Assignee { get; private set; }
    public IEnumerable<Person> Assignees { get; private set; }

    public void Assign(Person personToAssign)
    {
        Assignee = personToAssign;
    }
    
    public void Assign(IEnumerable<Person> peopleToAssign)
    {
        
    }
}

Now we’re able to compile the code again. However, when we run all the tests again we find out that the assert statement that we’ve just added fails the test. In order to make this test pass, we need to add the necessary implementation to the newly overloaded Assign method.

public class Task
{
    public Person Assignee { get; private set; }
    public IEnumerable<Person> Assignees { get; private set; }

    public void Assign(Person personToAssign)
    {
        Assignee = personToAssign;
    }
    
    public void Assign(IEnumerable<Person> peopleToAssign)
    {
        Assignees = peopleToAssign;
    }
}

When we run the test suite again, we notice that all the tests are green. We have working software again.

The rest of the code base still uses the Assign method that accepts a single Person object. So our next step is to migrate all the client code of the Task class to use the new version of the Assign method. We go about this migration by writing failing tests and making them pass by using the Assign method that accepts a collection of Person objects. When the original Assign method has been fully replaced, we can remove the corresponding test as well as the method itself.

Notice that during this workflow, we were always able to run all the tests without ignoring any existing ones. Let’s compare the Test-First approach to a Test-Last approach using the same example. We’ll immediately start off by making the necessary changes in the Task class.

public class Task
{
    public IEnumerable<Person> Assignees { get; private set; }

    public void Assign(IEnumerable<Person> peopleToAssign)
    {
        Assignees = peopleToAssign;
    }
}

We’ve renamed the Assignee property to Assignees while we also changed its type to IEnumerable<Person>. Likewise, we’ve changed the parameter of the Assign method to IEnumerable<Person> peopleToAssign. By making this change the way we did, we broke the contract of the Task class. The result is that the code doesn’t compile anymore.

First, the code of the test that verifies the Assign method needs to be changed. We complain a bit to our colleagues about how tests are holding us back while we’re developing code. After we’re done complaining, we comment out the line SUT.Assign(person); and slap the Ignore attribute on the test method. We tell ourselves that we’re going to fix this test later.

[TestFixture]
public class TaskTests
{
    [Test]
    [Ignore("Meh")]
    public void It_should_be_able_to_assign_a_person_to_a_task()
    {
        var person = new Person("Joe");
        
        var SUT = new Task();
        //SUT.Assign(person);
        
        Assert.That(SUT.Assignees, Is.SameAs(person));
    }
}

Next, we find out that there are other places in the production code that need attention as well. There could be a single caller of the Assign method, or there could be several places throughout the code base that call this method. Suppose that there are three places in the code that need to be altered to reflect the changes that we’ve made. After adding some additional Ignore attributes for disabling some other tests, everything compiles now.

Suppose that we’re now dragged into a meeting. When the meeting has ended, we (conveniently) forgot about the ignored tests. After all, we want to continue working on our feature. “Stressed and always in a hurry”.

Notice that after this endeavour we don’t even have a clue whether we have working software or not. Please note that the simple example that has been presented here in and of itself is not important. We could have made the necessary changes using the refactoring capabilities of any modern IDE. That’s not the point. The point is the difference between both workflows.

Whenever we feel the need to disable the execution of a test, we might want to consider finding a better approach. Tests should not be there to stand in our way. Rather they can help guide us throughout the development process. Test-Driven Development is not only about writing tests first. It’s about being able to work in very short iterations. The important part is that by the end of every iteration, we have code that works. Writing tests first is just the means that guides us through that workflow.

Jan Van Ryswyck

Thank you for visiting my blog. I’m a professional software developer since Y2K. A blogger since Y2K+5. Provider of training and coaching in XP practices. Curator of the Awesome Talks list. Past organizer of the European Virtual ALT.NET meetings. Thinking and learning about all kinds of technologies since forever.