Blog

The Test Pyramid

November 7, 2019

In the previous blog post, we touched on two different categories of automated tests, namely solitary and sociable tests. We also mentioned that in order to build maintainable and high-quality software systems, we need both solitary as well as sociable tests, but not in an equal amount. Although tests from both of these categories have their strong points and weaknesses, it’s more desirable to have several solitary tests combined with only a few accompanying sociable tests.

This is where the test pyramid comes in. Initially described by Mike Cohn in the book Succeeding with Agile, the test pyramid provides a visual representation of a beneficial balance of automated tests.

The test pyramid and it's different layers

The concept of the test pyramid is made up from the following parts:

  • At the base, we have solitary tests. These should make up the largest amount of automated tests in the software system.
  • In the middle, we have sociable tests. These should definitely be part of the suite of automated tests, but in a significantly lesser amount than the number of sociable tests.
  • At the top, we have a handful to a dozen broad system tests. This is usually in the form of UI or API tests that exercise the most important parts of the system. Try to resist the urge to create lots of these kind of tests as they can quickly turn against you and soon become a maintenance nightmare.

At the bottom of the test pyramid, we have the most isolation and also the fastest performance. This is where we get the most feedback about the design of the system. The more we move up the test pyramid, the more integration is employed and verified which results in slower tests and less feedback.

There are four primary reasons as to why we prefer that the majority of tests are solitary tests:

  1. Sociable tests can be very slow and non-deterministic. This is due to the fact that they often cross their process boundary. They make failure diagnostics more difficult because they are further removed from the cause of the failure.
  2. Sociable tests can therefore be overly verbose as they require a lot more code in order to setup parts of the system under test. It requires more effort and therefore takes more time to write.
  3. Solitary tests apply a certain pressure on the design of the system. They clearly indicate the design flaws that might exist. Sociable tests on the other hand don’t provide such useful feedback about the design of the system because they are inherently farther removed from the details.
  4. Sociable tests are highly susceptible to cascading failures. We’re going to elaborate further on this issue in a next post.

Do not bet on only a single category of these tests! Applying all of these categories to the system is the best way to achieve a well-tested code-base. However, a surprisingly large number of development teams have made the mistake of applying a testing pyramid that is upside down. Such an “ice cream cone” shaped automated testing strategy implies that there are more sociable tests than solitary tests. This anti-pattern usually stems from an overreliance on manual testing and/or a lack of applying the Test-Driven Development process.

Do bear in mind that the exact number of tests in each layer of the test pyramid is not really that relevant. However, the test pyramid is a very useful model to think and reason about. Let it be your guide on all your test automation efforts.

If you and your team want to learn more about how to write maintainable unit tests and get the most out of TDD practices, make sure to have look at our trainings and workshops or check out the books section. Feel free to reach out at infonull@nullprincipal-itnull.be.

Profile picture of Jan Van Ryswyck

Jan Van Ryswyck

Thank you for visiting my blog. I’m a professional software developer since Y2K. A blogger since Y2K+5. Provider of training and coaching in XP practices. Curator of the Awesome Talks list. Past organizer of the European Virtual ALT.NET meetings. Thinking and learning about all kinds of technologies since forever.

Comments

Latest articles

Tags

Disclaimer

The opinions expressed on this blog are my own personal opinions. These do NOT represent anyone else’s view on the world in any way whatsoever.

About

Thank you for visiting my website. I’m a professional software developer since Y2K. A blogger since Y2K+5. Author of Writing Maintainable Unit Tests. Provider of training and coaching in XP practices. Curator of the Awesome Talks list. Thinking and learning about all kinds of technologies since forever.

Latest articles

Contact information

(+32) 496 38 00 82

infonull@nullprincipal-itnull.be

principal-it.eu
principal-it.be
janvanryswyck.com