Hosted by:
Unit testing expertise at your fingertips!
Home | Discuss | Lists

Assertion Roulette

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 224 of xUnit Test Patterns for the latest information.

It is hard to tell which of several assertions within the same test method caused a test failure.


A test fails. Upon examining the output of the Test Runner (page X), we cannot determine exactly which assertion had failed.


When a test fails during an automated Integration Build[SCM], it may be hard to tell exactly which assertion failed. If the problem cannot be reproduced on a developer's machine (as may be the case if the problem is caused by environmental issues or Resource Optimism (see Erratic Test on page X)) fixing the problem may be difficult and time-consuming.


Cause: Eager Test

A single test verifies too much functionality.


A test exercises several methods of the system under test (SUT) or calls the same method several times interspersed with fixture setup logic and assertions.

   public void testFlightMileage_asKm2() throws Exception {
      // setup fixture
      // exercise contructor
      Flight newFlight = new Flight(validFlightNumber);
      // verify constructed object
      assertEquals(validFlightNumber, newFlight.number);
      assertEquals("", newFlight.airlineCode);
      // setup mileage
      // exercise mileage translater
      int actualKilometres = newFlight.getMileageAsKm();    
      // verify results
      int expectedKilometres = 1810;
      assertEquals( expectedKilometres, actualKilometres);
      // now try it with a canceled flight:
      try {
         fail("Expected exception");
      } catch (InvalidRequestException e) {
         assertEquals( "Cannot get cancelled flight mileage", e.getMessage());
Example EagerTest embedded from java/com/xunitpatterns/testtemplates/BadExamples.java

Another possible symptom is that the test automater(s) wants to modify the Test Automation Framework (page X) to keep going after an assertion has failed so that the rest of the assertions can be executed.

Root Cause

Eager Test is often caused by trying to minimize the number of unit tests (whether consciously or unconsciously) by verifying many test conditions in a single Test Method (page X). While this is a good practice for manually executed tests that have "liveware" interpreting the results and adjusting the tests in real time, it just doesn't work very well for Fully Automated Tests (see Goals of Test Automation on page X).

Another common cause is using xUnit to automate customer tests that require many steps thus verifying many aspects of the SUT in each test. These tests are necessarily longer than unit tests but care should be taken to keep them as short as possible (but no shorter!)

Possible Solution

For unit tests, we break up the test into a suite of Single Condition Tests (see Principles of Test Automation on page X) by teasing apart the Eager Test. It may be possible to do this by using Extract Method[Fowler] refactoring to pull out independent pieces into their own Test Methods. Sometimes, it is easier to clone the test once for each test condition and then clean up each Test Method by removing any code that is not required for that particular test conditions. Any code required to set up the fixture or put the SUT into the correct starting state can be extracted into a Creation Method (page X). A good IDE or compiler will then help us determine what variables are no longer being used.

If we are automating customer tests using xUnit and this has resulted in many steps in each test because the work-flows require complex fixture setup,we could consider using some other way to set up the fixture for the latter parts of the test. If we can use Back Door Setup (see Back Door Manipulation on page X) to create the fixture for the last part of the test independently of the first part, we have just succeeded in turning one test into two and have improved our Defect Localization (see Goals of Test Automation). We should repeat the process as many times as it takes to make the tests short enough to be single glance readable and to Communicate Intent (see Principles of Test Automation) clearly.

Cause: Missing Assertion Message


A test fails. Upon examining the output of the Test Runner, we cannot determine exactly which assertion had failed.

Root Cause

This is caused by the use of an Assertion Methods (page X) calls with identical or missing Assertion Messages (page X). It is most commonly a problem when running tests using a Command-Line Test Runner (see Test Runner) or a Test Runner that is not integrated with the program text editor or development environment. In this tests, we have a number of Equality Assertions (see Assertion Method):

   public void testInvoice_addLineItem7() {
      LineItem expItem = new LineItem(inv, product, QUANTITY);
      // Exercise
      inv.addItemQuantity(product, QUANTITY);
      // Verify
      List lineItems = inv.getLineItems();
      LineItem actual = (LineItem)lineItems.get(0);
      assertEquals(expItem.getInv(), actual.getInv());
      assertEquals(expItem.getProd(), actual.getProd());
      assertEquals(expItem.getQuantity(), actual.getQuantity());
Example NaiveInlineAssertions embedded from java/com/clrstream/camug/example/test/InvoiceTest.java

When an assertion fails, will we know which one it was? An Equality Assertions typically prints out both the expected and actual values but it may be hard to tell which assertion failed if the expected values are similar or print out cryptically. A good rule of thumb is to include at least a minimal Assertion Message whenever we have more that one call to the same kind of Assertion Method.

Possible Solution

If the problem occurred while running a test using a Graphical Test Runner (see Test Runner) with IDE integration, we should be able to click on the appropriate line in the stack trace back to have the IDE highlight the failed assertion. Failing this, we can turn on the debugger and single-step through the test to see which assertion statement fails.

If the problem occurred while running a test using a Command-Line Test Runner, we can try running the test from a Graphical Test Runner with IDE integration to determine the offending assertion. If that doesn't work, we may have to resort to using line numbers (if available) or using a process of elimination to deduce which of the assertions it couldn't be to narrow down the choice. Of course, we could just bite the bullet and add a unique Assertion Message (even just a number!) to each call to an Assertion Method.

Further Reading

Assertion Roulette and Eager Test were first described in a paper at XP2001 called "Refactoring Test Code" [RTC].

Page generated at Wed Feb 09 16:39:49 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved

All Categories
Introductory Narratives
Web Site Instructions
Code Refactorings
Database Patterns
DfT Patterns
External Patterns
Fixture Setup Patterns
Fixture Teardown Patterns
Front Matter
Result Verification Patterns
Test Double Patterns
Test Organization
Test Refactorings
Test Smells
Test Strategy
Value Patterns
XUnit Basics
xUnit Members
All "Test Smells"
Code Smells
--Obscure Test
--Conditional Test Logic
--Hard-to-Test Code
--Test Code Duplication
--Test Logic in Production
Behavior Smells
--Assertion Roulette
----Eager Test
----Missing Assertion Message
--Erratic Test
--Fragile Test
--Frequent Debugging
--Manual Intervention
--Slow Tests
Project Smells
--Buggy Tests
--Developers Not Writing Tests
--High Test Maintenance Cost
--Production Bugs