Test case dependencies

Every so often some engineer has the bright idea of introducing a way to express dependencies between test cases, for example in the Bash test framework, usually with the objective of optimising a test run and/or saving developer time in the code-build-test cycle.

This is almost always a BAD IDEA for the following reasons:

  • makes it difficult or impossible to run individual test cases
  • makes it hard or impossible to run test cases in parallel
  • over-complicates diagnosis of test failures (already complex enough!)
  • hinders re-factoring, extending and restructuring of test cases
  • makes test cases more fragile; changing one may break others
  • there are better ways to solve the problem

Developer cycle

The problem of developer time can be tackled as follows (using the Bash test framework as an example):

  • run only a single test case in the code-build-test cycle, using the --filter option
  • take advantage of the default execution order of test cases (lexical order of definition in the Bash source file) and use the -F|--stop-on-failure and/or -E|--stop-on-error options
  • if the defined order is not convenient, construct a temporary script that invokes several individual test cases in a specified order, terminating upon the first failure

Slow and stateful System Under Test

Sometimes the motivation to introduce test case dependencies arises from a slow and stateful System Under Test (SUT). If the final state of a (passed) costly test serves as the fixture for another test, and there is no faster way to create the fixture, then a dependency from the first to the second is an obvious way to save time when running both. But this is a false saving.

The slow & stateful SUT problem is better tackled by combining all the dependent tests into a single test case. A single, gigantic test case is hard to understand and costly to run, but a set of small, interdependent test cases is even harder to understand and maintain, and just as costly, and gives the false impression that they can be run independently. If you need to be run only the first step of the test case during development, then create a temporary test case consisting of only the first step, by copying and pasting (or better, factoring the common code into a shared function) and use that, discarding once no longer needed.