- maybe the specification was fuzzy,
- or some library did not behave as documented,
- or a corner case could hardly be anticipated...
As a starting point, let us consider a simple, hypothetical coding situation. We are working on a word manipulation library. The library exports a unique namespace WordManipulations, which contains a class Sentence. This class wraps a string and provides some simple text manipulation operations. The first step consists in adding a new project, called Tests, dedicated to non-regression tests. It references both the dynamic linked library (DLL) nunit.framework and the project WordManipulations. We then create a class Suite00_Sentence in order to group all tests related to class Sentence. We place the attribute [TestFixture] to signal to NUnit that it is a test suite. Our first test method will be called Test000_CreationFromString and is similarly tagged by the NUnit attribute [Test]:
At this point, the organisation of the whole solution should look like this:using NUnit.Framework;
public class Suite00_Sentence
public void Test000_CreationFromString()
Let us now check everything compiles nicely and our first test executes correctly. In SharpDevelop, simply click on the play button of the unit testing side-panel. If you like it better, you can also directly use the NUnit interface. To do so, first compile the project Tests. Then go to directory Tests/bin/Debug in the Windows file explorer and double-click on file Tests.dll. This should invoke NUnit. From there, simply click Run. In both cases, a green light should indicate the test suite ran successfully.
For the next step, let us fill in the body of Test000_CreationFromString. It specifies that an instance of class Sentence can be built from a string:
For this test to succeed, we implement the corresponding constructor in class Sentence as follows:public void Test000_CreationFromString()
Sentence empty = new Sentence("");
Let us go a bit further. Say we want a method LastWord to return the index of the last word in a sentence. An additional test simulates the most simple use case of LastWord. Namely, when the input is "a b", then the expected output should be 2:public class Sentence
private string content;
public Sentence(string content)
this.content = content;
Class Assert from the NUnit framework provides several ways to check the expected output of tests.public void Test001_LastWord()
Sentence input = new Sentence("a b");
int result = input.LastWord();
An implementation of LastWord that validates this test could be for instance:
However, soon somebody named Mary Poppins reports an array out of bounds on "Supercalifragilisticexpialidocious". We immediately write a non-regression test that reproduces the bug:public int LastWord()
int i = this.content.Length - 1;
char currentChar = this.content[i];
while (currentChar != ' ')
currentChar = this.content[i];
return (i + 1);
Before starting a long debugging session, we simplify the test to the bone. It turns out that the current version of LastWord even fails on an empty string. So we modify our test accordingly:public void
= new Sentence("Supercalifragilisticexpialidocious");
To pass this test, we rewrite LastWord:public void
Sentence input = new Sentence("");
We then run all the tests written so far. They all succeed, so we can go back to the initial bug report. The method seems not to raise any exception anymore on a single word sentence. However, it does not seem to return the correct value either. So we add another test:public int LastWord()
int i = this.content.Length;
} while ((i > 0) && (this.content[i] != ' '));
We modify our code once more, and hopefully last, time:[Test]
public void Test003_LastWordOnSingleWordSentence()
Sentence input = new Sentence("a");
int result = input.LastWord();
In the end, we produced four test cases and a much clearer source code than the one we started with. As you may have guessed, I am particularly fond of early returns. I actually believe, there is still (at least) one bug. Can you find it?public int LastWord()
for (int i = this.content.Length - 1; i > 0; i--)
if (this.content[i] == ' ') return (i + 1);
Let me draw some general rules from this example and lay out the methodology of non-regression testing. A non-regression test is a piece of code that checks for the absence of a particular bug. It should fail in the presence of the bug and succeed in its absence. Tests have various purposes:
- robustness tests check the program does not stop abruptly,
- functional tests check the program computes the expected value,
- performance tests check the program runs within its allocated time and memory budget.
- a prelude sets up the context and/or prepares the input,
- an action triggers the bug,
- an assertion checks the expected behaviour.
- Test suites should be named after the class they target and prefixed by "SuiteXX_", where XX stands for a two digits number,
- Test methods should be named after the behaviour they check and prefixed by "TestXXX_", where XXX stands for a three digits number.
- evaluate the quality of classes external interfaces,
- choose the rôle of each method,
- explore scenarios previously identified during design.
- during code refactoring, you fear to break some invariant, first write some tests,
- code learning/review: you must work on a particularly obsure piece of software, for every bit of understanding you painfully acquire, write some tests,
- code quality ramp up: use the report of a code coverage tool to write some tests.
When following a non-regression methodology, every bug becomes the occasion to improve both specification and code quality... forever. Thus, after a while, one begins to appreciate bugs for their true worth. In addition to this profound psychological twist, non-regression testing has other impacts on the overall development process. Since at all times, the project runs without breaking its past behaviour, coding becomes a matter of incremental refinement. Project risk is managed almost to the point that the software may be wrapped and shipped any minute.
TestRunMethodName, TestTriggerEventName and TestGetFieldName. However, the visibility of many classes, which should ideally be internal, needs to be elevated to public in order to become testable. This tends to burden a namespace unnecessarily. If you have an elegant solution to this problem, I would really like to hear about it!
To summarize this post:
- either during feature addition or debugging, tests should be written before coding,
- tests should be as simple and concise as possible,
- as I like to repeat ad nauseam to my team members:
My questions for you today are:
- Do you do non-regression testing?
- What are your practices like?
- Which unit testing framework do you use?
- Do you have any idea how to test classes without making them public?