Today, let me offer a slightly different kind of post. For once, I will not give any advice but rather propose a riddle for you to solve.
As I have already mentioned in a previous post, regularly running automatic test suites (à la JUnit) is a guarantee against program decay. With a continuous integration server such as Jenkins, reports periodically built from test results present an up-to-date status of the code condition. Test suites are a barrier against regression. Thus enabling, sometimes extremely aggressive, code refactoring. Some teams even use automatic tests as a non perishable form of documentation, or alternatively as executable specifications. The cost of the manual verification phase can also be reduced, by progressively converting the most repetitive testing scenarios into automatic tests. In short, automatic tests let you attain a surprisingly high level of software quality.
So, it is considered good practice, to at least add one automatic test, for every bug found and fixed. To adopt test driven development may even be more rewarding, but that is another story. However, making a manual verification scenario totally automatic may sometimes prove particularly tricky. This is, in particular, the case for any piece of software which depends on some low level library upon which the programmer has no control whatsoever. Herein lies the crux of today's riddle: how to write automatic tests for code relying on non modifiable external libraries.
Let us consider a concrete Java example:
public class Program {
public static void main(String[] arguments) throws Exception {
if (arguments.length < 1) return;
Program program = new Program();
program.process(arguments[0]);
}
public void process(String fileName) throws Exception {
...
FileInputStream file = new FileInputStream(fileName);
...
int value = file.read();
...
file.close();
}
}
During its execution (method process), this program opens the file (new FileInputStream(fileName)) whose name is passed as argument to the entry point (method main). It then reads some content from this file (int value = file.read()).
Let us suppose the validation team found the following problem: during one of its run, the program brutally stopped. The call to method read unexpectedly threw a java.io.IOException. This may happen, for instance, when accessing a distant file system which suddenly becomes unavailable because of a severed cable. This is clearly an execution context which is difficult to systematically reproduce at each run of an automatic test.
How would you write an automatic non-regression test which replays this scenario?
How would you write an automatic non-regression test which replays this scenario?
Oedipus and the Sphinx
 |
| by Ingres, 1808 |
.jpg) |
| by Gustave Moreau, 1864 |
 |
| by Salvador Dali, 1960 |
I first try to trick the low-level library into exhibiting the intended behaviour, and am most of the time succesful. A trick that could work here is, using reflection (or whatever it is called in java), to access the underlying file descriptor, and destroy it with a manual system call (I did it in C#).
ReplyDeleteOnly when I fail do I resort to writing a wrapper for the stream class, that normally forwards calls to the underlying stream, but can be hijacked during automated tests.
My rationale is that, since the low-level libraries commonly have lots of undocumented idiosyncrasies, it is very difficult to write a realistic mockup. I therefore prefer using the real classes as much as possible.
Nice succinct and complete answer. I guess it was too easy! (even though many programmers seem not to have even a clue)
ReplyDeleteI tend to be more in the mockup category (because the tests tend to be less arcane and writing them lead to design improvements), but agree with your pragmatic approach.
With respect to exceptions, it seems to me that Java is a lot more faithful to documentation than C#.