Devel::Cover recommendations... or maybe not?

[Andy Armstrong]

On 15 Mar 2007, at 15:17, Tara Andrews wrote:
> On 3/15/07, Pete Sergeant <pete at clueball.com> wrote:
>
>> and perhaps you have to occasionally tie yourself in
>> knots trying to hit certain failure conditions (especially file- 
>> system
>> error ones),
>
> OK, I'll bite.  How?
>
> Seriously, a list of hints somewhere of "how to programmatically
> manufacture useful sets of edge cases" would be very useful to people
> who want complete testing for their software, but don't have a wizzo
> tester to get us from A to B.

To test hard to simulate failures there's a simple old technique you  
can use. I first saw it in C but it works just as well in Perl.

Say you want to find out what happens when each of a number of malloc 
() calls throughout a program fails. In C the logic around recovery  
from malloc() failures tends to be quite complex. The answer is to  
construct a test suite that exercises each malloc() at least once.  
The whole test run might call malloc(), say, 200 times.

To test the malloc failure handling logic replace the malloc calls  
with a wrapper like this:

static int fail_malloc = 1;

void *my_malloc(size_t size) {
     static int which_malloc = 0;
     if (++which_malloc == fail_malloc) {
         /* Simulate failure */
         return NULL;
     }
     return malloc(size);
}

Then you run the test 200 times incrementing fail_malloc each time.  
Assuming that the test is deterministic it'll make the same pattern  
of calls to malloc on each run (at least until one of them fails -  
then it might behave differently as a result of the failure). So at  
the end of your 200 runs you know with complete certainty that you've  
exercised every possible malloc failure in the program.

Depending on how complex the logic is you might also want to cover  
various combinations of failures .e.g. what happens if all mallocs()  
after the 17th fail? If every other malloc fails? If possible it's  
better to design the code so that multiple failures on successive  
calls to malloc() are unlikely to interact with each other.

The same technique can be used with Perl to, e.g., simulate failure  
of a filesystem call. Replace the functions you want to watch with  
debug versions that can be set to fail after a certain number of  
invocations and then run the test repeatedly until you've exercised  
all the possible failures.

In general that's the kind of thing you need to do to get coverage of  
this-never-happens-but-we-need-to-handle-it-in-case-it-ever-does cases.

-- 
Andy Armstrong, hexten.net