Haskell/Testing

Quickcheck
Consider the following function:

How would we effectively test this function in Haskell? We'll use refactoring and QuickCheck.

Keeping things pure
The getList function is hard to test because  does IO out in the world, so there's no internal way to verify things. The other statements in our  block are all wrapped up with the IO.

Let's untangle our function so we can at least test the referentially transparent parts with QuickCheck. We can take advantage of lazy IO firstly, to avoid all the unpleasant low-level IO handling.

So the first step is to factor out the IO part of the function into a thin "skin" layer:

Testing with QuickCheck
Now, we can test the 'guts' of the algorithm, the take5 function, in isolation. Let's use QuickCheck. First we need an Arbitrary instance for the Char type — that takes care of generating random Chars for us to test with. Restrict it to a range of nice chars just for simplicity:

Let's fire up GHCi and try some generic properties (it's nice that we can use the QuickCheck testing framework directly from the Haskell REPL). An easy one first, a [Char] is equal to itself:

OK, passed 100 tests.
 * A> quickCheck ((\s -> s == s) :: [Char] -> Bool)

What just happened? QuickCheck generated 100 random [Char] values, and applied our property, checking the result was True for all cases. QuickCheck generated the test sets for us!

A more interesting property now: reversing twice returns the identity:

OK, passed 100 tests.
 * A> quickCheck ((\s -> (reverse.reverse) s == s) :: [Char] -> Bool)

Great!

Testing take5
The first step to testing with QuickCheck is to work out some properties that are true of the function, for all inputs. That is, we need to find invariants.

A simple invariant might be: $$\forall s.\; \mathit{length} (\mathit{take}5\; s) = 5$$

So let's write that as a QuickCheck property:

Which we can then run in QuickCheck as: Falsifiable, after 0 tests: ""
 * A> quickCheck (\s -> length (take5 s) == 5)

Ah! QuickCheck caught us out. If the input string contains less than 5 filterable characters, the resulting string will be no more than 5 characters long. So let's weaken the property a bit: $$\forall s.\; \mathit{length} (\mathit{take}5\; s) \le 5$$

That is, take5 returns a string of at most 5 characters long. Let's test this: OK, passed 100 tests.
 * A> quickCheck (\s -> length (take5 s) <= 5)

Good!

Another property
Another thing to check would be that the correct characters are returned. That is, for all returned characters, those characters are members of the set ['a','b','c','d','e'].

We can specify that as: $$\forall s. \forall e. (e \in take5\; s) \Rightarrow (e \in \{a, b, c, d, e\}) $$

And in QuickCheck: OK, passed 100 tests.
 * A> quickCheck (\s -> all (`elem` ['a'..'e']) (take5 s))

Excellent. So we can have some confidence that the function neither returns strings that are too long nor includes invalid characters.

Coverage
One issue with the default QuickCheck configuration, when testing [Char]: the standard 100 tests isn't enough for our situation. In fact, QuickCheck never generates a String greater than 5 characters long when using the supplied Arbitrary instance for Char! We can confirm this:

OK, passed 100 tests.
 * A> quickCheck (\s -> length (take5 s) < 5)

QuickCheck wastes its time generating different Chars, when what we really need is longer strings. One solution to this is to modify QuickCheck's default configuration to test deeper:

This instructs the system to find at least 10000 test cases before concluding that all is well. Let's check that it is generating longer strings:

Falsifiable, after 125 tests: ";:iD^*NNi~Y\\RegMob\DEL@krsx/=dcf7kub|EQi\DELD*"
 * A> deepCheck (\s -> length (take5 s) < 5)

We can check the test data QuickCheck is generating using the 'verboseCheck' hook. Here, testing on integers lists:

0: [] 1: [0] 2: [] 3: [] 4: [] 5: [1,2,1,1] 6: [2] 7: [-2,4,-4,0,0] Falsifiable, after 7 tests: [-2,4,-4,0,0]
 * A> verboseCheck (\s -> length s < 5)

More information on QuickCheck

 * http://haskell.org/haskellwiki/Introduction_to_QuickCheck
 * http://haskell.org/haskellwiki/QuickCheck_as_a_test_set_generator

HUnit
Sometimes it is easier to give an example for a test than to define one from a general rule. HUnit provides a unit testing framework which helps you to do just this. You could also abuse QuickCheck by providing a general rule which just so happens to fit your example; but it's probably less work in that case to just use HUnit.


 * TODO: give an example of HUnit test, and a small tour of it

More details for working with HUnit can be found in its user's guide.