A bit of testing never hurts ..

Each time I touch the layouting subsystem of the reporting engine, I am scared to death. Layouting is a task that is not simple and almost never straight forward. Unless you are willing to consume loads of memory (up to the point where you would have to invent swapping system to free memory for larger reports) the layouting system needs to be balanced carefully.

The layout rules must be deterministic, complete and at the same time easy enough to understand to tell them to the user.

Our layouting engine is basically a huge state machine, where nodes get added and removed constantly to represent both an accurate picture of what has been processed (so that space is not allocated twice) and what is currently due to be processed. The engine itself streams data though its guts and only holds on to the parts that have not yet been printed. It allows us to have a really minimalistic memory footprint, nearly regardless of the total number of pages rendered. One page or a million, the amount of memory used stays a rather flat line.

On the negative side, we sit on a huge pile of states, all interdependent and interwoven and nearly impossible to simulate from outside. Some parts can be extracted, but the majority of the code needs to be run together to behave correctly.

Writing test cases for that is not fun - until now.

Over the last few weeks I wrote a set of "golden sample" tests. A golden sample is a test that runs a given report and compares its output against a known good state. Such tests cannot provide protection from unknown evils, but we are now at least in a position to validate that the existing reports run as before.

And best of all: These tests are easy to set up and easy to maintain. As long as we uphold the promise that old reports behave the same in newer reporting engines, we can just drop the report definitions into the sample pool, generate golden outputs and let them stay there for future generations of testers.

The test system could easily used outside of the development cycle as well. If you let your reports run against a test database you could verify whether the latest upgrade broke your reports or whether test system and production system *really* produce the same results. What do you think - would a "golden test" make sense in your organization?