XML Unit Testing

Developers have known for a long time that unit testing can make a difference in software. In particular, solid unit tests help increase its robustness and make it easier to refactor source code. Extreme Programming (XP) in particular considers testing a core piece of its lightweight methodology (but many developers do unit tests without doing XP).

Orbeon Presentation Server (OPS) hasn't always had many unit tests, but it was decided at some point that tests were necessary, and we started building a test framework for it. We hate the Not Invented Here (NIH) syndrome in software, and it is therefore slightly incorrect to say that we built a new framework. JUnit is essentially designed to run tests on Java code, not on XML, so we created a layer, leveraged by JUnit, that allows describing the tests in a way that is much more natural to OPS, and at the same time, we can still leverage JUnit and its front-ends, like the one integrated in IntelliJ IDEA.

One of the concepts at the center of OPS is that of XML processor. It is a software component, with the particularity that its interface to the outside world consists of incoming and outgoing XML documents. XML processors are typically used within XML pipelines described with XPL, the OPS XML pipeline language. So one common test scenario consists in feeding an XML processor with one or more XML documents, and checking that the XML processor returns one or more expected XML documents.

This idea is at the core of the ProcessorTest class, which extends the JUnit TestCase class. ProcessorTest reads a configuration file called tests.xml that describes all the XML-based unit tests. tests.xml XIncludes other files, so that tests can be nicely grouped by the type of functionality they are testing. One of the simplest tests you can imagine is this one:

<test description="Identity" name="oxf:identity">
    <input name="data">
        <root>
            <a/>
            <b/>
        </root>
    </input>
    <output name="data">
        <root>
            <a/>
            <b/>
        </root>
    </output>
</test>

It just makes sure the Identity processor produces on its output the same document it receives on its input, for a very simple input document consisting of just a few elements. From there, you can build much more complicated tests based on other OPS processors, including testing entire XML pipelines written with XPL.

Overview of XML Processor Testing

The story is not over, because there are other scenarios that need testing in OPS, for example caching, a tricky piece of functionality. You want to be able to tests such scenarios as "If this file hasn't changed on disk, make sure the cached result is used, otherwise redo the processing." To do so, a special Test Script processor was devised.

The Test Script processor is an XML processor like any other in OPS. Its particularity is that it is able to run specific test scripts operating on the guts of OPS itself, in particular the XPL engine.

While XML processors can perform operations in a vacuum (like for example doing a pure XSLT transformation of an input to an output based on a stylesheet), often XML processors depend on external factors. In OPS, this is represented by the ExternalContext interface, which abstracts requests and responses so that XML processors can work in Servlet, Portlet, command-line, and embedded contexts.

To test scenarios depending on such an external context, one could launch a Servlet container in a controlled environment, and run requests against it. This is not a bad way of proceeding, and it is in fact the ultimate way, in that it reproduces actual running conditions. The disadvantage is that it is cumbersome to setup.

So the Test Script processor uses a simpler method, which consists in building internally an ExternalContext instance from an XML document describing what that context looks like. This is actually quite cool, because with a simple XML document, you can describe an HTTP request as seen by OPS. Setting the request can also be scripted by the Test Script processor, so you can do things like testing the Request generator. For examples using the Test Script processor, check out the source code and look for tests-cache.xml.

The OPS test framework does not yet support producing an XML document representing the ExternalContext response, but this can be added fairly easily. In addition to that, some scripting facility that allows transforming a response into a new request will allow creating test scripts that simulate entire request / response cycles.

The number of OPS unit tests has rapidly grown over the last six months and today there are about 120 unit tests. This is a good start and has already caught bugs before developers check in their code, and more during the continuous integration builds. Because yes, you really want to have a system like Anthill continuously building your application, and making sure that a build that does not pass the automated unit tests suite is considered broken.

Of course, many more unit tests should be added to OPS over time. Some areas of functionality are not yet covered. But developers have now taken the habit of writing tests whenever they fix a bug or implement new functionality. We have much more confidence that the code we write does the right thing, and does not break existing functionality. This is simply invaluable.