Monday, March 25, 2013

Automated browser testing with Selenium on Travis-CI

We've written before about the importance of having a set of comprehensive unit tests for Orbeon Forms. For instance, we put in place the necessary infrastructure to test code in Scala back in 2011 and to test XML pipelines back in 2005. With this, every time a check-in is made, a build is automatically created and 687 tests (and counting) are executed. If any test fails, we get a notification, and fixing the build becomes our priority.

But until now, we couldn't automatically run tests that requires a browser, which means that a significant portion of Orbeon Forms couldn't be automatically tested, including all our client-side code (JavaScript and CoffeeScript), interactions between client-side and server-side code, and higher level components such as Form Builder and Form Runner.

We now have this infrastructure in place:

  • Running the tests automatically – Everytime we do a check-in to our Github repository, Travis-CI checks the code out, compiles the code, runs the tests, and sends us an email if any test fails.
  • Running Tomcat from Ant – We still use1 the venerable Ant as our build system, so it is up to Ant to compile the code and run the tests. The new tests require a server to run, and for that we use Tomcat. During the build, we install Tomcat in the build directory, and configure it to pick up its configuration files from test/catalina_base, by setting the catalina.base property. Tomcat needs to run in parallel with the tests, which we do with Ant's Parallel task. Before running the tests we need to make sure Tomcat is successfully started, which can be done very easily with Ant's Waitfor task. To see how this is done, check the target name="test" in our build.xml.
  • Starting a browser – Our tests run in a real browser: Firefox2. Since the test machine isn't connected to an actual screen, we run Xvfb through our .travis.yml.
  • Writing the tests – To drive Firefox, we use Selenium 2 with WebDriver. Our tests are written in Scala, with ScalaTest and its Selenium DSL.

  1. And in case reading that we are still building on Ant makes you worry about our sanity, rest assured that we are thinking of moving to a more modern build system, like sbt.
  2. ChromeDriver currently can't be used on Tavis-CI as Chrome does not work in an OpenVZ container, which is used by Travis-CI. But in the future, while still using Travis-CI, we might run the tests on more browsers through Sauce.

No comments:

Post a Comment