It is said that "if your software is bug free, you're just not testing enough". With that in mind, we spent quite a bit of time during the last months on our internal testing infrastructure. Let's take a closer look at the results so far.

Since it was first mentioned in a blog post back in January, qooxdoo's automated testing setup has grown and matured quite a bit. As of this week, the testing infrastructure consists of 6 PCs running 4 different operating systems (Windows XP, Vista, OS X and Linux) with a total of 18 browsers, from IE6 all the way up to the latest devel version of Google Chrome. While not every conceivable combination of OS, browser and application is tested,  each nightly test run consists of over 160 individual tests.

Test schedule

Each night all the framework applications are generated, unit tests are run and applications are subjected to simulated interaction tests. Additionally, the framework classes are scanned for missing documentation and checked for potentially problematic code by ecmalint.

A reduced number of tests is also run each Friday afternoon to make sure the week's devel versions of the demo apps are free of show stopping bugs.

Selenium running the Feedreader simulation

Selenium running the Feedreader simulation

Software components

The backbone of the testing system is a test controller script, written in Python: Its first task is to update a local qooxdoo trunk checkout and to run the Generator commands necessary to build the framework applications, api reference and unit tests. Once the apps are ready, the test controller script proceeds to launch a sequence of simulated interaction tests (a.k.a. simulations), each of which logs any errors it encounters.  Finally, the controller script aggregates the test results and generates HTML-formatted reports which are then emailed to the qooxdoo team members.

The simulations leverage Selenium RC, enhanced by the qooxdoo user extensions from the Simulator contribution. These add a number of locators and commands that are used in the simulation code to interact with a qooxdoo application and its widgets.

For example, the Feedreader test mimicks a user selecting a feed item, changing the application language and adding a new feed. After each step, the application is inspected to make sure the action had the expected result.
The Testrunner simulation, on the other hand, features very little simulated interaction: Unit test packages are run, and any test failures or errors are logged.

All Simulations are written in JavaScript and executed in Mozilla Rhino which provides access to the Selenium Java class.

HTML report of the API documentation test

HTML report of the API documentation test

Conclusion

Even though we've yet to further stabilize the infrastructure and to reach some of the loftier goals of Continuous Integration (such as building and testing (almost) continuously rather than just nightly), automated testing already helps us notice bugs faster and has taken at least some of the burden of testing in a wide variety of browsers off the core devs' shoulders. You certainly benefit from a more stable and higher-quality framework.