WebDriver Support

For quite a while now, qooxdoo has included support for automated application testing using Selenium RC. The Simulator contribution provides qooxdoo-specific extensions for Selenium Core and language bindings to use them in Java, Python and Perl, while the simulation-build and simulation-run jobs allow developers to write tests in qooxdoo-style JavaScript and launch them using the Generator.

However, the Selenium project has since included WebDriver, which uses a more “native” and thus more reliable approach to remotely control browsers, as well as a much nicer API. Selenium RC has been deprecated and getting it to work with newer browser versions is increasingly painful.

That’s why we’ve been working on qxwebdriver-java, a project that aims to recreate the Simulator’s feature set while leveraging the WebDriver API. Even though we might prefer JavaScript for its flexibility, we chose Java as the user-facing language. The main reason for this is that, at least in our experience, most QA teams rely on the proven combination of Maven and Jenkins to set up a Continuous Integration infrastructure.

qxwebdriver-java is still experimental. The API is subject to change without notice, not all of qooxdoo’s built-in widgets are supported and there is much still to be optimized. Don’t let that stop you from playing with it, giving us feedback on the usual channels and sending pull requests. Thanks!

Portable Test Runner

Recently, we’ve been thinking about how we can make some of the qooxdoo framework’s nifty features accessible to developers working on JavaScript projects not based on an application skeleton. One result of these deliberations is the Portable Test Runner. In a nutshell, it’s a single .js file with no external dependencies containing qooxdoo’s entire unit testing system, including Assertions, Sinon, Requirements and a Test Runner GUI. No compile step is necessary, just create an HTML file referencing testrunner-portable.js as well as any JS libs you wish to test, add your test definitions and load in a browser. Here’s a simple example: The (non-qx) library “ape” provides a “Gorilla” class, with a constructor that takes a string parameter. This test checks if the “getName” method returns that string:

 
  <script src="http://localhost/testrunner-portable.js" type="text/javascript"><!--mce:0--></script>
  <script src="http://localhost/ape.js" type="text/javascript"><!--mce:1--></script>
  <script type="text/javascript"><!--mce:2--></script>

The important thing to note here is the map argument for testrunner.define: It’s equivalent to the members section of a class extending qx.dev.unit.TestCase and including qx.dev.unit.MMock and qx.dev.unit.MRequirements, allowing full access to these APIs. Multiple test classes can be defined by additional calls to testrunner.define.

And here’s what it looks like in a browser:

Download the portable Test Runner here:

The portable Test Runner is still considered experimental, so if you run into any problems or have ideas on how it could be improved, please leave a comment or post on the qooxdoo-devel mailing list.

Tutorial: Automated UI Testing with the qooxdoo Simulator

Having previously covered unit testing, it’s time to take a look at qooxdoo’s built-in facilities for automated UI testing. Over the course of this tutorial, we’ll set up the required infrastructure and develop a test case that interacts with the Twitter application from the previous tutorials. As always, the finished code is available on GitHub.

Click to enlarge

Simulator: Selenium support for qooxdoo

The Simulator component provides the infrastructure necessary to write GUI tests for qooxdoo applications and execute them in a real web browser by way of a Selenium server.
The Simulator is based on those parts of the Selenium project that were formerly known as “Selenium RC” and are now referred to as “Selenium 1″. While this tutorial doesn’t require in-depth Selenium knowledge, you should at least familiarize yourself with its basic concepts and capabilities before reading on.

The testing API: QxSelenium

Simulator Test cases are defined as qooxdoo classes inheriting from simulator.unit.TestCase. Similar to unit tests, they live in the namespace of the application they’re testing and support the setUp/testSomething/tearDown pattern. Test methods interact with an application by using the QxSelenium API. This consists of the DefaultSelenium API plus several qooxdoo-specific additions. You can get an API reference for these by running generate.py api in the qooxdoo SDK’s component/simulator directory and then opening /component/simulator/api/ in your browser.

Setting up the infrastructure

For the purposes of this tutorial, we’ll assume that you’re using a working directory named workspace which contains the Twitter tutorial application in a subdirectory named qooxdoo-tutorial. Replace these paths with your own as appropriate.

The test browser will load the application under test (AUT) over HTTP, so make sure you’re running a web server and qooxdoo-tutorial is accessible. If you don’t want to install a full-blown HTTP server like Apache, you can use Python’s built-in web server module. To do so, open a new shell in your workspace directory and run this command:

python -m SimpleHTTPServer

You should now be able to open the tutorial application by browsing to http://localhost:8000/qooxdoo-tutorial/build/index.html.

Also, a regular Java Runtime Environment (JRE) is necessary on your machine to run Selenium.

Required Libraries

The Simulator depends on these external libraries:

Starting the Selenium Server

In a real testing environment, the Selenium server will probably run on a separate machine – in fact, the same client might use different servers to run tests e.g. in Internet Explorer on Windows, Safari on OS X and Firefox on Linux. To keep this tutorial straightforward, however, we’ll run the server on the same machine as the AUT.
Wherever selenium-server.jar is located, in order to test qooxdoo-based applications it needs to use the qooxdoo user extensions for Selenium. They’re located in the Simulator component within the qooxdoo SDK, so start the server with the -userExtensions option set accordingly by running this command in a new shell window:

java -jar selenium-server-standalone-2.5.0.jar -userExtensions <QOOXDOO_PATH>/component/simulator/tool/user-extensions/user-extensions.js

The server should now be listening on the default port, 4444.

Test Configuration Settings

The Simulator needs several configuration settings in order to run:

  • The paths for the Rhino and Selenium Client Driver JARs
  • the host name and port of the Selenium server
  • the browser to be used for the test
  • and the URI for the application under test

All these settings are defined by overriding the simulation-run job in config.json (don’t forget to uncomment the “jobs” section if necessary):

"simulation-run" :
{
  "let" :
  {
    "SIMULATOR_CLASSPATH" : [
      "../selenium-java-2.5.0.jar",
      "../libs/*",
      "../js.jar"]
  },
 
  "environment" :
  {
    "simulator.selServer"   : "localhost",
    "simulator.selPort"     : 4444,
    "simulator.testBrowser" : "*firefox",
    "simulator.autHost"     : "http://localhost:8000",
    "simulator.autPath"     : "/qooxdoo-tutorial/build/index.html"
  }
}

The simulator.testBrowser key is particularly noteworthy. The value must be one of the browser launcher strings supported by Selenium. *firefox (for Firefox 3+) and *googlechrome should work fine on any platform provided you’re using Selenium 2.x as described in this tutorial. *safari usually only works on OS X. Internet Explorer requires some additional configuration but generally works fine for what it is. Whichever browser you choose, it must be installed on the machine that runs the Selenium Server.

The simulator.autHost and simulator.autPath settings are combined to form the URI of the tested application. Adjust these depending on your web server configuration. Also note that you can test either the source or build version of the application – just make sure it’s generated before launching the test suite by running generate.py build or generate.py source.

Making the jobs available

The Twitter tutorial application was created before the simulation-* generator jobs existed, so if you downloaded the tutorial code from Github, you’ll get a “No such job” error if you try to run them. To fix this, you need to add both simulation-build and simulation-run to the “export” list at the top of the application’s config.json file. This is not necessary for application skeletons created by more recent qooxdoo SDKs (1.3 and later).

Defining a test case

Now that we’ve got our infrastructure set up, we can finally start writing tests. First, create a new subfolder named simulation in qooxdoo-tutorial/source/class/twitter. This is the default location for Simulator tests. In this folder, create a new file named Settings.js. This will be our test case that is going to interact with the Twitter application’s settings dialog. For now, just add a test method stub that will cause the test to fail:

qx.Class.define("twitter.simulation.Settings", {
 
  extend : simulator.unit.TestCase,
 
  members :
  {
    testChangeLanguage : function()
    {
      this.fail("Test not implemented!");
    }
  }
});

Building and running the test suite

Time to see the Simulator in action. In the Twitter application’s directory, run generate.py simulation-build to create the test application. Note that there is no simulation-source job (yet) so you must run simulation-build every time you modify your test classes.

Once the build job is finished, run generate.py simulation-run. Assuming everything’s set up correctly, two Firefox windows should (very briefly) open up and you should see the result of the failing test right on the shell:

----------------------------------------------------------------------------
    Initializing: qooxdoo-tutorial
----------------------------------------------------------------------------
>>> Processing configuration
  - Warning: ! Shadowing job "simulation-run" with local one

----------------------------------------------------------------------------
    Executing: simulation-run
----------------------------------------------------------------------------
>>> Running Simulation...
>>> Load runtime: 87ms
>>> Loading tests...
>>> 1 tests ready
>>> Simulator run on Thu, 08 Sep 2011 14:22:29 GMT
>>> Application under test: http://localhost:8000/qooxdoo-tutorial/build/index.html
>>> Platform: Linux
>>> User agent: Mozilla/5.0 (X11; Linux i686; rv:6.0.2) Gecko/20100101 Firefox/6.0.2
>>> Running tests...
>>> Main runtime: 8887ms
>>> Finalize runtime: 0ms
>>> Assertion error! Test not implemented!: Called fail().
>>> Stack trace: 

>>> ERROR  twitter.simulation.Settings:testChangeLanguage
>>> Test not implemented!: Called fail().

>>> Test suite finished.
>>> 0 passed, 1 failed, 0 skipped.
>>> Simulator run finished in: 0 minutes 15 seconds.
>>> Done (0m17.20)

You’ll notice a warning about the “simulation-run” job being shadowed. Since we’re doing that on purpose, we can silence this warning by adding the top-level key “config-warnings” to config.json:

"config-warnings" :
{
  "job-shadowing" : ["simulation-run"]
},

Test development

Let’s replace that stub with something useful now: We want Selenium to use the Twitter application’s preferences window to change the language.
But first, we should set Selenium’s execution speed (the delay after each command is excuted) to a value that will allow us to actually see what’s going on, say one second. To do so, replace the this.fail line:

testChangeLanguage : function()
{
  this.getQxSelenium().setSpeed(1000);
}

The first real action of the test will be to click the “Preferences” button. This leads us to one of the main challenges when developing Selenium tests: How to locate the right element.

Locator strategies

Elements can be located using several different strategies, generic as well as as qooxdoo-specific ones. See the manual for an overview:

Simulator: Locating elements

In this tutorial, we’ll focus on the qxhv locator. Just like qxh, it traverses the application’s widget hierarchy, using a syntax similar to XPath to match the widgets it finds to criteria defined by the user.

Note: The Selenium IDE Firefox add-on and the qooxdoo Inspector can be very helpful tools for finding locators and debugging Selenium tests.

The qxhv locator allows us to find any widget with a given “label” property value:

qxhv=*/[@label=Preferences]

A word about locales

As you’ll be aware if you’ve completed the Translation tutorial, the Twitter application is localized and will automatically switch the display language if the locale of the browser it’s opened in matches one of the supported languages (German, English, French and Romanian). This means that depending on the locale of the browser you’re using to run the test suite, you may have to adjust the target value of the Preferences label locator step, e.g. qxhv=*/[@label=Einstellungen] for a German language browser.

Executing commands

To simulate a user clicking on the target identified by the locator, we need to combine it with the qxClick command:

// Click the Preferences button
var preferencesButtonLocator = "qxhv=*/[@label=Preferences]";
this.getQxSelenium().qxClick(preferencesButtonLocator);

This should open the Preferences window. To make sure the command worked, we can employ the isElementPresent command, then use an assertion so the test will fail if the window didn’t open:

// Check if the Preferences window opened
var settingsWindowLocator = "qxhv=[@classname=twitter.SettingsWindow]";
var settingsWindowPresent = this.getQxSelenium().isElementPresent(settingsWindowLocator);
this.assertTrue(settingsWindowPresent);

If the settings window was a qx.ui.window.Window, we could simply use the class name as the locator step. But that only works with classes from the qx.* name space. For a custom widget class like twitter.SettingsWindow, we need to search by classname, a plain JavaScript attribute supported by all qooxdoo objects. The @propertyName=value locator step covers these as well.

All right, time to execute the test again (don’t forget to run simulation-build again first). Assuming all went well and the test passed, the next step is to select one of the language options from the Preferences window. qx.ui.form.RadioButton also has a label property (inherited from qx.ui.basic.Atom), so we’ll use that:

// Click the radio button for Romanian
var romanianLabelLocator = "qxhv=[@classname=twitter.SettingsWindow]/*/[@label=Romanian]";
this.getQxSelenium().qxClick(romanianLabelLocator);

Obviously, if your browser’s locale is Romanian, this option will already be selected so you should choose a different one.

Following that, we want to close the Preferences window. The close button doesn’t have a label, but we can find it by looking for the file name of its icon:

// Click the window's close button
var windowCloseButtonLocator = "qxhv=[@classname=twitter.SettingsWindow]/qx.ui.container.Composite/[@icon=close\.gif]";
this.getQxSelenium().qxClick(windowCloseButtonLocator);

We don’t need to use the full resource ID of the icon since the [@property=value] step treats the value as a regular expression.

Again, we’ll use isElementPresent to check the result:

// Check if the window was closed
settingsWindowPresent = this.getQxSelenium().isElementPresent(settingsWindowLocator);
this.assertFalse(settingsWindowPresent);

This would be a good time to re-generate and run the test to make sure everything works as expected.

Verifying the language change

For the final step of this tutorial, we’ll check if the language change was correctly applied to the twitter application. The first approach might be to use isElementPresent to check for the Preferences button with the translated label value (e.g. “Preferinte” for Romanian). That won’t work, however, since the value of the “label” property is a  qx.locale.LocalizedString object, so the [@property=value] locator step will try to call toString on it. This will return the original, untranslated label so the check will fail. To get the visible, translated string, we need to call the LocalizedString’s translate() method. That’s where QxSelenium.getRunInContext comes in: It takes a locator and a snippet of JavaScript code which it uses as the body of a new function. This function will then be called in the context of the widget identified by the locator, i.e. “this” will reference the widget instance. The function’s return value is then serialized as JSON and returned by getRunInContext. We can use this to compare the translated label value to what we’re expecting:

// Get the translated string for the Preferences button label
var translatedLabel = this.getQxSelenium().getRunInContext(preferencesButtonLocator,
"return this.getLabel().translate().toString()");
// Check if the label was translated
this.assertEquals("Preferinte", translatedLabel);

And that’s it for this introduction to the Simulator. If you have further questions or encounter any problems getting the tutorial code to run, leave a comment or contact us on the qooxdoo-devel mailing list.

Basic Testrunner

With qooxdoo 1.5. we introduced the basic skeleton, a template for applications running in “browserless” JavaScript environments such as node.js or Rhino. Recently, we added experimental test and test-source generator jobs to this skeleton. Much like their counterparts from e.g. the GUI skeleton, they create a test runner for the application’s unit tests (usually located in source/class/<custom>/test). The basic runner will automatically execute all unit tests and log the results to the shell when invoked:

generate.py test
rhino test/script/testrunner-basic.js

The output will look something like this:

Load runtime: 92ms
PASS  basicapp.test.DemoTest:testSimple
PASS  basicapp.test.DemoTest:testAdvanced
Main runtime: 19ms
Finalize runtime: 0ms

Due to the lack of a full browser environment, some features of qooxdoo’s unit testing framework, such as MMock and MRequirements aren’t currently supported, although that might change soon. Asynchronous tests, however, should already work as usual.