Here is the presentation about testing Web apps with Selenium RC.

The presentation has three parts. The first is a very generic introduction to Web testing in general and should be beneficial to people who want to get an orientation in the field; the second part introduces Selenium RC, this should be beneficial for people looking at concrete test products; and the third part shows how to extend Selenium RC with custom code, which should be beneficial for people committed to Selenium RC.

| View | Upload your own

The slide show is double length, first the normal presentation slides for a quick overview, and then all the slides again with notes which should make it easier to get to grips with the material without the oral narrative.

If you are thinking about testing web apps, you might get interested in taking and comparing screen shots. I've been involved with Selenium RC a bit these days, a test automation tool for web apps. Selenium RC (or SRC for short) offers a captureScreenshot facility. The problem was: If you want to compare images mechanically, you want to make sure they contain only what you are interested in - the web page under test. The SRC command captures the whole screen. So I put a bit of time and effort into having SRC take screen shots from the browser page only. If possible, no desktop, no other windows, no chrome. This makes the screen images smaller and rips off the visual noise that will spoil the comparisons. What we have come up with seems to work so far, and if you are interested in the technical details, surf over to a post on Selenium RC's user forum.

If this works out nicely we might set up a test automation to test low-level and widget-level rendering capabilities of qooxdoo. With a few suitable test apps (like those from the DemoBrowser) we will run through the test once, take all the screenshots, making sure the pics taken are correct. Then run the tests automatically, taking the same screenshots again and comparing them to the master copies. Doing this again and again while developing the framework we should finally get a nice regression test. With this infrastructure you can compare the images directly (e.g. using tools from the ImageMagick suite), compare them on the basis of their digest (MD5, SHA), throw away the correct ones, keep the problematic for human inspection, create "diff" images and so on.

Sounds like fun.

The current legacy_0_7_x branch of qooxdoo now supports a make test-source target (besides the existing make test target). So if you invoke that target in a Skeleton-based application environment, it will create a TestRunner instance for your application. The new thing: With make test-source the test frame can link into the source tree of your application. This allows for test-driven development where you simultaneously develop your source classes, the test classes and run the tests. See the TestRunner wiki page for more information. (Thanks to Istvan Ballok who provided the initial patch for this).

See the TestRunner in action.

Testing web apps in general and qooxdoo apps in particular has been a recurring issue in the community (e.g. see this wiki page and the links provided here).

The new Simulator project aims to facilitate testing of qooxdoo apps through integration with Selenium, a popular, open-source web testing framework. The nice thing about Selenium is that it also lends itself well to automating web pages ("simulations"), hence the name.

The project is still in its early stage, its main contribution being a qooxdoo-specific extension to the Selenium engine. This extension adds commands and so called locators to Selenium that hook into qooxdoo's object system and ease the way a test interacts with qooxdoo. If you are concerned with testing qooxdoo apps, or are already using Selenium for your work, this qooxdoo-contrib contribution might be interesting. Other parts of the project include a small sample application with an interactive Selenium shell and ready-to-run test scripts that can also be used as templates for your own endeavours.

Check it out and visit the Simulator project home page.

This is a small continuation of our original box model test. We added some tests to explicitely check for overflow problems already noticed during development of qooxdoo 0.5. We simply wanted to be sure about the current state of browsers before starting the development our new layout engine.

Results

Good news first: Konqueror (version 3.5.3) renders equally perfect for both overflow:hidden and overflow:visible. The box-sizing is correct. In all examples! Again kudos to the developers!

Opera 9.0 shows the same errors for overflows as without overflows. Fine. We are already happy if we need not to handle another special situation...

Internet Explorer 6.0 renders 2 boxes smaller in Standard Mode and even completely fails to render 2 boxes using "overflow:hidden" in Quirks Mode. The boxes were already wrong layouted before. So luckily it is not getting worse.

Firefox 1.5 shows differences compared to Firefox 1.0. Now we noticed the first box model difference between Gecko 1.7 and Gecko 1.8. Firefox 1.5 renders two more boxes wrong than without the overflow statement, both using "box-sizing:border-box". All other boxes are identical.

Surprisingly, for Firefox 1.0 the "overflow:hidden" fixes all the bugs on the right columns, i.e. making "box-sizing:border-box" to work perfectly, but breaking many boxes on the left side. This means Firefox 1.0 is the only browser in this roundup which fails to render the W3C-default "content-box". In general it supports all other CSS standards used in the tests (compared to Internet Explorer for example).

Complete set of test scenarios

• Standard Mode, Overflow visible
• Standard Mode, Overflow hidden
• Quirks Mode, Overflow visible
• Quirks Mode, Overflow hidden

Updated Screenshots

• Firefox 1.0 (Overflow, Visible, Overflow: Hidden)
• Firefox 1.5 (Overflow: Visible, Overflow: Hidden)

In preparation of our new layout engine we did a roundup of all modern browsers' abilities to render under the two different box models. It definitely is one of the most compelling questions in advanced web development: "Which browser supports what layout features and how well?". Like most other javascript toolkits qooxdoo currently depends on a box-sizing of "border-box". While not being the default box model in current browsers, many people think that it is the most natural way to define web layouts, if not layouts in general. The W3C-favored "content-box" model on the other hand is often seen as a rather counter-intuitive solution. Our tests particularly reveal how well the common use cases work in "border-box" regime.

Tested browsers and Screenshots

• Firefox 1.5 (Standard, Quirks)
• Internet Explorer 6.0 (Standard, Quirks)
• Internet Explorer 7.0 Beta 3 (Standard, Quirks)
• Opera 9.0 (Standard, Quirks)
• Konqueror 3.5 (Standard, Quirks)
• Safari 2.0 (Standard, Quirks)

Firefox 1.0 and Firefox 2.0 beta 1 behave identically to Firefox 1.5. Browsers based on Gecko 1.6 behave differently (screenshots may be provided soon).

Results

Konqueror is the only current browser which renders all test examples flawlessly. There seems to be no box model layout bug. Just perfect. Kudos to the developers! Internet Explorer 7.0 (only in Standards Mode), Firefox 1.5 and Opera 9.0 show perfect results as well, but only in the "content-box" tests. Unfortunately, each of these browsers exhibits individual problems in the "border-box" model.

Internet Explorer 6.0 and Internet Explorer 7.0 render equally (wrong) in Quirks Mode. Internet Explorer 7.0 comes with support for stretching (i.e. define left and right or top and bottom together) and min- and max-dimensions, but only in Standards Mode. At least one of the very few improvements due to the IE7 revamp. Both Internet Explorer versions do not support the handy "box-sizing" property at all. Choosing either standard compliant or quirks mode is achieved by an appropriate doctype declaration as usual. Keeping in mind IE's inability to work in a "border-box" regime, it is obvious that the second part of each test document (the two columns to the right) are always wrong. While Internet Explorer 7.0 renders flawlessly in Standards Mode, Internet Explorer 6.0, lacking many CSS features, only renders exactly one of all the squares correctly. Which one depends on the current render mode, of course. Quite dissappointing, but not unexpected results.

Mozilla Firefox as of version 1.5 shows an almost perfect rendering behavior. Surprisingly, it doesn't follow the "box-sizing" rule, if no width or height is explicitly defined. This means for example that it isn't possible to only define a min-width without setting a (bogus) width as well. Long distance calling cards. A work-around for such a bug in "border-box" mode could be to add an appropriate width or height to each min-width, min-height, max-width or max-height statement. As already mentioned the W3C default value "content-box" yields absolutely perfect results.

Opera 9.0 has many strange problems. Sometimes the box is too large, sometimes too small. It appears to be somewhat inconsistent as well: while the box widths are often ok, the box heights are plain wrong. Hard to make sense of such an asymmetric behavior. Opera 9.0 in the "content-box" model is perfectly well-behaved, though.

Please have a look at our tests documents online, either our standard compliant test or quirks mode test.

Any ideas or comments welcome!