Here's another round of news from last week's project work.

Framework

Virtual Widgets

Last Friday we finished the first implementation of a prefetching feature for the virtual pane. Prefetching is a technique to improve the scroll performance of virtual widgets. Usually only the visible part of a virtual pane is rendered. This is how the current Table is implemented and it is the default for the new virtual pane. However, now it is also possible to precompute parts of the pane, which are just below or above the visible viewport. If the user scrolls the pane and the new visible area of the pane is covered by the precomputed area, no new content has to be generated. In these cases scrolling is as smooth as in non virtual widgets.

This can can be observed in the messenger demo. This demo has been extended and improved. Thanks to data binding (see further) it is now possible to add and remove contacts. Each list entry inside the messenger pane is represented by a Buddy Model which is held in an qx.data.Array. A controller class invokes an update of the GUI whenever a property of a Buddy is changed or the Array itself is modified.

Further we have added a Gallery example, which demonstrates another use case for virtual rendering. The demo shows two versions of the gallery widget. The first uses standard qooxdoo Atoms to render the cells and the second uses HTML based cell rendering.

Data Binding

During the whole week, documentation was a big topic for the data binding components. API-Documentation has been added to all three controller already introduced in the data binding blog post.
As new features for the controllers, filtering and sorting are in planning they will be added hopefully during the next week.

Bugfixes

A very annoying bug in IE concerning inline applications was just fixed today. Basically you have two ways to set up your inline application (in both scenarios you have to provide an element which is used as root):

• The application decides how much space it needs and renders itself respectively
• The application uses the size of the given root element

The latter case led IE to run into an infinite loop. This issue was reported under #1878.

Generator

Generator's main dispatch method has been revamped, so that jobs are run with more optimal tool chain setup (e.g. scanning libs, etc.), using trigger categories of increasing demand (some background can be found in this bug). There is a new (draft) page on default generator jobs which are automatically available if you are working with a skeleton-based application. Among others a bug has been fixed that messed with capitalization in file names on Windows platforms.

Applications

Testrunner

To make it easier to switch between a source and a build version of your unit tests, running "generate.py test-source" will provide you with an additional test/index-source.html file that loads the source version by default. This makes it also easier to stay with the source AUT across reloads of the whole app.

Contrib

Simulator

The Simulator contribution is currently being updated to work with qooxdoo 0.8 applications, and the project's SVN home will be re-organized accordingly. Only a few tweaks were necessary to make the custom qxh locators work, but due to safety features introduced in Firefox 3 they will break in Selenium IDE. Hopefully the Selenium IDE devs will find a solution for this soon. Until then, stick with Firefox 2 when testing qooxdoo applications in Selenium IDE. All test runs fine in Selenium RC.

Community

Atamert Ölçgen has written the first part of a nice tutorial on qooxdoo layouts. Way to go, Atamert!

That's it for now - take care 'till next week.

The new data binding layer in qooxdoo is getting into shape. After implementing the basic layer, the so-called Single Value Binding, model classes have been addressed:

Model classes

You can use any plain qooxdoo class as a model. qooxdoo's powerful dynamic properties are used to store and manage the data. An array-like model also requires its own implementation, because the data binding always needs to know, if something changes in the array. Therefore the native JavaScript Array cannot be used. The new data array can be found in qx.data.Array, and it can be used in almost the same way as the native Array, e.g. using methods like push() or the length attribute. The only difference is that you can not access the items in square bracket notation. Instead, to read data you use the "getItem(index)" method. "setItem(index, value)" on the other hand will store the value at the given index.

Architecture

Building upon the fundamental single-value binding layer and the model classes, controllers for connecting model and view have been implemented. The following diagram shows the general structure of the data binding component.

On the right hand side of the diagram you notice data stores. Those stores are responsible for fetching the data from a source, e.g. a JSON file or something similar. They will be implemented in the upcoming weeks.
On the left hand side the controllers can be seen which have already been implemented. They are built on top of the single-value binding. Currently, there are three controllers available. You may look for the table controller, which is still missing to call this list complete. The new Table, to be part of the currently emerging virtual widget infrastructure, will have built-in support for data binding. Until such a new Table has been introduced, binding of tables will not be available.

In the following section the three existing controllers will be introduced:

Object controller

The most simple and lightweight controller is the object controller. It connects a model object with one or more views. The data in the model can be anything a property can hold, i.e. a primitive data type like String or Number, or a reference type like a map. With that you can for instance bind views like textfields, sliders and other widgets visualizing primitive JavaScript types. But you can not only use views as targets. A target can be anything that has a property with the proper type.
Take a look at the following code example to see the object controller in action:

// create two sliders
var slider1 = new qx.ui.form.Slider();
var slider2 = new qx.ui.form.Slider();
// create a controller and use the first slider as a model
var controller = new qx.data.controller.Object(slider1);
// add the second slider as a target
controller.addTarget(slider2, "value", "value");

This code snippet ensures that every value set by slider1 will automatically be set as value of slider two.
As you can see, the object controller only wraps the fundamental single-value binding, trying to make its usage a little bit easier.

List controller

A list controller could - as the name suggests - be used for list-like widgets. The supported list-like widgets in qooxdoo are List, SelectBox and ComboBox, all in the qx.ui.form package. The controller expects a data array as a data model, that contains the model objects. These objects are displayed in the list and can either have some primitive type or be real qooxdoo objects.
A selection exists on the controller, which deals with model objects instead of the ListItems that are usually used by lists and selections of lists.
The following code snippet shows how to bind an array of strings to a List widget:

// data used to fill the model
var data = ["a", "b", "c", "d", "e"];
// create the model
var model = new qx.data.Array(data);
// create a list widget
var list = new qx.ui.form.List();
// create the controller
var controller = new qx.data.controller.List(model, list);

Now every change in the model array will invoke a change in the list widget.

Tree controller

Of course, also the tree does have its own controller. With that controller the Tree widget can automatically be filled with data from qooxdoo objects containing the data. The selection deals in a more abstract way - like the selection of a list controller - with model objects instead of actual tree folders.

A reasonable tree controller code sample would be too large for this blog post. But as a first proof of the applicability of all the controllers the inner workings of the feed reader (source code) have been changed significantly using the new data binding features. Feel free to take a look at the implementation and get a (hopefully good) feeling what it means to be using qooxdoo data binding. Comments welcome!

Another successful working week with progress in many areas:

Data Binding

There has been good advancement in implementing data binding in qooxdoo. The new controllers for binding list and tree widgets are almost done. As a first proof of its applicability the inner workings of the feed reader have been changed significantly using the data binding components. The next step will be the realization of data stores, which for instance will allow to bring JSON files into the data structures needed for the data binding. A more detailed blog post about the state of the data binding implementation is to appear next week.

New Selection API

In the course of the current implementation of data binding we have seen that the existing API for selection handling is not as consistent as it should be. Tree, List and SelectBox implement different methods and events for selection handling. To fix the issue, we have implemented a new selection API. It is split into two interfaces ISingleSelection and IMultiSelection for single and multi selection handling, respectively. The new interfaces are now available in SVN trunk. The changes were made while deprecating the old methods. So if you work with the current SVN trunk, check the deprecation messages in your console and fix it by using the new methods.

Virtual Widgets

Work on virtual widgets is also progressing nicely. Check out yesterday's blog post, which has many details about the new concepts and the upcoming implementation.

Generator

There was a bug fixed that disallowed complex name spaces at the application level; this also touched on URI handling. Some revamping has started to have a more efficient dispatch of job actions in the main Generator module. The -v command line switch of generate.py now also causes byte sizes of the resulting class code to be printed when the build job is run. This allows to track how much size per class is gained by optimization options during build.

Bug fixes

For a complete list of bugs fixed during the last working week, use this bugzilla query. One of the bugs to mention was in the Label class that caused exceptions during locale switching when uninitialized labels were around. This issue has been resolved, which also fixed four unit tests in the framework.

Automated Testing

Speaking of unit tests: In order to help us keep up with our relatively fast-paced release schedule, we've been looking at ways to automatically build and run the framework tests in the spirit of Continuous Integration. The result is a process which uses the existing rudimentary "Build-And-Test" (BAT) infrastructure to update a local trunk checkout, generate the framework and start a Selenium RC test script. This script loads qooxdoo's Test Runner, runs the framework unit tests and dumps the results into Selenium's log file. Finally, an HTML report is generated and sent by email. Once some final kinks are ironed out, this process will be run nightly (or presumably at a higher frequency), hopefully aiding us in quickly recognizing issues during the development process.

Documentation

There was some interest in creating new themes for qooxdoo and, consequently, in creating clipped and combined images. Some documentation is already available. Maybe some interested people start on an extended version that will surface eventually. A migration guide for 0.8 to 0.8.1 has also been added to the manual.

New documentation in the form a several snippets has entered the qooxdoo wiki. Hope you benefit from those tips & tricks:

• Snippet: Create client variants of your application
• Snippet: Support for finding potential memory leaks
• Remote Table Model: Sorting your data

In general, we encourage everybody to participate in the ongoing effort to correct, improve and extend qooxdoo's documentation. Remember that the entire homepage is a wiki that everybody can log into and start modifying the docs right away.

Online demos

You certainly all know the qooxdoo online demos. Besides the regular demos that always point to the latest stable qooxdoo version (called "current", i.e. 0.8.1 right now), there are also links to the latest development snapshots of the demos (called "devel", i.e. 0.8.2-pre now). We began to implement a schedule for publishing new versions of those devel snapshots. Each Friday at 3pm local time (GMT+1) we'll take the current trunk, generate the demo applications and put them online. Of course, it may happen that trunk isn't always stable enough at that point in time, so that may lead to partial or even no updates. But we intend to meet those requirements (and ask all committers to try to stabilize their work at those timestamps as well), so that the qooxdoo community is served freshly-baked qooxdoo demos each Friday.

So much for a mouth-watering weekly status report. Enjoy.

Why does it matter?

With qooxdoo you are able to build full-blown single-page applications, typically with a large amount of widgets. Since the user stays on such a single page for quite a long time and the application is highly dynamic, new widgets are frequently created and existing ones are destroyed during the application's life cycle.
Every widget consumes memory and the aim of the application developer has to be to deal with the memory in an efficient way. Normally the application developer has not to think about the so-called garbage collection - it is built-in into every browser engine.
But the engines are more or less buggy, and to help the browsers qooxdoo offers destructors. Check out the manual page on Memory management for further details. To get an overview of qooxdoo's powerful support for finding memory leaks, read on:

Configure your application

At first you need to configure your application in a slightly different way to enable the logging of destructor messages. The snippet Finding potential memory leaks shows you how to do this.

Test your application

After you successfully customized your configuration you can re-generate your application with your newly created job and then (re-)load your application in the browser. In order to find out if any destructor reports about incomplete object disposal you can shutdown your application manually by executing

qx.core.ObjectRegistry.shutdown()

in your Firebug console (or similar JS console) and you'll end up with log messages like


35443 DEBUG: testgui.Report[1004]: Disposing: [object testgui.Report]FireBug.js (line 75)
Missing destruct definition for '_scroller' in qx.ui.table.pane.FocusIndicator[1111]: [object qx.ui.table.pane.Scroller]Log.js (line 557)
Missing destruct definition for '_lastMouseDownCell' in qx.ui.table.pane.Scroller[1083]: [object Object]Log.js (line 557)
036394 DEBUG: testgui.Form[3306]: Disposing: [object testgui.Form]FireBug.js (line 75)
Missing destruct definition for '_dateFormat' in qx.ui.component.DateChooserButton[3579]: [object qx.util.format.DateFormat]Log.js (line 557)
Missing destruct definition for '_dateFormat' in qx.ui.component.DateChooserButton[3666]: [object qx.util.format.DateFormat]Log.js (line 557)


Now go through this "checklist" provided by the log messages and fill up your "destruct" methods with the missing disposal code. Then you're done!

Last week we announced that we are starting to work on a general infrastructure for virtual widgets. We have received very positive and constructive feedback for this project. Today we presented our first ideas for the virtual widget infrastructure to the qooxdoo team.

Our basic idea is to have a strict separation between rendering and interaction. We want to provide a set of basic components, which can be used to create very different virtual widgets. The core of this framework will be the virtual scroll pane. The pane contains a set of layers, which contribute different parts of the rendered view. A layer is responsible to render one aspect of a given range of visible cells. Layers can e.g. be used to render row background colors, grid lines or the cell data. They are designed in a way, which makes it easy to write custom layers for special purposes.

One interesting experiment is a layer which uses real qooxdoo widgets to render the cells. First tests show that the scroll performance is adequate if the number of visible cells is not too high. This is very exciting because it offers a whole new set of use cases. As proof of concept we hacked together the user interface of an instant messenger application.

The interesting fact about this demo is that the performance is not limited by the size of the buddy list. The startup and scroll speed is exactly the same no matter whether the messenger contains 20 or 2000 contacts.

We are still exploring the possible uses of such a framework so if you have any ideas we'd like to hear about them.

In a recent post I wrote about I18N in qooxdoo. Another topic in that vein is Unicode handling.

Unicode is a general standard on encoding characters from natural languages all over the world. Basically, it is enumerating all those characters, starting from 0 to well beyond 1 million. On computers, these numbers are then instantiated, again using one of several encodings. One of the most popular of these encodings is UTF-8, a variable-length encoding where one character (actually, the Unicode number of that character which is called a code point) is encoded in one to up to four octets (8-bit bytes). The encoding goes like this:

Scalar Value                 First Byte Second Byte Third Byte Fourth Byte
--------------------------------------------------------------------------
00000000 0xxxxxxx            0xxxxxxx
00000yyy yyxxxxxx            110yyyyy   10xxxxxx
zzzzyyyy yyxxxxxx            1110zzzz   10yyyyyy    10xxxxxx
000uuuuu zzzzyyyy yyxxxxxx   11110uuu   10uuzzzz    10yyyyyy   10xxxxxx

(source: unicode.org)

The table shows how bits are distributed from the original code point ("Scalar Value") to the bytes representing it. There are other encodings, like the fixed-length UCS encodings. The nice thing about UTF-8 is that it is space-efficient and, maybe more importantly, fully backwards compatible with ASCII: a single-byte UTF-8 value is identical to its ASCII value (7-bit ASCII, that is). For code points beyond that a part of each byte is just tagging that byte (the leading 1* bits), while the lower portion of the byte holds the actual code point bits.

qooxdoo treats all source code files and generated output, in fact all text files, as UTF-8. This assures, among other things, that you can use all kinds of funny characters in string literals and comments in your code. Unicode characters in arguments to tr() are passed correctly to the corresponding .po files, and their Unicode translations are correctly retrieved at run time. Since browsers usually support UTF-8 they are even rendered nicely on screen .

Welcome to another weekly status update. This time we start with the tooling:

Generator

The Generator saw a bit of activity in the internationalization (i18n) sector. A recent blog post gave an overview of the support for translation within qooxdoo, and some bugs where fixed by providing patches for David Jean Louis' very decent polib implementation. As an aside, there was a lively little discussion about object method dispatch performance in Python on David's issue tracker.

Virtual Widgets

The Table is one of qooxdoo's most popular widgets. It allows for displaying huge data sets as it only renders and updates the rather small amount of those elements that are actually visible.

Most of the open issues of the Table in qooxdoo 0.8.x have been fixed within the last few days, which is a good base for tackling the next challenging tasks: This week the very first steps have been made in re-thinking the concepts for "virtualizing" widgets, backed by collecting both the existing features of the Table as well as possible new features of planned new Table or Table-like implementations. The community took a great interest in responding to the call for ideas, so thanks for all the initial feedback and your ongoing collaboration on that subject. We'll frequently post about the technical progress, so stay tuned for first proof-of-concepts.

CSS Selector

As already reported last week, qooxdoo now also comes with a built-in CSS selector engine. Work on integrating the increasingly popular Sizzle engine continued, mostly for allowing a rather jQuery-like coding style when working with collections of elements. This is still work in progress, but if you are interested in such a possible low-level usage of qooxdoo, you might want to check out those early new features.

Other improvements

The basic cookie API, that got lost towards qooxdoo 0.8, was reintroduced into SVN trunk to again be available with the next release. The long-lasting task of bugfixing and extending HtmlArea, the HTML editing component available from qooxdoo-contrib, is nearing its successful completion. This topic deserves a detailed blog post of its own, which should be ready during the next few weeks.

For a complete list of bugs fixed during the last working week, use this bugzilla query.

That's it for now. Have a good working week!

In qooxdoo we have two virtual widgets, the Table and the TreeVirtual. These widgets are virtual in the sense that they only create HTML markup for the visible part of the data. This enables us to render tables which can display any number of rows. The (scrolling) performance is only limited by the number of table cells displayed on the screen and not by the amount of data. Being used in almost all qooxdoo projects the Table is probably the most used complex qooxdoo widget. Over time many people have contributed to the Table and added new features. The result is that the Table can be adapted for many different uses but at the cost of a complex API and a rather monolithic code base. It is not possible to reuse only parts of the table. It's an all-or-nothing situation.

In spite of all these features there are some use cases which still cannot be satisfied by the current Table. Two often requested features which touch the core of the Table are:

1. Virtual horizontal scrolling: The Table generates only HTML markup for full rows. This slows down tables with a lot of columns.
2. Variable row heights: All rows in the Table always have the same height. It is not possible to give specific rows an individual height.

We plan to apply the lessons we have learned with the Table to a more general infrastructure for virtual widgets. The core will be a virtual pane, which will support virtual scrolling in both directions together with variable row and column sizes. This pane itself will have no external dependencies to other parts of the Table. One design goal is that this infrastructure can be used to create different kinds of virtual widgets. Other widgets, which would benefit from virtual rendering, include a gallery/icon list, combo boxes or trees.

Don't be afraid, the old Table is not going to silently disappear. We will continue to fix bugs and distribute it with future releases but expect new features to appear in the new Table.

We are very interested in user feedback. What do you like about the qooxdoo table? What do you dislike? Right now we are collecting ideas, features and use cases for this project on a wiki page. Feel feel to add your ideas, or leave a comment to this blog.

"I-what?!" Well, I18N stands for "Internationalization", a word that starts with an "I" and ends with an "N" and has roughly 18 characters in between. It's all about making a system support different natural languages and locales in its user interface.

If you are creating a qooxdoo application and want to support I18N, an important part is to allow for the translation of user visible strings. In a typical qooxdoo app you have those kinds of strings all over the place: as labels of buttons and menus, user instructions, help texts and pop-up messages. In the first step those strings simply appear as literals in your JS code.

The support of qooxdoo to aid you in translation now is two-fold: For one, you can wrap all strings in calls to one of qooxdoo's translate methods, like qx.locale.Manager.tr() (static) or this.trn() (instance). In your config.json you specify a list of locales, like "en", "en_us", "fr" asf., you want to see supported (in the LOCALES macro). When you then run generate.py translation, for each given locale a .po file is generated.

.po files are essentially text files that contain a list of translatable strings. The strings from the JS source code are used as keys in these files. Then, a translator can add a proper translation to each key. So, if you halfway expected qooxdoo to magically do the actual translation, I have to disappoint you; we're not quite there . But .po files are a fairly common way to organize translatable strings of applications, and were devised in the context of GNU's gettext project. Since we strive to rely on Python only in our tooling, we use David Jean Louis' polib module for Python to create and manage .po files.

After the keys in the .po files have been translated, the next invocation of generate.py source or build will pick up those translations and add them to the generated application output where they are available to the application at run time. A call to qx.locale.Manager.getInstance().setLocale() allows you to dynamically switch between locales. All strings that take part in translation are immediately resolved with their translation of the chosen language. To see this in action take a look at the Feedreader demo and play with the Preferences dialog.

One thing to note here is that calls to the tr*() methods in the code are evaluated twice: First, when generating the .po files (this is sort of a preprocessing step); and second, at run time when they return the proper translation to the running code. This also means that the first step needs string literals to pick them up as keys for the .po files. At run time tr() also runs fine with variable arguments so you can use this in your code. But you have to make sure that all values the variable might take appear elsewhere as string literals in a call to tr(). Otherwise your variable might take a value at run time for which no translation exists. (Some people resolve this issue by injecting dummy classes in their application that only hold calls to tr() on string literals. These classes do nothing at run time, and can actually be pruned from the build).

For more information on translation support see this wiki page.

We're back from holiday season and are adjusting to the regular work schedule. Since most of the core team members returned as recently as last Wednesday, this weekly status report is slightly out-of-sync.

qooxdoo 0.8.1

Hope you enjoyed the 0.8.1 release right before Xmas? We did some extended and thorough testing before releasing it on 2008-12-18, so we could head off on vacation one or two days later. During holiday season qooxdoo development came to a full-stop as planned. Well, almost, since some team members still took the chance to tackle some interesting tasks in their free time. We'll report shortly about the exciting stuff accomplished, so stay tuned.

Generator

The following changes already went into the 0.8.1 release, but didn't make it into a weekly status post before Xmas. It's about the new form of the compile-dist configuration key. Most of the old keys are retained, although sometimes under a slightly different name and interpretation. The key has been structured with sub-keys to provide better grouping of keys. Also, many semantic overlaps between keys have been removed, so the overall number could be reduced. There are no more required sub-keys (sic!), since common defaults are provided for each. The most common key to set is probably the paths/file key that specifies the (main) output file for the generated script. The former uri went into uris/script, resourceUri into uris/resource. For all the details, see the wiki reference entry.
This is an incompatible change, and old configs might break silently (i.e. provide wrong results)!

CSS Selector

One of qooxdoo strengths in creating rich internet applications is that it doesn't require the developer to know and use the low-level web technologies HTML, DOM and particularly CSS. This really pays off, not only in eliminating potential cross-browser issues (CSS nightmares anyone?). It also allows the developer to concentrate on a clean application design only by leveraging object-oriented JavaScript.

When integrating qooxdoo into existing pages of a more traditional web site, one of the features of other low-level libraries as Prototype or jQuery was still missing: a CSS selector engine. This missing piece is now also available in qooxdoo, thanks to Sebastian's work on integrating John Resig's Sizzle. Why re-invent the wheel, particularly for such a common functionality? Even if it seems to bring forward some selector "mono culture", as other libraries like jQuery or Dojo (are about to) include Sizzle as well...

In order to allow for an easy updating of Sizzle (expected to be a moving target for quite a while), the class qx.bom.Selector is merely a smart wrapper around the original Sizzle code and exposes some of its public methods. See the code in SVN trunk and the corresponding bom/Selector sample in the demo browser. Of course, this is work in progress and will see more improvements and extensions over time.

That's it for now, first weekly status update of the year is out. Guess you have also seen the recent blog post The year in qooxdoo (2008). If not, again all our best wishes for a Happy New Year 2009!