Branching in Configuration Files

In general there is no way of conditional branching in the configuration DSL of the tool chain. qooxdoo configuration files are JSON-based and have no “if” construct of any kind. So there is no way of directly expressing e.g. “If the value of this macro is true include this list of classes, otherwise include a different list”. But you can achieve much of the same by using the value of macros in references to other job names. Here is how to do that.

Includer Jobs

In qooxdoo configurations the general way to inject settings into a job is by using the extend key. So if you want settings from one job to propagate into another job you make the second job extend the first:

"jobA" : {
  "environment" : { "foo" : "bar" }
},

"jobB" : {
  "extend" : [ "jobA" ],
  ... // more job settings
}

So now jobB will get the environment setting from jobA as if you had written them into jobB directly (There is some conflict resolution going on if jobB already has an environment key). This of course makes more sense if you want to have more than one job inherit these settings, like when you substitute “jobB” with “source” and “build”. It’s a common way in qooxdoo configs to maintain multiply used settings in a single place.

As in this example the job names in the extend key can refer to jobs that do nothing on their own and are just provided to hold some setting to be used in other jobs (often referred to as “includer jobs”).

Using Macros in Job Names

What’s also interesting here is that the names in the extend key can contain macros. This allows you to select a job according to the value of some macro.

"jobA1" : {
  "environment" : { "foo" : "bar" }
},

"jobA2" : {
   "environment" : { "foo" : "xyz" }
},

"jobB" : {
  "extend" : [ "job${JobSuffix}" ]
  ...
}

By setting the value of the JobSuffix macro to either “A1″ or “A2″ you now select which job is being included into the extending job, and by that select the configuration keys and values that come with it.

In this example you could either specify the concrete value of JobSuffix in the global “let” section of the configuration file

"let" : {
  "JobSuffix" : "A1"     // or "A2"
}

or you could provide it on the command line when invoking the generator:

generate.py -m JobSuffix:A2 ...

Pairs of Includers

Often you will create pairs of includer jobs like in the above example for the same set of settings, to provide alternative values. To pick up the example from the beginning, to provide two different include lists to a source job you could write:

"oneInclude" : {
  "=include" : [ "foo.ClassA", "foo.theme.ThemeA" ]
},

"otherInclude" : {
  "=include" : [ "foo.ClassB", "foo.theme.ThemeB" ]

"source" : {
  "extend" : [ "${IncJob}Include" ]
}

(Don’t worry about the equal sign in "=include" for the moment). Now you only have to provide the value for the IncJob macro, either "one" or "other", and your source job will use the corresponding include lists.

This basically is it, you can now construct different jobs by just assigning different values to macros. Mind that this kind of selecting includer jobs will only work with values of macros, and only if those values are strings. This also means you cannot e.g. refer to the value of a specific environment setting. But in many cases you can work the other way round and make the value of an environment setting part of an includer job which is then selected by the value of a macro.

Compile-time Globals Checking

If you are slightly familiar with qooxdoo’s tool chain you probably came across the lint job that scrutinizes your code for known pitfalls. One of the major checks lint is doing is to find unknown global symbols. These are name references in your code that are neither known built-in classes, objects or functions (like Date, RegExp, arguments or isNaN), nor qooxdoo-style classes found in any of the involved libraries. These unknown globals can potentially break your code at runtime if you don’t take any measure outside the qooxdoo application to provide them, like including another script that contains their definitions in the index.html.

During compile time, i.e. when creating a runnable application from the library classes, the Generator does a restricted lint checking to detect some of the more fundamental issues. Unknown globals is one of them, and necessarily so as the Generator uses unresolved globals to add classes to the application build (so called “class list/dependency exploration”). When the new lint module was introduced all this checking was offloaded onto it with a suitable configuration, which worked fine functionally but was considered too slow performance-wise.

With recent master this has been amended. Unknown globals-checking is again done during dependency exploration to take advantage of the already existing logic. This was much faster but a major issue with the old implementation was the lack of support for @-hints embedded in JSDoc type code comments. These hints allow you to silence warnings about unknown symbols that you know will be available at runtime, and to guide further dependency exploration.

In order to add @-hint support and make it fast a new approach was taken. With every syntax tree of a class an adjacent tree of hint nodes is now calculated, derived from the corresponding JSDoc comments. Later when checking unresolved symbols this hint tree can be traversed and inspected very quickly, in order to decide whether an unknown symbols should be pursued, the corresponding class be added to the build, warned about or left alone.

The result is a fast unknown globals check that adds nearly no performance penalty to the existing compile functionality.

qooxdoo-contrib 2.0 – The Blueprint

Discussing an overhaul of the existing qooxdoo contribution infrastructure has been around for awhile, leaving traces in both the mailing list and our wiki. Beyond all written evidence we have been thinking and tinkering with a new concept on and off for quite some months. This posts sketches a basic outline of the new approach, in order to gather your feedback early on.

The new approach is not like npm or the package handling of Go but somewhere in the middle. It’s intentionally kept simple, and will mostly work through conventions of the people dealing with it rather than code. Most of this affects maintainers of contributions. Users of contributions will see only little change (affecting their config.json).

Reprise – What we wanted to achieve

The existing system maintains the code for all contributions in a single SourceForge SVN repository where they are also looked up and from where they are downloaded. The main goal with the new design is to decouple contributions from a specific store, so contrib authors would be free to develop their contribution on a platform they choose, whichever source control system and hosting provider they prefer (SourceForge, Github, Launchpad, Bitbucket, …).

Leaving the issue of code maintenance to the hands of the authors, the remaining issues are that of finding and using such distributed contributions. For the finding part we want to establish a qooxdoo-contrib catalog.

 The Catalog – A Github Repo

The catalog is just the index of known contributions with a bit of necessary meta information for each. To make maintenance of the catalog simple and straight-forward, we decided to implement it as a Github repository. It will have the same top-level directory structure you might know from the Sourceforge qooxdoo-contrib repo, namely

  • the contribution names top-level
  • beneath each contribution a set of directories with version names, and pot.a general Readme
  • in each version directory a Manifest.json file that describes this particular version of the contribution

The Manifest.json file is exactly the Manifest file you know from qooxdoo libraries, with a few new keys. Contribution maintainers can copy it straight over from their project to the catalog repo. You need to make sure that all necessary fields in it will be filled, particularly the info/download field. A new Generator job, validate-manifest, will help you with that.

That means maintaining your contribution in the catalog will require you to fork the catalog repo, make necessary changes and then issue a pull request which we will incorporate. (This obviously also means you will need a Github account to be a contribution maintainer, but we hope this requirement will be trivial).

Contributions come as Archives

The important thing here is that the info/download field will point to an archive (.zip or .tar.gz) containing the contribution. This puts an extra load on the contrib maintainers as they need to create archives from their contributions. But this is also more efficient and more robust for downloading than spidering Web-hosted files and directories, as is done in the old system (Anyone who tried to spider a directory tree from decorated HTML knows what I’m talking about). So while you authors are now free to maintain the source code of your contribution wherever you prefer you will need to find a place on the Web where you can place the corresponding archives.

Another new field in the Manifest.json for contributions will be information/checksum which will be a Sha1 checksum over the archive. This will be used by the Generator to both validate the archive after download as well as determine whether a contribution has changed online and needs to be re-downloaded.

Author’s Workflow

So the basic workflow for an author having a new revision of his contribution’s version is:

  • Create an archive containing the contribution and put it online.
  • Edit the contribution’s Manifest.json to contain the download location and a Sha1 checksum over the archive.
  • Copy this Manifest.json to the appropriate path in the catalog repo.
  • Send a pull request for the catalog.

Once the pull request has been incorporated your new revision is available to contribution users.

For an author choosing Github to host his contribution all this can be quite easily achieved (except for the checksum thing) by maintaining contribution versions as Git branches and exploiting the archive downloads that Github offers. You just use an URL like https://codeload.github.com/<user>/<contrib>/zip/<branch_name> as the Manifest’s download URL.

 Changes for Contribution Users

Users of contributions will just have to adapt their library/manifest configuration keys in config.json. For the new system they will use http:// URLs as values which point to the contribution and version in the catalog. Here is how that would look like for the popular UploadWidget:

{
  "libraries" : {
    "library" : [
      {
        "manifest" : "https://raw.github.com/qooxdoo/contrib-catalog/master/UploadWidget/0.2/Manifest.json"
      }
    ]
  }
}

 

That’s it. It looks a bit technical but will also allow hosting libraries online outside of any catalog, or for alternative catalogs e.g. maintained on Intranets. The traditional contrib:// pseudo scheme will continue to work for some time to support contributions living in the old system.

 Outlook

That’s the basic system and how to work with it. As said we encourage feedback, especially from contribution authors who will have the most exposure to it. It’s simple but will hopefully serve the basic needs of authors and users.

It’s easy to envision enhancements, like a nice, good-looking, searchable, sortable, category-driven Web interface to the catalog. It is also a question whether there should be a dedicated download step for the Generator (like generate.py update-contribs), instead of doing it all in the background and while building an application. But let’s discuss those when the basic system is in place. For now we just want to do the first step in the right direction.

Generator: New ‘source-server-reload’ Job

There is a new experimental job in master, source-server-reload. The idea is to have the application automatically reload in the browser when source files change on disk. It is realized through the source-server mini web server that we introduced earlier. The reloading web server monitors the generated script (usually under source/script/<application>.js) and notifies the application in the browser about the new version. The client then reloads itself.

This also works if you load the main application through a standard web server like Apache or through the file:// protocol. The mini web server can still notify the application about necessary reloads, but you need to insert a script tag in your app’s index.html, e.g. like

<script type="text/javascript" src="http://localhost:44161/_active_reload/active_reload.js"/>

 

assuming that 44161 is the port the mini server runs on.

The new job can be nicely combined with the watch job that automatically re-runs the Generator when source files change. So start the watch job in one terminal and the reloading web server in another, and get changes of source files automatically propagated to the browser.