Category Archives: Tool Chain

Generator: New Compiler Hints

Posted on by
2

qooxdoo supports a set of compiler hints (in other environments sometimes called pragmas) that instruct the Generator when creating an application. Characteristic of compiler hints is that they are embedded in the class source code, usually in a comment towards the top of the class file. In that, compiler hints have a sort of syntax and semantics that are hidden from the “normal” language (JavaScript) inside comments.

The Generator would then use these hints to include resources and classes into a build that it wouldn’t otherwise. Likewise, the Generator could be instructed to not care about a certain name reference in the code, e.g. when that name belonged to a separate JS library which would only be available at runtime, or is a dependency that should not be included in a build.

#-Hints

So far, compiler hints were given in a block comment starting with a # (hash or pound) sign. Then a keyword ensued, like asset, ignore or require, and then some sort of parameters, usually enclosed in parens. All this was wrapped in a JavaScript block comment. Here is an example of such a comment block:

/* ***********************************
#asset(custom/*)
#require(qx.dev.unit.TestCase)
#ignore(Dygraph)
*************************************/

@-Hints

Over the years we also adopted a variant of the JSDoc standard for documenting classes and methods within qooxdoo. These are again specially crafted JS comment blocks that have a certain inner structure, or syntax. Each such comment block allows you to describe e.g. the method at hand in a free text.

But JSDoc also provides you with special keyed entries – JSDoc calls them “documentation tags” – that can be used to describe certain aspects of the method more formally, like the type and meaning of its formal parameters or its return value (I will use the terms “tag”, “attribute” and “hint” here interchangeably). Due to their formal structure JSDoc comments can be processed programmatically, be extracted, analyzed and transformed, one result of which you can see in our Apiviewer.

Over time, along with the standard tags like @param or @return we established special tags that we found helpful in working with the code an its documentation. Different parts of qooxdoo would use these additional tags, such as the Lint checker (@lint) or the Demobrowser (@tag). A full reference for the tags we support is given in the manual.

So effectively we had to maintain two different types of special JavaScript comments, with two different parsers (however simple), two places to look up things, etc., when one of them would actually suffice.

One Syntax to Rule them All

It seemed natural and efficient to normalize everything into JSDoc comments and extend the set of recognized attributes with those utilized by the Generator at compile time. We did that, and with the next qooxdoo release #-hints will be deprecated in favor of the corresponding @-hints. It’s mostly a one-to-one replacement, with an occasional tweak. The migration guide for the upcoming version has all the details.

Special Case: @ignore

Particularly, the @ignore hint introduces some novelties over its #ignore sibling. Like all @-hints it is capable of variadic arguments. So instead of writing

#ignore(foo)
#ignore(bar)

you can write

@ignore(foo, bar)

to the same effect. (Of course you can still have two separate @ignore entries if you prefer that).

Double Relevance

More importantly, @ignore has relevance both as a compiler hint, but also for the lint checker, and in that supersedes the equally deprecated @lint ignoreUndefined() hint. I.e. using @ignore means

  • don’t warn about this symbol if you cannot resolve it (for the lint checker)
  • don’t include this symbol and its dependencies in the build (whether it is known or not – for the compiler. There was a blog post recently about issues around implementing this functionality.)

Name Globbing

As one more detail, @ignore is more strict about globbing. Already the old #ignore supported globs like #ignore(foo.*) to ignore entire namespaces, but also did some automatic globbing when it thought that foo was actually a class and not a namespace. The effect was that #ignore(foo) would also ignore foo.getBar when this looked like an attribute reference on a class object. (As you can imagine the problem lies in the term “looked like”. This decision could be safely made for known classes but not for unknown symbols. In an unknown “foo.bar”, is “bar” a nested namespace or a class attribute?!).

@ignore(foo) will ignore foo and only foo. If you also want foo.getBar be ignored either list it explicitly (as in @ignore(foo, foo.getBar)) or use a wildcard (as in @ignore(foo.*)) which will ignore both

Scoped Application

But most importantly, @ignore is lexically scoped. This was a major requirement, and one reason to integrate compiler hints with the JSDoc system. When people used an unknown symbol in one method, they wanted to ignore that specifically for that method and not globally for the whole file. As a consequence, using the same symbol in a sibling method you would again get an “Unknown global symbol” warning, which was desired. This was not available with the old #ignore.

So if an unknown name is found in a particular line of code, a lookup happens to the next enclosing lexical scope if this name should be ignored. If there is no such information in the enclosing JSDoc comment the search is repeated upwards, e.g. in the JSDoc preceding the class definition. If necessary this is repeated all the way up to the top-most JSDoc, which effectively takes the place of the old #ignore hint. So the old functionality is covered by a simple replacement of the different comment blocks, but the new system also allows a much finer control.

This scoped look up is not the case for other compiler hints like @require or @use which still scope over the entire class file.

So, all in all the new @-hints should be drop-in replacements for the old #-hints while being in line with the established JSDoc syntax, and being more versatile in some cases.

Branching in Configuration Files

Posted on by
Reply

In general there is no way of conditional branching in the configuration DSL of the tool chain. qooxdoo configuration files are JSON-based and have no “if” construct of any kind. So there is no way of directly expressing e.g. “If the value of this macro is true include this list of classes, otherwise include a different list”. But you can achieve much of the same by using the value of macros in references to other job names. Here is how to do that.

Includer Jobs

In qooxdoo configurations the general way to inject settings into a job is by using the extend key. So if you want settings from one job to propagate into another job you make the second job extend the first:

"jobA" : {
  "environment" : { "foo" : "bar" }
},

"jobB" : {
  "extend" : [ "jobA" ],
  ... // more job settings
}

So now jobB will get the environment setting from jobA as if you had written them into jobB directly (There is some conflict resolution going on if jobB already has an environment key). This of course makes more sense if you want to have more than one job inherit these settings, like when you substitute “jobB” with “source” and “build”. It’s a common way in qooxdoo configs to maintain multiply used settings in a single place.

As in this example the job names in the extend key can refer to jobs that do nothing on their own and are just provided to hold some setting to be used in other jobs (often referred to as “includer jobs”).

Using Macros in Job Names

What’s also interesting here is that the names in the extend key can contain macros. This allows you to select a job according to the value of some macro.

"jobA1" : {
  "environment" : { "foo" : "bar" }
},

"jobA2" : {
   "environment" : { "foo" : "xyz" }
},

"jobB" : {
  "extend" : [ "job${JobSuffix}" ]
  ...
}

By setting the value of the JobSuffix macro to either “A1″ or “A2″ you now select which job is being included into the extending job, and by that select the configuration keys and values that come with it.

In this example you could either specify the concrete value of JobSuffix in the global “let” section of the configuration file

"let" : {
  "JobSuffix" : "A1"     // or "A2"
}

or you could provide it on the command line when invoking the generator:

generate.py -m JobSuffix:A2 ...

Pairs of Includers

Often you will create pairs of includer jobs like in the above example for the same set of settings, to provide alternative values. To pick up the example from the beginning, to provide two different include lists to a source job you could write:

"oneInclude" : {
  "=include" : [ "foo.ClassA", "foo.theme.ThemeA" ]
},

"otherInclude" : {
  "=include" : [ "foo.ClassB", "foo.theme.ThemeB" ]

"source" : {
  "extend" : [ "${IncJob}Include" ]
}

(Don’t worry about the equal sign in "=include" for the moment). Now you only have to provide the value for the IncJob macro, either "one" or "other", and your source job will use the corresponding include lists.

This basically is it, you can now construct different jobs by just assigning different values to macros. Mind that this kind of selecting includer jobs will only work with values of macros, and only if those values are strings. This also means you cannot e.g. refer to the value of a specific environment setting. But in many cases you can work the other way round and make the value of an environment setting part of an includer job which is then selected by the value of a macro.

Compile-time Globals Checking

Posted on by
2

If you are slightly familiar with qooxdoo’s tool chain you probably came across the lint job that scrutinizes your code for known pitfalls. One of the major checks lint is doing is to find unknown global symbols. These are name references in your code that are neither known built-in classes, objects or functions (like Date, RegExp, arguments or isNaN), nor qooxdoo-style classes found in any of the involved libraries. These unknown globals can potentially break your code at runtime if you don’t take any measure outside the qooxdoo application to provide them, like including another script that contains their definitions in the index.html.

During compile time, i.e. when creating a runnable application from the library classes, the Generator does a restricted lint checking to detect some of the more fundamental issues. Unknown globals is one of them, and necessarily so as the Generator uses unresolved globals to add classes to the application build (so called “class list/dependency exploration”). When the new lint module was introduced all this checking was offloaded onto it with a suitable configuration, which worked fine functionally but was considered too slow performance-wise.

With recent master this has been amended. Unknown globals-checking is again done during dependency exploration to take advantage of the already existing logic. This was much faster but a major issue with the old implementation was the lack of support for @-hints embedded in JSDoc type code comments. These hints allow you to silence warnings about unknown symbols that you know will be available at runtime, and to guide further dependency exploration.

In order to add @-hint support and make it fast a new approach was taken. With every syntax tree of a class an adjacent tree of hint nodes is now calculated, derived from the corresponding JSDoc comments. Later when checking unresolved symbols this hint tree can be traversed and inspected very quickly, in order to decide whether an unknown symbols should be pursued, the corresponding class be added to the build, warned about or left alone.

The result is a fast unknown globals check that adds nearly no performance penalty to the existing compile functionality.

Generator: New ‘source-server-reload’ Job

Posted on by
2

There is a new experimental job in master, source-server-reload. The idea is to have the application automatically reload in the browser when source files change on disk. It is realized through the source-server mini web server that we introduced earlier. The reloading web server monitors the generated script (usually under source/script/<application>.js) and notifies the application in the browser about the new version. The client then reloads itself.

This also works if you load the main application through a standard web server like Apache or through the file:// protocol. The mini web server can still notify the application about necessary reloads, but you need to insert a script tag in your app’s index.html, e.g. like

<script type="text/javascript" src="http://localhost:44161/_active_reload/active_reload.js"/>

 

assuming that 44161 is the port the mini server runs on.

The new job can be nicely combined with the watch job that automatically re-runs the Generator when source files change. So start the watch job in one terminal and the reloading web server in another, and get changes of source files automatically propagated to the browser.