Wednesday, June 29, 2016

New indexes boost performance with relational databases in Orbeon Forms 2016.2

When you create a form with Form Builder and use Orbeon Forms' implementation of the persistence API for relational databases, data collected by the form at runtime is stored in a set of 4 generic predefined tables. The tables are generic in the sense that they are not form-specific. This has a number of benefits:
  • It gives more control to DBAs: they can create the tables when Orbeon Forms is installed, and since at runtime Orbeon Forms doesn't need to have the permission to create tables, this allows DBAs to have a full knowledge of the database schema, and to optimize its storage or create additional indexes, should they wish to do so.
  • It keeps the database schema simple, and uniform across deployments. (Contrast this to an approach where you would have one table per form, and per version thereof.) Because of this, no "garbage collection" of unused tables (unused form versions) is necessary, upgrades to new versions of Orbeon Forms that require a change to the database schema are simpler, all the operations performed by Orbeon Forms can be part of a transaction (not all databases support DDL to be part of a transaction), and more.
For the tables to be generic:
  • Metadata (such as the form data id, the data owners, the time of the last change), which is the same for all forms, is stored in regular columns.
  • Values entered by users in your form are stored as XML, using a single "generic" xml column.
This works extremely well for operations where Orbeon Forms needs to either read or store all the values for a given instance of a form, like when users save or open form data. However, things gets more complicated when you want to do reporting on the submitted data, like:
  • For a given form, in a table, show the values of 5 significant fields for the last 50 submissions.
  • For a given form, find form data where the field last name has the value Smith.
Those operations are performed by summary pages, and the search API. In Orbeon Forms 2016.1 and earlier, they required SQL to extract data from the XML stored in the relational database. Unfortunately:
  • Some databases only have minimal support for XML.
  • Even for those databases with good support:
    • Searching and extracting values from XML data still isn't as fast as if the data was stored in "regular columns".
    • Optimizing those queries on XML data is highly database-dependant, and requires the creation of indexes or other techniques that DBAs are not necessarily familiar with.
Because of this, in Orbeon Forms 2016.1 and earlier, the performance of summary pages and the search API degraded as more data was added to the database. To solve this problem, Orbeon Forms 2016.2 introduces index tables. Data is still stored as XML as it was before, but the subset thereof needed for summary pages and the search API is also stored in the index tables. As a result, summary pages and the search API never need to access data stored in XML, and can thus run much faster.

This also means that the values in the index tables need to be kept up-to-date by Orbeon Forms. This happens automatically as data gets saved, or forms deployed. However, if you're upgrading to 2016.2, for the summary pages and the search API to work properly you'll need to first populate those indexes, in an operation referred to as reindexing. For more on this, see how to reindex your Orbeon Forms database when upgrading to 2016.2.

Thursday, June 23, 2016

Saying farewell to HTML tables

A very important underlying feature of Orbeon Forms is the grid. A grid groups controls in rows and columns, and can optionally repeat rows.

When we first implemented grids many years ago, we used tables for the underlying HTML markup. In HTML, tables serve two main purpose:
  1. Presenting tabular data (which is their original purpose).
  2. Laying out other content.
The reason that tables got to be used for laying out content is that they offered layout features which no other HTML and CSS construct supported. Over time, however, CSS has introduced most of the necessary support, in particular display: table and related, as well as the more recent Flexible Box Layout (also known as "flexbox").

Because assistive technology such as screen readers usually describe tables to the user as being tables when reading a page, there is widespread agreement that one should not use tables for purposes other than presenting tabular data so as to not confuse the user.

So in the upcoming Orbeon Forms 2016.2 we have changed the layout of grids to no longer use HTML tables when possible.

There are two cases where we still use HTML tables:
The case of row spans is hopefully a temporary limitation: it is just plain harder to emulate row spans with pure CSS, and we hope that this won't be a big issue because the use of row spans in Orbeon Forms is fairly rare. Further, we now place the ARIA presentation role on such tables to help screen readers.

For the case of repeated grids, using HTML tables makes some sense because the data is actually presented in tabular form with headings, rows and columns.

The specific HTML layout is documented here.

As an aside, we have also removed the use of HTML tables in the error summary component.

We hope you like these changes which will be available in Orbeon Forms 2016.2!

Thursday, May 19, 2016

JavaScript components made easier

Orbeon Forms's toolbox, which presents the choice of form controls available when you build a form, is fully customizable. Each form control points to a file containing information about the control and, in the case of custom controls (also known as components), the behavior of the control itself.

Some components do not require any custom JavaScript code: you implement all the logic with XForms. These components often combine other existing controls (such as a date components made of separate input fields or dropdown menus) and "glue" them together, visually as well as functionally (for example the value of multiple fields are put together to build an ISO date string).

On the other hand some components encapsulate functionality mainly implemented in JavaScript. And in such cases, XForms and JavaScript must communicate. For example, when a component becomes visible, some JavaScript initialization must run. Similarly, a user interaction might dispatch a JavaScript event which must result in a change of the control's value in the form.

Orbeon Forms does this with the help of a JavaScript class called the component's companion class. Each JavaScript-based component defines such a class, which handles the component's lifecycle and holds custom data and functions. One instance of this class is automatically created by Orbeon Forms for each instance of relevant (visible) control. We call these instances companion instances.

And the good news is that since Orbeon Forms 2016.1, a new javascript-lifecycle mode lets Orbeon Forms handle important aspects of the component's lifecycle. This makes it much easier to implement JavaScript-based components.

All the details are available in the documentation of JavaScript Companion Classes. We hope you'll like this new feature of Orbeon Forms 2016.1!

Thursday, May 5, 2016

Speaking JSON

JSON is the most popular format for browser/server communication. It is used more and more for all types of web services and as time goes it becomes more and more likely that you will encounter web services based on JSON rather than XML.

So Orbeon Forms had to support JSON, which is why we worked hard to ship Orbeon Forms 2016.1 with JSON support out of the box!

We considered two levels of JSON support:

  • First, within the forms engine.
  • Second, from Form Builder and Form Runner.

With Orbeon Forms 2016.1, JSON support is fully integrated at the XForms level, and submissions (service calls) can send and receive JSON.

Because natively XForms is XML-based, JSON goes through a transformation when it is received, following the proposed XForms 2.0 specification. This is designed so that you can access the resulting XML with path expressions in a way very similar to how you do it in JavaScript. Say you have this JSON:


In JavaScript you use the expression mother.given to find “Mary”. From XForms, you use instead mother/given. It is pretty natural in many cases. For more details, see also submissions JSON support.

Similarly, XML goes through the reverse transformation when you need to send JSON to a service. The idea is that you can roundtrip JSON from a service.

With Orbeon Forms 2016.1, Form Builder and Form Runner only have limited JSON support, but receiving JSON from a service already works: if the service response contains application/json, it is automatically processed, and you can access the received data in actions via path expressions. For more deatils, see HTTP services JSON support.

In the future, we plan to make sending JSON to services from Form Builder / Form Runner possible and user-friendly, and we also want to allow sending form data in JSON format to services. Stay tuned!

Wednesday, April 20, 2016

Orbeon Forms 2016.1

Today we released Orbeon Forms 2016.1! This release comes with lots of new features and bug-fixes (in fact more than we had initially planned). We hope you'll like them!

NOTE: This release was formerly referred to as Orbeon Forms 4.11. You can read more on the new versioning scheme further below.

Major features and enhancements

Improved documentation site

While this is not directly associated with this release, we should mention that our documentation site has now migrated to a new location, simply Please let us know if anything is missing or incorrect!

New handwritten signature form control

This brand new form control allows you to include a handwritten signature with a mouse or touch screen. This works great also on the iPhone and iPad!

For more information, see the component documentation. This is an Orbeon Forms PE feature.

Whitespace trimming

This feature allows you to trim leading and trailing whitespace in values entered by users. It is enabled by default for new text fields and also lets you enforce that required fields cannot be left blank.

For more information, see the blog post.

Improvements to itemset actions

Actions have always been able to update a control's set of items, but that itemset was global. When using repeated grids or sections, itemsets can now be local to just one repetition. This enhancements even enables dependent dropdown menus.

Itemsets are stored alongside form data. This means that if you edit or review existing data, you are going to have the exact same sets of items present when data was entered in the first place (unless, that is, you run a service to refresh the itemsets again).

For more information, see the documentation.

Previous / next navigation in settings

Quickly navigate to the previous or next control, grid or section in the form, without missing anything.

For more information, see the blog post.

JSON support

The XForm engine has the ability to receive and send JSON documents as per XForms 2.0. With Form Builder, HTTP Services support receiving data in JSON format.

For more information, see the  Form Builder documentation and the XForms documentation.

Other features and enhancements

Form Builder

  • Reorganized toolbox. We have reviewed the grouping of controls in the toolbox for clarity. In particular, the Text Output control is renamed Calculated Value and grouped with the Explanation control. Selection controls are enhanced and better organized.
  • New common constraints. You can directly validate whether a number must be positive, positive or zero, negative or zero, or negative, without writing a formula. You can also set a maximum number of significant fractional digits for decimal numbers. (doc)
  • JSON support. Services support receiving data in JSON format. (doc)
  • Evaluate initial values in new iterations. Previously, the “Initial Value” formula would apply to initial iterations in a repeated grid or section when the form loaded, but not to new iterations added by the user while filling out the form. There is a new option to specify whether the “Initial Value” formulas applies to new fields when you add a new row to a repeated grid or a new section repetition to a repeated section. (doc, blog post)
  • Initial number of iterations within a nested repeated section. The new “Initial Number of Iterations Uses Template” option specifies, when an enclosing repeated section creates a new iteration, how many iterations the enclosed repeated grid or section will contain. (doc, blog post)
  • Controlling whether sections are open or collapsed initially. You can do this directly from the Form Builder UI. (Form Builder doc, doc)
  • Controlling whether sections are collapsible. You can control this globally or per form with properties, but you can also do this directly from the Form Builder UI for individual sections. For example, you could decide that the first section of your form cannot be collapsed, while subsequent ones can be. (Form Builder doc, doc)
  • HTTP Service editor improvements. The HTTP Service editor clarifies which HTTP methods take a body and features a new section to directly specify URL parameters when using the GET or DELETE method. (doc)
  • Number control for the Integer datatype. The number control previously only worked with the Decimal datatype, and a plain text field showed for the Integer datatype. Now we use the number control for Integer too, so you get the benefits of number formatting for this datatype.
  • Improved custom control settings. Custom control settings are much more flexible and you can use any form controls and layouts you want: dropdown menus, radio buttons, grids, and so on. (doc, blog post)
  • Extension points. Form Builder now allows you to add a custom Form Settings tab and Control Settings tab. (doc)
  • Versioning. You can overwrite not only the current version of the published form definition, but also previous versions. (doc)

New and improved form controls

  • Radio buttons with "Other". This new control shows a list of radio buttons predefined by the form author, but alsoa custom choice ("Other") which the user has the option to specify instead. (Form Builder doc, component doc)
  • Yes/No Answer. This control displays a simple "Yes" and "No" choice. It stores a value of true when "Yes" is selected and false when "No" is selected. (Form Builder doc, doc)
  • Single Checkbox. This control displays a single checkbox and by default stores a value of true when selected and false when deselected. (Form Builder doc, doc)
  • Wizard improvements. The wizard's table of contents indicates pages with errors more clearly. In validated mode, pages which cannot be navigated to are also indicated better. Last but not least, in validated mode, Save, Submit, Send, and other buttons which appear within the wizard only show when the user reaches the last page of the wizard. Finally, we made a few cosmetic improvements as well.

Mobile improvements

  • Improved Number capture. We show the iOS numeric keypad or the "numbers and punctuation" keyboard pane whenever possible. This allows for much easier numeric input. (docblog post)
  • Improved responsiveness. Mobile Safari notoriously introduces a delay upon touch, which makes interactive web apps sometimes awkward to work with. This iOS issue is fixed with iOS 9.3, but for users of older iOS versions, we have made clicks fast as well.
  • Zooming and dialogs centering. If you zoom in a form and a dialog shows, we nicely zoom out so the dialog can be read properly.

Other Form Runner improvements

  • Number rounding. The Number field has a new option to round values for formatting and capture values. (doc)
  • Improved loading indicator. Orbeon Forms features new and improved ways to tell you what's going on when entering data, saving, and submitting forms. (blog post)
  • Relational flat view improvements. The flat view supports nested sections and nested section templates. (doc)
  • Minimal appearance for repeated grids and sections. This optional appearance makes repeated grids and sections appear leaner. (blog post, grid component doc, section component doc)
  • Option to control email attachments. You can control whether all attachments, no attachment, or only selected attachments must be included in an email. (Form Builder doc, Form Runner doc)
  • Multi-page TIFF generation. In addition to generating PDF files, Form Runner can produce multi-page TIFF files. Those can be used for archival, for example. This is an Orbeon Forms PE feature. (doc)
  • Accessibility improvements. We have fixed a few accessibility issues.
  • Versioning fixes. We fixed a number of issues related to versioning when processing attachments and sending emails.
  • Language selector. We have switched to a dropdown menu for language selection, which makes the UI less crowded when a form is localized in many languages.
  • Calling JavaScript with parameters. You can pass parameters to JavaScript functions from XForms. (doc)
  • Easier XBL companion classes. It is much easier to expose JavaScript Companion Classes. This allows you to nicely integrate existing JavaScript widgets with Orbeon Forms. (doc)

Other new features and bug-fixes

Including the major features and enhancements above, we closed over 150 issues since Orbeon Forms 4.10


See Localizing Orbeon Forms for the latest localization support. Localization depends on volunteers, so please let us know if you want to help!

Browser support

  • Form Builder (creating forms)
    • Chrome 49 (latest stable version)
    • Firefox 45 (latest stable version) and the current Firefox ESR
    • IE 11 or Edge
    • Safari 8 or 9
  • Form Runner (accessing form)
    • All browsers supported by Form Builder (see above)
    • IE8, IE9 and IE10
    • Safari Mobile on iOS 7, iOS 8 and iOS 9
    • Chrome for Android (stable channel)

Compatibility notes

Deprecations and removals

  • Removal of legacy Rich Text Editor (RTE).
    • The rich text area expressed with <xf:textarea mediatype="text/html"> was using the YUI Rich Text Editor (RTE) which is no longer maintained by Yahoo! and has bugs under IE11 and newer. In this version, <xf:textarea mediatype="text/html"> now uses TinyMCE.
    • This is relevant only if you are writing XForms by hand, as forms created with Form Builder were already using TinyMCE.
  • Removal of Boolean Input field from the toolbox.
    • In this version, we introduce a "Yes/No Answer" control, and we have removed the "Boolean Input" control from the toolbox. Existing forms are not affected by this. You can still create new "Boolean Input" fields by:
      1. Adding an "Yes/No Answer" field.
      2. Changing its appearance to "Boolean Input".
  • Removal of Double datatype in Form Builder.
    • For compatibility reasons, the list still shows "Double-precision floating point" for existing controls with that data type, but not otherwise.
    • If you are using the Double data type, we recommend that you replace it with the Decimal type instead.
    • See this blog post for more information.
  • Removal of legacy Excel output processors.
    • The oxf:to-xls-converter and oxf:xls-serializer processors have been removed. These processors have not been maintained in a very long time, use the old Excel format, and are not used by Form Runner.
  • Deprecation of Scrollable List control.
    • This is not a control which is used often, and it is present mainly because browsers implement it and not much else. The component remains present but is removed from the Form Builder toolbox.
  • Deprecation of SQL processor variables (XML platform).
    • When using oxf:sql, support for the long deprecated $sql:position and $sql:row-position variables has been removed. Use the sql:position() XPath function instead.
  • Deprecation of legacy processors.
    • The following processors are deprecated and we plan to remove them in a future version, as they haven't been maintained and are based on ancient libraries and technologies. If you use any of them, please let us know.
      • Charts. The oxf:chart, oxf:chart-converter, and oxf:chart-serializer processors are deprecated.
      • XSL-FO. The oxf:xslfo-converter, oxf:xslfo-serializer and oxf:pdf-serializer processors are deprecated.
      • XMLDB. The oxf:xmldb-query, oxf:xmldb-insert, oxf:xmldb-update, and oxf:xmldb-delete are deprecated.
      • Excel. The oxf:xls-generator is deprecated.
      • Struts. The oxf:struts and oxf:struts-errors-generator are deprecated.

Other compatibility notes

  • XML data format.
    • The XML data format, as saved to the persistence layer, may contain additional XML attributes and elements. As of this version, this only happens when using the "Set Response Selection Control Items" action result.
    • All such attributes and elements are in the Form Runner namespace, defined as xmlns:fr="".
    • The following attributes may be added:
      • fr:itemsetid
      • fr:itemsetmap
    • The following element may be added, with children elements:
      • fr:metadata
    • If you implement your own persistence layer, you must make sure that all elements and attributes in the Form Runner namespace are preserved.
    • In the future, more attributes and elements in the Form Runner namespace may be added.
  • Default number of digits for number field.
    • This used to be set to 0 digits.
    • Now this is set to any number of significant digits present.
  • Setting URL parameters for GET/DELETE services.
    • In an action, setting a URL parameter for an HTTP service with a path of the form /foo/bar, where foo is a custom root element name, no longer works. Use /*/bar instead.
  • Actions setting a control's itemset.
    • When using "Set Response Selection Control Items", if the destination control is within a repeated grid or section, the behavior is changed:
      • It used to be that the itemset for all iterations of the selection control would be updated in all cases.
      • The new behavior is that the closest such controls are updated. This allows, for example, setting the itemset of a repeated control based on a change of value of another control in the same repeated iteration.
      • See the documentation for more information.
    • When such actions are used, saved form data in the database contains additional attributes and elements in the Form Runner namespace (see XML data format above). The purpose of those is to persist the content of the itemsets obtained dynamically.

About the new Orbeon Forms versioning scheme

Starting with this release we are using a new versioning scheme with the number of the year first followed by the number of the major release during that year. Since 2013 we have had three (sometimes four) major releases a year, and we plan to continue this schedule, so the plan this year is to have 2016.1, 2016.2, etc. and next year 2017.1, 2017.2 and so on, with minor releases, when needed, having an extra digit like 2016.1.1, 2016.1.2, etc. This doesn't change anything to your subscriptions or support.

Download and feedback

You can download the latest version of Orbeon Forms from the downloads page.

Don't forget to grab a trial license for the PE version.

Please send feedback:
We hope you enjoy this release!

Wednesday, April 13, 2016

How do you tell users "something is going on"?

Consider the following 3 different actions a user could take while on a form:
  1. Entering a new value in a field and tabbing out of that field.
  2. Pressing the save button.
  3. Pressing the submit button.
Each action will have an effect. For instance:
  1. When entering a new value, that new value might need to be validated, or another field re-calculated based on this new value.
  2. When saving, a round trip to the database is typically needed.
  3. When submitting, the data might need to be sent to another backend system, and a new page sent to the user.
Whatever the effect of the action is, there might be a period of time between the instant when the action is performed and the time the result of that action is ready to be shown to users. We hope that this period can be so short that it is close to imperceptible (say, < 100 ms). But in practice, this can't always be done, which means that we should provide users with some indication that "something is happening":
  1. While the form is updated in response to users changing a value, starting with version 4.11, Orbeon Forms shows a non-intrusive slim loading bar at the top of the page, as done natively by Safari when loading a page, and by popular websites like YouTube and GitHub during Ajax requests. This replaces the red "Loading..." indicator shown at the top-right of the page in Orbeon Forms 4.10 and earlier.
  2. When a button like save is pressed, you want to give users immediate feedback that something is happening, without preventing users from interacting with the rest of the page. (If they do a change while data is saved, that change just won't be saved, and users will get a chance to save it later.) Starting with version 4.11, when buttons like save are pressed, a spinner inside the button, as in the animation below.
  3. When users hit submit, they are often taken to another page. If they manage to change the value of a field after they hit submit, but before the next page loads, should they expect that change to be taken into account? Even if you can find plenty of websites that allow fields to be changed after the submit button is pressed, we think this practice is as at best confusing, and should be avoided. This is why, out-of-the-box, when a button like submit is pressed, while the action is performed, a spinner is shown at the center of the page, and the page is greyed-out preventing any user input. You can also see this in the animation below.

Finally, starting with version 4.11, you can configure, for each button, which loading indicator you'd like Orbeon Forms to use:
  • a spinner inside the button, without blocking user's input, called the inline loading indicator;
  • a spinner in the middle of the page, blocking user's input, called the modal loading indicator;
  • no spinner at all.
We think that those improvements go towards keeping users better informed of what is going on as they interact with a form, and allow form authors to have more control over how users are informed of what is happening.

Thursday, March 24, 2016

Flexible custom control settings

With Orbeon Forms, all form controls share certain properties, like the control name. However, some controls take additional properties which are specific to them and which you set at design time in Form Builder. For instance the Dynamic Data Dropdown and Autocomplete controls take:
  • a resource URL
  • XPath expressions to retrieve items, labels, and values from a service response
Controls like the Dynamic Data Dropdown are not set in stone: they are built on top of the Orbeon component framework, which allows you to create your own custom controls. And this is where the component author defines, when needed, which custom properties Form Builder must expose in the metadata of such components.

Until recently, you could only use plain text fields for custom properties. We have now made them much more flexible and you can use any form controls and layouts you want: dropdown menus, radio buttons, grids, and so on.

This opens the door to more and more flexible custom components. And the beauty of it is that it's entirely customizable. For example, the Dynamic Data Dropdown and Autocomplete controls now lay out the fields in a grid.

See also the Control Settings documentation, as well as the Control metadata documentation. We hope you'll like this feature, which will be available in Orbeon Forms 4.11.