Tuesday, August 23, 2016

Orbeon Forms 2016.2

Today we released Orbeon Forms 2016.2! This release features several important features and enhancements.

Major features and enhancements

New indexes for relational database performance

If you're using a relational database and rely on summary pages or the search API, this change will drastically improve the performance of those operations. If you're using a relational database and are upgrading to Orbeon Forms 2016.2, you should pay particular attention to this change, as it requires that you run a new reindexing operation as part of the upgrade procedure, and thus we recommend you proceed with the upgrade with special care, and make sure to read:

Form Options in Form Builder

Form Builder's Form Settings dialog now features a new Form Options tab, with the following options:
  • Whether the form is a singleton form. This option existed in previous versions under General Settings. (doc)
  • Whether the form uses the Wizard View. By default, the published form uses the configuration set via properties. (doc)
  • Appearance of control labels and control hints. The main use is to specify that text input fields hints appear as inline placeholders. By default, the published form uses the configuration set via properties. (doc)
Form Options

We expect that further form options will be added in the future.

Ability to send binary PDF to services

This feature has been requested often, and we are happy to include it in Orbeon Forms 2016.2! You can now specify that your submit or send action (or in fact any process) sends the form in PDF format to an HTTP endpoint. The same goes for sending a TIFF binary. For details, see the blog post and documentation.

Versioning comments

When using versioning, you can now add or update a textual comment associated with the given For details, see the documentation.

Versioning Comments

Form Runner function library

Form Runner now exposes a documented function library with easy to use functions. As of Orbeon Forms 2016.2, the library contains 28 functions, and we plan to add more in the future. For more details, see the blog post and documentation.

Accessibility / HTML layout

We have changed the layout of grids to no longer use HTML tables when possible. In addition, we have also removed the use of HTML tables in the error summary component. For more details, see the blog post and  documentation on the new HTML layout of grids.

Wizard enhancements

The wizard view features several enhancements:
  • We introduce a new option to show a separate table of contents instead of always showing it to the left of the content.
  • You can enable subsection navigation, which shows one nested subsection at a time instead of showing all subsections together.
  • The wizard now shows improved status options for each section.
Wizard Status

Buttons visibility

You can now control the visibility of buttons, and whether they appear disabled or not, using XPath expressions. For more details, see the documentation.

Other features and enhancements

Form Runner

  • Repeated grid/section: option to always insert at end (#2747)
  • Form Metadata API returns latest updated version instead of latest version (#2780)
  • Form Metadata API: option to return all non-deleted form versions (#2781)
  • Flat view support for SQL Server (#2790)
  • Proxy portlet: property to disable namespace mapping (#2764)

XForms engine

  • Ability to configure a custom function library (#2812)
  • XPath: support let expressions (#2816)
  • Internal changes to remove an old Java dependency on DOM4J. This will also be useful for our progress towards native/offline clients.

Other platform enhancements

  • XML processor to convert XML to JSON (#2796)
  • Ability for services to receive JSON (#2793)

Other new features and bug-fixes

Including the major features and enhancements above, we closed over 70 issues since Orbeon Forms 2016.1.

Internationalization

See Localizing Orbeon Forms for the latest localization support. Localization depends on volunteers, so please let us know if you want to help!

Browser support

  • Form Builder (creating forms)
    • Chrome 42 (latest stable version)
    • Firefox 48 (latest stable version) and the current Firefox ESR
    • IE 11 or Edge
    • Safari 8 or 9
  • Form Runner (accessing form)
    • All browsers supported by Form Builder (see above)
    • IE8, IE9 and IE10
    • Safari Mobile on iOS 7, iOS 8 and iOS 9
    • Chrome for Android (stable channel)

Compatibility notes

Persistence layer

The relational database schema contains changes in this version. Please refer to the notes at the beginning of this post on "New indexes for relational database performance".

Deprecations and removals

  • The oxf:from-xls-converter processor is deprecated and will be removed in a subsequent release.
  • The deprecated XMLDB processors are removed.
  • The oxf.fr.detail.hints.inline property is deprecated. Use oxf.fr.detail.hint.appearance instead. (doc)
  • xforms-disable-hint-as-tooltip is deprecated. Use oxf.fr.detail.hint.appearance instead. (doc)

Download and feedback

You can download the latest version of Orbeon Forms from the downloads page.

Don't forget to grab a trial license for the PE version.

Please send feedback:
We hope you enjoy this release!

Wednesday, August 10, 2016

Submitting a PDF file to an external service

PDF icon by Double-J Design
In addition to saving data to a database, Orbeon Forms can submit (or send) data to external services. If you just want to send the captured data as XML, it is very easy:
  1. Add the built-in "Send" or "Submit" button
  2. Configure the URL of the service
  3. Done!
Sometimes, you want to do more, and you can configure the details through what Orbeon Forms calls simple processes. These define a series of steps to perform in response to the user activating a button, including error recovery. You can, for example, specify that a "Submit" button must:
  1. Check that the data is valid.
  2. Save it to the local database.
  3. Email it to the user's email address.
  4. Send it to a workflow service via HTTP.
Until recently, though, it wasn't easily possible to send the form exported as a binary PDF to an external service. Orbeon Forms has a couple of options for PDF production, and there was a way to send a callback URL for the PDF to a service, but that was difficult to setup.

Orbeon Forms 2016.2 fixes this and includes a way to directly send the PDF to a service. So now you can, for example, write a "Submit" process which sends the XML data and the PDF to one or multiple HTTP services. You do this by passing the new pdf token to the built-in send action. (Similarly, you can now send a binary TIFF document with the new tiff token.)

We hope you will like this simple but hopefully useful new feature!

Wednesday, August 3, 2016

A stable function library for Form Runner

When using formulas or custom components with Orbeon Forms, there is often a need to access information about the current form, such as:
  • What is the current form name?
  • If this component currently running at design-time in Form Builder?
  • Is the form in read-only mode?
  • Is the form data currently valid?
  • Is the last page of the wizard currently visible?
In the past, some of these scenarios were possible by writing XPath expressions which usually accessed the internals of Form Runner. For example the following would return the current form name:
xxf:instance('fr-parameters-instance')/form/string()
Other scenarios were not even reasonably possible. Now, instead, Orbeon Forms exposes a documented Form Runner function library with easy to use functions. You now write the above simply:
fr:form-name()
You can use this function without worrying that it might go away or stop working due to a change in the Form Runner internals, and it is properly documented, alongside the other 27 Form Runner functions we are introducing for Orbeon Forms 2016.2. We plan to add more as we go, with 2 further functions already documented for 2016.3.

We hope you enjoy this feature, which will be available in the upcoming Orbeon Forms 2016.2.

Wednesday, June 29, 2016

New indexes boost performance with relational databases in Orbeon Forms 2016.2


When you create a form with Form Builder and use Orbeon Forms' implementation of the persistence API for relational databases, data collected by the form at runtime is stored in a set of 4 generic predefined tables. The tables are generic in the sense that they are not form-specific. This has a number of benefits:
  • It gives more control to DBAs: they can create the tables when Orbeon Forms is installed, and since at runtime Orbeon Forms doesn't need to have the permission to create tables, this allows DBAs to have a full knowledge of the database schema, and to optimize its storage or create additional indexes, should they wish to do so.
  • It keeps the database schema simple, and uniform across deployments. (Contrast this to an approach where you would have one table per form, and per version thereof.) Because of this, no "garbage collection" of unused tables (unused form versions) is necessary, upgrades to new versions of Orbeon Forms that require a change to the database schema are simpler, all the operations performed by Orbeon Forms can be part of a transaction (not all databases support DDL to be part of a transaction), and more.
For the tables to be generic:
  • Metadata (such as the form data id, the data owners, the time of the last change), which is the same for all forms, is stored in regular columns.
  • Values entered by users in your form are stored as XML, using a single "generic" xml column.
This works extremely well for operations where Orbeon Forms needs to either read or store all the values for a given instance of a form, like when users save or open form data. However, things gets more complicated when you want to do reporting on the submitted data, like:
  • For a given form, in a table, show the values of 5 significant fields for the last 50 submissions.
  • For a given form, find form data where the field last name has the value Smith.
Those operations are performed by summary pages, and the search API. In Orbeon Forms 2016.1 and earlier, they required SQL to extract data from the XML stored in the relational database. Unfortunately:
  • Some databases only have minimal support for XML.
  • Even for those databases with good support:
    • Searching and extracting values from XML data still isn't as fast as if the data was stored in "regular columns".
    • Optimizing those queries on XML data is highly database-dependant, and requires the creation of indexes or other techniques that DBAs are not necessarily familiar with.
Because of this, in Orbeon Forms 2016.1 and earlier, the performance of summary pages and the search API degraded as more data was added to the database. To solve this problem, Orbeon Forms 2016.2 introduces index tables. Data is still stored as XML as it was before, but the subset thereof needed for summary pages and the search API is also stored in the index tables. As a result, summary pages and the search API never need to access data stored in XML, and can thus run much faster.

This also means that the values in the index tables need to be kept up-to-date by Orbeon Forms. This happens automatically as data gets saved, or forms deployed. However, if you're upgrading to 2016.2, for the summary pages and the search API to work properly you'll need to first populate those indexes, in an operation referred to as reindexing. For more on this, see how to reindex your Orbeon Forms database when upgrading to 2016.2.

Thursday, June 23, 2016

Saying farewell to HTML tables

A very important underlying feature of Orbeon Forms is the grid. A grid groups controls in rows and columns, and can optionally repeat rows.

When we first implemented grids many years ago, we used tables for the underlying HTML markup. In HTML, tables serve two main purpose:
  1. Presenting tabular data (which is their original purpose).
  2. Laying out other content.
The reason that tables got to be used for laying out content is that they offered layout features which no other HTML and CSS construct supported. Over time, however, CSS has introduced most of the necessary support, in particular display: table and related, as well as the more recent Flexible Box Layout (also known as "flexbox").

Because assistive technology such as screen readers usually describe tables to the user as being tables when reading a page, there is widespread agreement that one should not use tables for purposes other than presenting tabular data so as to not confuse the user.

So in the upcoming Orbeon Forms 2016.2 we have changed the layout of grids to no longer use HTML tables when possible.

There are two cases where we still use HTML tables:
The case of row spans is hopefully a temporary limitation: it is just plain harder to emulate row spans with pure CSS, and we hope that this won't be a big issue because the use of row spans in Orbeon Forms is fairly rare. Further, we now place the ARIA presentation role on such tables to help screen readers.

For the case of repeated grids, using HTML tables makes some sense because the data is actually presented in tabular form with headings, rows and columns.

The specific HTML layout is documented here.

As an aside, we have also removed the use of HTML tables in the error summary component.

We hope you like these changes which will be available in Orbeon Forms 2016.2!

Thursday, May 19, 2016

JavaScript components made easier

Orbeon Forms's toolbox, which presents the choice of form controls available when you build a form, is fully customizable. Each form control points to a file containing information about the control and, in the case of custom controls (also known as components), the behavior of the control itself.

Some components do not require any custom JavaScript code: you implement all the logic with XForms. These components often combine other existing controls (such as a date components made of separate input fields or dropdown menus) and "glue" them together, visually as well as functionally (for example the value of multiple fields are put together to build an ISO date string).

On the other hand some components encapsulate functionality mainly implemented in JavaScript. And in such cases, XForms and JavaScript must communicate. For example, when a component becomes visible, some JavaScript initialization must run. Similarly, a user interaction might dispatch a JavaScript event which must result in a change of the control's value in the form.

Orbeon Forms does this with the help of a JavaScript class called the component's companion class. Each JavaScript-based component defines such a class, which handles the component's lifecycle and holds custom data and functions. One instance of this class is automatically created by Orbeon Forms for each instance of relevant (visible) control. We call these instances companion instances.

And the good news is that since Orbeon Forms 2016.1, a new javascript-lifecycle mode lets Orbeon Forms handle important aspects of the component's lifecycle. This makes it much easier to implement JavaScript-based components.

All the details are available in the documentation of JavaScript Companion Classes. We hope you'll like this new feature of Orbeon Forms 2016.1!

Thursday, May 5, 2016

Speaking JSON

JSON is the most popular format for browser/server communication. It is used more and more for all types of web services and as time goes it becomes more and more likely that you will encounter web services based on JSON rather than XML.

So Orbeon Forms had to support JSON, which is why we worked hard to ship Orbeon Forms 2016.1 with JSON support out of the box!

We considered two levels of JSON support:

  • First, within the forms engine.
  • Second, from Form Builder and Form Runner.

With Orbeon Forms 2016.1, JSON support is fully integrated at the XForms level, and submissions (service calls) can send and receive JSON.

Because natively XForms is XML-based, JSON goes through a transformation when it is received, following the proposed XForms 2.0 specification. This is designed so that you can access the resulting XML with path expressions in a way very similar to how you do it in JavaScript. Say you have this JSON:

{
   "father":{
      "given":"Mark",
      "family":"Smith"
   },
   "mother":{
      "given":"Mary",
      "family":"Smith"
   }
}

In JavaScript you use the expression mother.given to find “Mary”. From XForms, you use instead mother/given. It is pretty natural in many cases. For more details, see also submissions JSON support.

Similarly, XML goes through the reverse transformation when you need to send JSON to a service. The idea is that you can roundtrip JSON from a service.

With Orbeon Forms 2016.1, Form Builder and Form Runner only have limited JSON support, but receiving JSON from a service already works: if the service response contains application/json, it is automatically processed, and you can access the received data in actions via path expressions. For more deatils, see HTTP services JSON support.

In the future, we plan to make sending JSON to services from Form Builder / Form Runner possible and user-friendly, and we also want to allow sending form data in JSON format to services. Stay tuned!