Thursday, January 22, 2015

Saying goodbye to internal HTTP connections

Orbeon Forms often needs to perform calls to itself, for example to:

  • call persistence layer implementations,
  • load internationalized resources,
  • navigate between pages,
  • proxy requests for dynamic XForms resources,
  • and request CSS, JavaScript and image assets when producing PDFs.

In previous versions, Orbeon Forms performed actual HTTP connections on itself. This caused problems:

  • Configuration: Orbeon Forms needed a host and port to connect. In some cases, this could be guessed, but in others, for example when a front-end was placed in front of Orbeon Forms, you had to override a configuration property. There were also reported cases of interference with some servlet filters installed by our users.
  • Authentication and authorization: Internal services typically need the same credentials as the incoming request. To do this we were forwarding cookies and other headers, which could fail in some corner cases and was sometimes difficult to configure.
  • Thread usage: This required more servlet container threads than needed. In some cases, a external request to Orbeon could cause 3 threads to be used, 2 of them just waiting on HTTP connections.
  • Deadlocks: When the pool of threads provided by Tomcat was exhausted, deadlocks could occur, especially since we recommended a small pool in order to avoid overloading the forms processor. [1]
  • Performance: Page and service calls were slower than needed, as HTTP connections needed to be established, and communicating over HTTP has a certain cost.

In short, change was needed! So with Orbeon Forms 4.7, we got on this and entirely removed of the use of HTTP for internal calls!

How did we do this?

We were already using an abstraction to perform requests within Orbeon Forms. This abstraction unifies how we access internal resources stored in the Orbeon application itself, as well as calls such as HTTP calls.

So we built on this [2] to provide an additional HTTP abstraction. We now have an ApacheHttpClient, which uses HTTP proper, and a new InternalHttpClient, which behaves like the real HTTP client from the perspective of the caller but in fact does its work entirely internally, without any network access.

From the perspective of the page or service called, it is as if the request was coming from an external HTTP client (with some minor twists).

This in one shot addresses all the issues listed above. So we are quite happy with this change!

  1. Orbeon Forms 4.8 introduces a new limiter filter to deal with this in a much better way.  ↩

  2. Including some fairly massive refactoring along the way!  ↩

Wednesday, January 14, 2015

Choosing the best versioning option when publishing a form

Orbeon Forms supports versioning of form definitions since Orbeon Forms 4.5 [1].
Since Orbeon Forms 4.6 [2], Form Builder also allows you to control whether a new form definition version must be created:

Creating a new version
Creating a new version

Or whether the last existing form definition version must be overwritten:

Overwriting an existing version
Overwriting an existing version

You notice that there are warnings associated with each option! Ideally, we wouldn’t have a need for them, but the use of each option has different consequences on published forms.
So which option should you chose when you republish a new form definition?
  1. If there is no data associated with the form definition in the database, it is safe to overwrite the latest form definition, unless you really want to keep the older version published.
  2. If there is data associated with the form definition in the database and the changes you have made to the form definition
    1. are not structural, it is also safe to overwrite the latest form definition.
    2. are structural, it is not safe to overwrite and you should create a new version.
The following operations count as structural changes:
  • adding controls, grids, or sections
  • removing controls, grids, or sections
  • moving controls, grids, or sections
  • renaming controls, grids, or sections
The following operations are not structural:
  • changing the form’s title or description
  • changing a control’s label, hint, help, or alert message
  • changing a section’s title or help
  • adding or removing repeated grid or section iterations
  • adding or removing languages
  • changing the form’s permissions
  • adding, changing, or removing the form’s PDF template [3]
In general, the following non-structural operations are safe, but can have consequences on data validation and access:
  • adding or removing services and actions
  • adding, changing, or removing validations or formulas
  • adding, changing, or removing an XML Schema
For example if validation rules change, the situation can arise where data stored in the database was valid at the time it was saved, but the new form definition considers them invalid. So viewing the data will work, but attempting to save without changes will not satisfy the new validation rules and will fail. It is important to be aware of this when making validation rules changes.
Similarly, changing actions or visibility rules can make the form behave differently when editing or viewing existing data. Here again, you must be cautious.
Now you might ask, what is the drawback of always creating a new form version when publishing?
Existing data in the database is associated with a specific form definition version. This means that existing data cannot be viewed or edited with a different form definition version. So a fix for a typo in a control label, for example, wouldn’t show when editing or viewing existing data. This may or may not be acceptable depending on your use case.
As discussed in the original post about versioning, we hope to enable more scenarios with basic data migration in the future.

  1. As of Orbeon Forms 4.8, versioning is supported with relational databases, but not with eXist. See also database support.  ↩
  2. That’s a couple of versions ago, since as we write Orbeon Forms 4.8 has just been released, but we didn’t get to talk about this feature until now!  ↩
  3. Note that changing the PDF template can cause incorrect PDF production for existing data if the PDF template is incorrectly setup.  ↩

Thursday, January 8, 2015

Orbeon Forms 4.8

Today we released Orbeon Forms 4.8!

Major features and enhancements
  • PostgreSQL support. Thanks in good part to an external contribution, Orbeon Forms supports PostgreSQL in addition to Oracle, MySQL, SQLServer, DB2 and eXist. (blog, #1941)
  • Better management of server resources. Under load, Orbeon Forms limits the number of concurrent form processing requests in order to reduce the likelihood of the server running out of memory (doc, #1971).
  • Support for repeated grid visibility. It is possible to completely hide (including the header row) or mark as readonly a repeated grid (doc#597 / #635).
  • Owner and group-based permissions eXist support. Owner and group-based permissions were only supported with relational databases. This release adds eXist support. See also Database Support. (doc, #1341)
  • Singleton forms. A singleton form is a form with only a single instance of form data visible to a user. Form Builder allows you to mark a form as such, and Form Runner then uses this information to prevent the creation of more than one instance of form data. (doc, #2009)
  • Improved XForms 2.0 support. 
    • The caseref attribute on xf:case is implemented (#1089).
    • The xf:case() function is implemented (#1951). It was already available as an extension function with xxf:case().

See Localizing Orbeon Forms for the latest localization support. Localization depends on volunteers, so please let us know if you want to help!

Other new features and bug-fixes

Including the major features and enhancements above, we closed over 30 issues since Orbeon Forms 4.7.1 (and over 60 since 4.7).
Current browser support
  • Form Builder (creating forms):
    • Chrome 39 (latest stable version) and Chrome 41 dev (current dev channel)
    • Firefox 34 (latest  stable version) and the current Firefox ESR
    • IE 11
    • Safari 8
    • Form Runner (accessing form):
      • All browsers supported by Form Builder (see above)
      • IE8, IE9 and IE10
      • Safari Mobile on iOS 6, iOS 7 and iOS 8
      • Chrome for Android (stable channel)
    Compatibility notes
    • The internal data format for repeated grids has changed following #635. This shouldn't affect users and form author much:
      • Form definitions are upgraded automatically when loaded in Form Builder, but published forms are not affected until republished, whether from Form Builder or from the Home page.
      • The change is visible when looking at the form definition source code in Form Builder.
      • The external data format, that is the data saved to the persistence layer, is unchanged, therefore existing form data in the database doesn't need migration.
      • The send action also uses the "old" format by default (this can be overridden). (doc)
      • POSTing initial data to a form also uses the "old" format by default (this can be overridden). (doc)
      • However initial data obtained from a service can only be in the "old" format at this time.
    • Relevance and readonly for repeated grids
      • The "Visibility" and "Readonly" formulas in Form Builder  apply to the entire grid, whereas they used to apply to individual iterations only (although this was not an intended feature). In the future we might add separate "Visibility" and "Readonly" formulas for iterations.
    • fr:grid component (doc)
      • The repeat="true" attribute value is deprecated. Instead use repeat="content" which expect different data layout.
      • The relevant and readonly MIPs apply to the entire grid with repeat="content".
      • The evaluation context for the min and max AVTs is properly specified and documented.
    • The xxf:tree and xxf:menu appearances on xf:select and xf:select1 have been removed. The way these appearances were implemented is considered obsolete and not showcased by Form Runner and Form Builder anyway.
    • The default session duration used to be 12 hours. It is reduced to 1 hour. The session heartbeat feature helps keeping the session alive.
    You can download the latest version of Orbeon Forms from the downloads page.
      Don't forget to grab a trial license for the PE version.

      Please send feedback:
      We hope you enjoy this release!