Wednesday, April 23, 2014

Smarter dates and times

Orbeon Forms prior to version 4.4 used to display full dates and times in the Summary page:

Full dates
Full dates

We didn’t like this because the resulting dates and times took a lot of precious space on screen.

One way to solve this is to indicate relative dates such as “3 hours ago”, “20 days ago”, or “1 year ago”. [1] But we don’t like this either because the resulting information is often rounded to the point of being meaningless.

Instead, we chose a very simple method inspired by applications such as Google Drive. [2] We use exactly 3 date and time formats:

  • if the instant is at most one day in the past, show hours and minutes only
  • if the year is the same as the current year, show the month and day only
  • otherwise, show year, month and day in a compact format

The result is as follows:

Short dates
Short dates

These shorter dates present enough meaningful information and always in a compact way. Of course this remains a tradeoff: for example if you want to save a screenshot for further reference, it would better to have full dates and times. But for most common uses, we think this is a pretty good approach. Let us know what you think!

This change is available in Orbeon Forms 4.4 and newer.


  1. As GitHub does, for example.  ↩

  2. Formerly Google Docs.  ↩

Friday, April 18, 2014

Solving the observer problem

Abstract

As most sophisticated web apps, Orbeon Forms has to deal with lots of asynchronous events in the browser, like Ajax requests, timers, and user interactions. We'll go over a real-life example of logic involving several asynchronous events, see the difficulty involved in implementing it with observers, and explore 3 alternatives: state machines, await, and Functional Reactive Programming (FRP). We'll see why, based on our experience, we don't consider state machines to be a good solution, and through an example explore the two remaining alternatives, await and FRP.

Example of the loading indicator

Orbeon Forms can show on the page a "loading indicator" to inform users that something is happening: data was sent to the server, and we're waiting for a response, maybe because we need to validate user input, save data, or call a service. Its logic works is as follows:

  1. In essence, when a request is in progress, we want the loading indicator to show.
  2. However, if the request takes very little time, we don't want to show the loading indicator.
  3. Also, when a request comes back, we don't want to hide the indicator if we already know we'll right away send another request to the server.

Stated in plain English, this is quite simple.

The problem with observers

The browser gives us events, and we can run code when those events happen. Code running upon some event is called an observer. For the loading indicator, we would have an observer when an Ajax request starts, and another one when it ends. Since we don't want to show the indicator right away, the observer on the "Ajax request started" starts a timer for say 200 ms. So we would have an observer on that timer, which on completion shows the indicator if the request is still in progress. But to know that, we need to have state (isRequestInProgress), shared between the two observers.

But a boolean isn't enough. Say you start a first request at t = 0 ms. At that point, you also start a timer for 200 ms. Say that first request ends at t = 100 ms. Now say a second request starts at t = 150 ms. At t = 200 ms, the first timer ends: a request is still in progress, so it shows the loading indicator. Bug! At that point the request has been going on for only 50 ms, so we shouldn't show the loading indicator just yet. One way to solve this problem is to use a counter, instead of a boolean, an increment/decrement that counter at the right time.

I won't go here through all the details of the implementation, but the end result is that the above logic is spread across a number of observers that share and update some global state. Logic written this way is surprisingly hard to maintain, and subtle bugs can lead to a loading indicator that stays "stuck", i.e. shown even if there no Ajax request in progress, or that doesn't show anymore.

Possible solutions

We're aware of 3 alternatives to observers:

  1. State machines
  2. await
  3. Functional reactive programming (FRP)

State machines

In Form Builder, we started using a state machine about 2 years ago, and had a mixed experience with that approach. A state machine solves part of the observer problem, by putting the high level logic in a single place, in a fairly declarative form. However, we found that complex state machines are hard to debug, as you can't debug the declaration of the state machine itself, but instead have to debug the generic implementation of the state machine.

Experimenting with await and FRP

Before going for await or FRP in production code, we wanted to do some experimentation. For this, we took the example described in the excellent paper Deprecating the Observer Pattern. The example uses the canvas and the mousedown, mousemove, and mouseup events to let the user draw on the canvas. You'll find below the CoffeeScript code for 3 implementations:

  1. With observers, using plain jQuery.
  2. With await, using IcedCoffeeScript.
  3. With FRP, using Bacon.js.

For each case, you can click on the link after the code to view the full source and run that example.

With observers

isDown = false

canvas.on 'mousedown', (e) -> 
    isDown = true
    startLine(e)

canvas.on 'mousemove', (e) -> 
    if isDown
        continueLine(e)

canvas.on 'mouseup', (e) -> 
    isDown = false

See and run the code

With await

while true

  await canvas.one('mousedown', defer(e))
  startLine(e)

  until e.type == 'mouseup'
      rv = new iced.Rendezvous
      canvas.one('mousemove', rv.id().defer(e))
      canvas.one('mouseup'  , rv.id().defer(e))
      await rv.wait(defer())
      continueLine(e)

See and run the code

With FRP

downStream         = canvas.asEventStream('mousedown')
upStream           = canvas.asEventStream('mouseup')
moveStream         = canvas.asEventStream('mousemove')

isDownProp         = downStream.map(true)
                               .merge(upStream.map(false))
                               .toProperty(false)
moveWhenDownStream = moveStream.filter(isDownProp)

downStream         .onValue(startLine)
moveWhenDownStream .onValue(continueLine)

See and run the code

Conclusion

We believe that both await and FRP are significant improvement over observers or a state machine. At this point, we have a slight preference for await, as we think that:

  • Most developers will find it easier to understand code using await than code using FRP, and we believe in Erik Meijer's mantra: "write baby code".
  • We think that code using await will be easier to debug than code using FRP, as it will be easier to put breakpoints in the right place, and follow the code execution.

Monday, April 7, 2014

Orbeon Forms 4.5

Today we released Orbeon Forms 4.5!

Major features

This release is rich in new features and enhancements:
  • IE11 support. Both Form Builder and deployed forms now work with with IE11.
  • Versioning of form definitions. Publishing a form definition doesn't overwrite the previously published form definition, but creates a new version of it. Orbeon Forms associates the version of the form definition used for creating and editing data. This is supported with relational databases (Oracle, MySQL, and DB2 at this time). (blog post)
  • Improved fields, checkboxes and radio buttons in preview and PDF. Automatic PDF output now shows all radio buttons and checkboxes, nicely highlights fields, supports optional page breaks before sections, and shows the form's title in the page's header and footer. (blog post)
  • Inserting and reordering of grid rows. Repeated grids now have the ability to reorder grid rows and to specify exactly where to insert a new row. (blog post)
  • Repeated sections. Form Builder and Form Runner now support repeated section content. This is very similar to repeating a group of grid rows, except that the entire content of a section can be repeated, including nested grids and subsections. (blog post)
  • Improved help messages appearance and positioning. Help messages have a fresher, more modern look, and are now more lightweight and better positioned. (blog post)
  • Hints for checkboxes and radio buttons. You can now associate hint messages with specific radio button or checkboxes. They appear as tooltips upon hover. (blog post)
  • Form definitions and form data duplication. The summary pages for both Form Builder and your own forms now can have a Duplicate button which allows you to duplicate form definitions (in Form Builder) and form data, including attachments. (blog post)
  • Improved dropdown menus. Dropdown menus are now a bit smarter! (blog post)
  • Improved Form Builder Choices editor. The Form Builder Choices editor allows you to reorder items and to directly localize labels and hints. (blog post)
  • Improved Form Builder variable resolution. We have fixed a long-standing issue which prevented the use of variables in repeated grids. (doc)
  • Detecting login pages in Ajax requests. Administrators can now tell Orbeon Forms which pages are login pages, to help with error recovery. (blog post)
Internationalization:
  • This version features full Spanish localization, courtesy of Bruno Buzzi of AGESIC.
  • This version also features Swedish localization for Form Runner.
See Localizing Orbeon Forms for more details about supported languages.
Other bug-fixes and features

Including the major features and enhancements above, we closed over 110 issues since Orbeon Forms 4.4. We should mention these notable bug-fixes and features:
  • XForms 2.0 bind() function (#1423)
  • Service request or response cannot target control in subsection (#1465)
  • Support repeats in actions (#1105)
  • PDF: Option to output a page break before a section (#1556)
  • FB: UI to edit control, grid and section classes (#1555)
  • PDF: Ability to set form title in header and/or footer (#1558)
  • Processes: reusable confirmation dialog (#1570)
  • XML Schema xs:import doesn't support relative paths (#1340)
Current browser support
  • Form Builder (creating forms):
    • Chrome (latest versions, both in the stable and dev channels)
    • Firefox (latest released version and current Firefox ESR)
    • IE10 and IE11
    • Additional support for Form Runner (accessing form):
      • IE7 (deprecated), IE8, and IE9
      • Safari Mobile on iOS 6 and iOS 7
      • Chrome for Android (stable channel)
    Compatibility notes
    • Date picker. The Date Picker control has been removed from Form Builder. It was similar to the normal Date control, except the date was shown as text in the page, not in a text field. The Date Picker didn't have much of a benefit over the regular Date control, but had a big drawback: it wasn't usable on mobile devices, which expect a text field to enable the native date picker. Note that while the control isn't available in the Form Builder sidebar anymore, the underlying implementation is still there, so your existing forms that use a Date Picker can still be edited in Form Builder, and will still function properly. (#1364)
    • Datatable. The fr:datatable component, deprecated in 4.0, has been completely removed.
    • Deployment. Integrated deployment, consisting in combining your Java/JSP applications with Orbeon Forms within the same WAR, is now deprecated. We recommend using separate deployment instead.
    • Oracle flat view. The truncation of section/control column names has changed. If the section or control name is 14 characters or less, it is kept as is, and the other part is truncated if needed. A numerical suffix is used instead for those columns which would introduce duplicates. (doc)
    You can download the latest version of Orbeon Forms from the downloads page.
      Don't forget to grab a trial license for the PE version.

      Please send feedback:
      We hope you enjoy this release!

      Friday, April 4, 2014

      Configure the URL of your services in properties

      Rationale

      A few weeks ago, we've seen how you can store Orbeon Forms configurations, in particular the properties-local.xml, outside of the Orbeon Forms war file. We've seen how this allows you to have a single war you deploy on all your environments, and keep differences in configurations between these environments outside of the war file.

      This is all good for the Orbeon Forms configurations. Here we'll see that you can use the same mechanism to define custom properties to specify which on what server your services run, or to what server you want data to be posted on submission.

      The xxf:property() XPath function

      Say you have forms you created with Form Builder, those forms use services, and in the service URL you want a different host name depending on the environment you're in. But of course, you don't want to change the forms as you run them on different environments. You can do this by defining a property, say:

      <property as="xs:string"  
                name="com.example.service.host"
                value="prod.example.com"/>
      

      Then, in Form Builder, in the HTTP Service Editor dialog, you refer to the value of the property with:

      http://{xxf:property(com.example.service.host)}/your/service
      

      Submission server

      When users submit a form, you can setup Orbeon Forms to POST the data captured by the form to one of your servers. You do this with the send action, e.g.:

      send(uri = "http://prod.example.com/your/service")
      

      However, if you'd like, you can also move the server URL to its own property:

      <property as="xs:anyURI"
                name="com.example.send.uri"
                value="http://prod.example.com/your/service"/>
      

      And refer to that property in the send action (note that there is .uri at the end of the value passed to send, as the parameter to send is only a prefix, to which send can add different suffixes; for more on this, see the send action):

      send(property = "com.example.send")
      

      Thursday, March 27, 2014

      Review and PDF improvements

      With Orbeon Forms 4.5, we are introducing several improvements to the Review mode and the PDF output.

      Selection controls

      We used to output all selection controls like this:

      • Single selection: output the selected item’s label.
      • Multiple selection: output all selected items’ labels with separators.
      • Boolean input: output “true” or “false”.

      But this was not ideal for checkboxes and radio buttons. In these cases, you probably want all options visible in your review or PDF as well. Also, this opens the door to amending printed forms by checking options by hand.

      The boolean input was worse, because nobody really wants to see “true” or “false” printed!

      So here is how things look like in 4.5:

      Review mode
      Review mode
      PDF output
      PDF output

      You notice that both checkboxes and radio buttons appear as boxes. That’s because on paper forms, unlike web UIs, there is more often than not no distinction between the two: you just check the boxes that apply, and explanatory text clarifies whether multiple choices are allowed. So for now we have decided on a unified appearance for both.

      Highlighting of fields

      We now also better highlight fields, for two reasons:

      • It was not always clear where a control’s label stopped and a control’s value started.
      • There is a use case for blank fields to be filled later on paper, in which case the field’s area must be clearly indicated.

      This applies not only to input fields, but also to other types of controls which take more of a “field” appearance in review mode and PDF, such as date pickers and dropdown menus.

      Highlighting of control fields
      Highlighting of control fields

      Page breaks before sections

      Form Builder now has a built-in option to add a PDF page break before a section. One use case is forms that have a signature section, where there are legal requirements to make sure that certain fields stay together and on the same page.

      Page Break Before option
      Page Break Before option

      Other improvements

      Along the way, we also made the following improvements:

      • The “[Select…]” dropdown title no longer appears in Review mode and PDF files (see Fun with dropdown menu titles).
      • We place the form title in the header and the footer of the PDF, alongside page numbering.

      With all this, the Review mode and PDF output are looking pretty good! These enhancements will be available in Orbeon Forms 4.5.

      Monday, March 17, 2014

      Storing configurations outside of the Orbeon Forms war file

      Rationale

      If you're reading this, you're most likely already familiar with the properties-local.xml file used to configure Orbeon Forms. This file typically goes inside the Orbeon Forms web app, in WEB-INF/resources/config. What if you'd like to package a single war for Orbeon Forms, and deploy that war on multiple servers on which you need to have slightly different configurations? Is it possible to store this configuration outside of the war file? It is, and we'll see how you can do this in the remainder of this post.

      Resource managers

      The directory WEB-INF/resources, as its name implies, contains files called resources. Resources can be configuration files, XSLT stylesheets, or CSS/JavaScript assets served to the browser. Resources can be stored in a variety of locations, hence Orbeon Forms loading them through a resource manager. Several resource managers come out-of-the-box with Orbeon Forms, and you could even implement your own. The 3 most frequently used resource managers are:

      • The class loader resource manager loads files from the class path. Most of the built-in resources are stored in jar files that ship with Orbeon Forms, and are thus loaded with the ClassLoader resource manager.
      • The web app resource manager typically loads files from the web app's WEB-INF/resources.
      • The file system resource manager loads files from a directory on disk you specify.

      Changing the web.xml

      By default, Orbeon Forms first attempts to load resources with the class loader resource manager, and if that fails tries with the web app resource manager. But you can change this by adding context params in the Orbeon Forms web.xml. For instance, with the following two context params added, Orbeon Forms will first try to load resources from /etc/orbeon:

      <context-param>
        <param-name>oxf.resources.priority.0</param-name>
        <param-value>
          org.orbeon.oxf.resources.FilesystemResourceManagerFactory
        </param-value>
      </context-param>
      <context-param>
        <param-name>
          oxf.resources.priority.0.oxf.resources.filesystem.sandbox-directory
        </param-name>
        <param-value>/etc/orbeon</param-value>
      </context-param>
      

      With this single change to the Orbeon Forms war, and the choice of a "standard" directory in which you'll place your configuration files, say /etc/orbeon, you can deploy the same war on multiple servers, and have different configurations in the /etc/orbeon of each server.

      Changing Tomcat's configuration

      Tomcat allows you to set the value of context parameters for an app in its own configuration, so you don't even have to change the web.xml. The following two lines added inside your Tomcat's <Context> element for the Orbeon Forms web app (e.g. in your server.xml) are equivalent to the change we did earlier to the web.xml.

      <Parameter
       override="false"
       name="oxf.resources.priority.0"
       value="org.orbeon.oxf.resources.FilesystemResourceManagerFactory"/>
      <Parameter
       override="false" 
       name="oxf.resources.priority.0.oxf.resources.filesystem.sandbox-directory"
       value="/etc/orbeon"/>
      

      Wednesday, March 12, 2014

      Choices editor improvements

      Orbeon Forms 4.5 introduces two features related to repeated content, discussed in other posts:

      Work on these enhancements also benefits the Form Builder Choices editor [1], and you can now reorder choices and have control over where to insert new ones:

      Reordering choices
      Reordering choices

      In addition, you can now easily switch between the form’s languages directly in the Choices editor, instead of having to close it, switch languages, and come back. This makes localization easier! [2]

      These enhancements will be available in Orbeon Forms 4.5.


      1. Which allows you to edit the choices for dropdown menus, radio buttons, checkboxes, and other selection controls.  ↩

      2. This feature was available in Orbeon Forms 3.9 but had gotten lost on the way to 4.0!  ↩