Monday, February 2, 2015

Orbeon Forms 4.8.1

Today we released Orbeon Forms 4.8.1 PE. This update to Orbeon Forms 4.8 PE contains bug-fixes and is recommended for all Orbeon Forms 4.8 PE users.

Specifically, this release addresses the following issues:
  • Form Builder
    • Incorrect label edited upon insert of XBL with nested labels (#2075)
  • Form Runner
    • Error when producing PDF in parallel with Ajax request (#2071)
    • Crash when paging on Summary page (6af5bed6c5)
    • fr:number: incorrect rounding when using decimal separator other than `.` (#2076)
    • Custom mips are not propagated to section templates (#2065)
  • Other
    • Ajax retry doesn't work with Firefox (#2074)
    • Object cache statistics are not used but can be costly (#2072)
    • NPE using xxf:evaluate-bind-property() (#2048)
You can download the latest version of Orbeon Forms from the downloads page.
    Don't forget to grab a trial license for the PE version.

    Please send feedback:
    We hope you enjoy this release!

    Thursday, January 22, 2015

    Saying goodbye to internal HTTP connections

    Orbeon Forms often needs to perform calls to itself, for example to:

    • call persistence layer implementations,
    • load internationalized resources,
    • navigate between pages,
    • proxy requests for dynamic XForms resources,
    • and request CSS, JavaScript and image assets when producing PDFs.

    In previous versions, Orbeon Forms performed actual HTTP connections on itself. This caused problems:

    • Configuration: Orbeon Forms needed a host and port to connect. In some cases, this could be guessed, but in others, for example when a front-end was placed in front of Orbeon Forms, you had to override a configuration property. There were also reported cases of interference with some servlet filters installed by our users.
    • Authentication and authorization: Internal services typically need the same credentials as the incoming request. To do this we were forwarding cookies and other headers, which could fail in some corner cases and was sometimes difficult to configure.
    • Thread usage: This required more servlet container threads than needed. In some cases, a external request to Orbeon could cause 3 threads to be used, 2 of them just waiting on HTTP connections.
    • Deadlocks: When the pool of threads provided by Tomcat was exhausted, deadlocks could occur, especially since we recommended a small pool in order to avoid overloading the forms processor. [1]
    • Performance: Page and service calls were slower than needed, as HTTP connections needed to be established, and communicating over HTTP has a certain cost.

    In short, change was needed! So with Orbeon Forms 4.7, we got on this and entirely removed of the use of HTTP for internal calls!

    How did we do this?

    We were already using an abstraction to perform requests within Orbeon Forms. This abstraction unifies how we access internal resources stored in the Orbeon application itself, as well as calls such as HTTP calls.

    So we built on this [2] to provide an additional HTTP abstraction. We now have an ApacheHttpClient, which uses HTTP proper, and a new InternalHttpClient, which behaves like the real HTTP client from the perspective of the caller but in fact does its work entirely internally, without any network access.

    From the perspective of the page or service called, it is as if the request was coming from an external HTTP client (with some minor twists).

    This in one shot addresses all the issues listed above. So we are quite happy with this change!


    1. Orbeon Forms 4.8 introduces a new limiter filter to deal with this in a much better way.  ↩

    2. Including some fairly massive refactoring along the way!  ↩

    Wednesday, January 14, 2015

    Choosing the best versioning option when publishing a form

    Orbeon Forms supports versioning of form definitions since Orbeon Forms 4.5 [1].

    Since Orbeon Forms 4.6 [2], Form Builder also allows you to control whether a new form definition version must be created:

    Creating a new version
    Creating a new version

    Or whether the last existing form definition version must be overwritten:

    Overwriting an existing version
    Overwriting an existing version

    You notice that there are warnings associated with each option! Ideally, we wouldn’t have a need for them, but the use of each option has different consequences on published forms.

    So which option should you chose when you republish a new form definition?

    1. If there is no data associated with the form definition in the database, it is safe to overwrite the latest form definition, unless you really want to keep the older version published.
    2. If there is data associated with the form definition in the database and the changes you have made to the form definition
      1. are not structural, it is also safe to overwrite the latest form definition.
      2. are structural, it is not safe to overwrite and you should create a new version.

    The following operations count as structural changes:

    • adding controls, grids, or sections
    • removing controls, grids, or sections
    • moving controls, grids, or sections
    • renaming controls, grids, or sections

    The following operations are not structural:

    • changing the form’s title or description
    • changing a control’s label, hint, help, or alert message
    • changing a section’s title or help
    • adding or removing repeated grid or section iterations
    • adding or removing languages
    • changing the form’s permissions
    • adding, changing, or removing the form’s PDF template [3]

    In general, the following non-structural operations are safe, but can have consequences on data validation and access:

    • adding or removing services and actions
    • adding, changing, or removing validations or formulas
    • adding, changing, or removing an XML Schema

    For example if validation rules change, the situation can arise where data stored in the database was valid at the time it was saved, but the new form definition considers them invalid. So viewing the data will work, but attempting to save without changes will not satisfy the new validation rules and will fail. It is important to be aware of this when making validation rules changes.

    Similarly, changing actions or visibility rules can make the form behave differently when editing or viewing existing data. Here again, you must be cautious.

    Now you might ask, what is the drawback of always creating a new form version when publishing?

    Existing data in the database is associated with a specific form definition version. This means that existing data cannot be viewed or edited with a different form definition version. So a fix for a typo in a control label, for example, wouldn’t show when editing or viewing existing data. This may or may not be acceptable depending on your use case.

    As discussed in the original post about versioning, we hope to enable more scenarios with basic data migration in the future.


    1. As of Orbeon Forms 4.8, versioning is supported with relational databases, but not with eXist. See also database support.  ↩

    2. That’s a couple of versions ago, since as we write Orbeon Forms 4.8 has just been released, but we didn’t get to talk about this feature until now!  ↩

    3. Note that changing the PDF template can cause incorrect PDF production for existing data if the PDF template is incorrectly setup.  ↩

    Thursday, January 8, 2015

    Orbeon Forms 4.8

    Today we released Orbeon Forms 4.8!

    Major features and enhancements
    • PostgreSQL support. Thanks in good part to an external contribution, Orbeon Forms supports PostgreSQL in addition to Oracle, MySQL, SQLServer, DB2 and eXist. (blog, #1941)
    • Better management of server resources. Under load, Orbeon Forms limits the number of concurrent form processing requests in order to reduce the likelihood of the server running out of memory (doc, #1971).
    • Support for repeated grid visibility. It is possible to completely hide (including the header row) or mark as readonly a repeated grid (doc#597 / #635).
    • Owner and group-based permissions eXist support. Owner and group-based permissions were only supported with relational databases. This release adds eXist support. See also Database Support. (doc, #1341)
    • Singleton forms. A singleton form is a form with only a single instance of form data visible to a user. Form Builder allows you to mark a form as such, and Form Runner then uses this information to prevent the creation of more than one instance of form data. (doc, #2009)
    • Improved XForms 2.0 support. 
      • The caseref attribute on xf:case is implemented (#1089).
      • The xf:case() function is implemented (#1951). It was already available as an extension function with xxf:case().
    Internationalization

    See Localizing Orbeon Forms for the latest localization support. Localization depends on volunteers, so please let us know if you want to help!

    Other new features and bug-fixes

    Including the major features and enhancements above, we closed over 30 issues since Orbeon Forms 4.7.1 (and over 60 since 4.7).
    Current browser support
    • Form Builder (creating forms):
      • Chrome 39 (latest stable version) and Chrome 41 dev (current dev channel)
      • Firefox 34 (latest  stable version) and the current Firefox ESR
      • IE 11
      • Safari 8
      • Form Runner (accessing form):
        • All browsers supported by Form Builder (see above)
        • IE8, IE9 and IE10
        • Safari Mobile on iOS 6, iOS 7 and iOS 8
        • Chrome for Android (stable channel)
      Compatibility notes
      • The internal data format for repeated grids has changed following #635. This shouldn't affect users and form author much:
        • Form definitions are upgraded automatically when loaded in Form Builder, but published forms are not affected until republished, whether from Form Builder or from the Home page.
        • The change is visible when looking at the form definition source code in Form Builder.
        • The external data format, that is the data saved to the persistence layer, is unchanged, therefore existing form data in the database doesn't need migration.
        • The send action also uses the "old" format by default (this can be overridden). (doc)
        • POSTing initial data to a form also uses the "old" format by default (this can be overridden). (doc)
        • However initial data obtained from a service can only be in the "old" format at this time.
      • Relevance and readonly for repeated grids
        • The "Visibility" and "Readonly" formulas in Form Builder  apply to the entire grid, whereas they used to apply to individual iterations only (although this was not an intended feature). In the future we might add separate "Visibility" and "Readonly" formulas for iterations.
      • fr:grid component (doc)
        • The repeat="true" attribute value is deprecated. Instead use repeat="content" which expect different data layout.
        • The relevant and readonly MIPs apply to the entire grid with repeat="content".
        • The evaluation context for the min and max AVTs is properly specified and documented.
      • The xxf:tree and xxf:menu appearances on xf:select and xf:select1 have been removed. The way these appearances were implemented is considered obsolete and not showcased by Form Runner and Form Builder anyway.
      • The default session duration used to be 12 hours. It is reduced to 1 hour. The session heartbeat feature helps keeping the session alive.
      You can download the latest version of Orbeon Forms from the downloads page.
        Don't forget to grab a trial license for the PE version.

        Please send feedback:
        We hope you enjoy this release!

        Monday, December 1, 2014

        Orbeon Forms 4.7.1

        Today we released Orbeon Forms 4.7.1 PE. This update to Orbeon Forms 4.7 PE contains bug-fixes, security fixes, performance improvements, and localization improvements. We recommend this update for all Orbeon Forms 4.7 PE users.

        Specifically, this release addresses the following issues:
        • Form Builder
          • Renaming an action duplicates it (#1913)
          • FB user sees all forms in summary page if role is not found (#1963)
          • Test mode: Validate button shows instead of inner buttons (#2002)
          • Publish: document parameter value is not encoded (#1985)
          • Exception when calling dataMaybeMigratedTo() (#1986)
          • FB: Keyboard focus is not on first dropdown when present upon new (#2010)
        • Form Runner
          • Automatic PDF performance improvements
            • Requests for resources must be internal (#1995)
            • Don't store XForms document in cache (#1999)
            • Automatic PDF output should cache images (#1996)
          • Other automatic PDF improvements
            • Grid rows without visible controls must not take space (#2004)
            • Page number/count loses alignment if form title is too long (#2005)
            • Don't let long title wrap in header or footer (#2006)
            • Show form logo on PDF header (#2007)
            • Textarea content must wrap (#2008)
          • PDF templates improvements
            • Automatically decide whether to use label or value (#736)
            • Non-boolean checkboxes don't work (#1987)
            • fr:number is always blank (#1955)
            • fr:xforms-inspector causes PDF template error (#1949)
          • XVT not evaluated for `uri` in `send` action (#1972)
          • xxf:itemset() not working anymore now that we use fr:dropdown-select1 (#1856)
          • Add a few missing Swedish resources
        • XForms
          • Incorrect check on headers in Connection (#1933)
          • Error dialog not working (#1938)
          • Don't log full stack trace of session expiration messages (#1979)
          • Serializer not to filter elements in no namespace (#1981)
          • Default value for Dynamic Data Dropdown is lost when pasting form source (#1962)
        You can download the latest version of Orbeon Forms from the downloads page.
          Don't forget to grab a trial license for the PE version.

          Please send feedback:
          We hope you enjoy this release!

          PostgreSQL support in Orbeon Forms

          According to the DB-Engines database ranking by popularity, the top 5 relational databases are: Oracle, MySQL, SQL Server, PostgreSQL, and DB2. In 4.6 we added support for SQL Server. The upcoming 4.8 adds support for PostgreSQL, thus completing Orbeon Forms support for the top-5 relational databases.

          Past the top-5, we find Access and SQLite, which are typically not used in the type of server-side environments our customers have in place. The next candidate would be Sybase, but we're then getting pretty low on the popularity curve, and we've yet to encounter a customer who wouldn't be able to use Orbeon Forms without Sybase support. So it is with a certain sense of accomplishment that we're adding the support for PostgreSQL, thus allowing Orbeon Forms to reasonably claim it "supports, out-of-the-box, all the top relational databases".

          And for this, we have to thank Aaron Spike, from the Martin Luther College, who implemented and contributed the implementation for PostgreSQL. Thank you, Aaron!

          Friday, September 26, 2014

          Orbeon Forms 4.7

          Today we released Orbeon Forms 4.7!

          Major features and enhancements
          • Server-side embedding API. Orbeon Forms 4.7 introduces a new way to integrate forms within Java (and other Java Virtual Machine-based languages) applications: server-side embedding. (doc, blog)
          • Internal service requests. Orbeon Forms often needs to perform requests to itself, such as calling its own persistence layer implementations, as well as for internal page navigation. In previous versions, Orbeon Forms performed HTTP requests on itself. This could cause problems, such as using more servlet container threads, slower performance, and interference with servlet filters. Internal requests no longer use actual HTTP connections, which addresses these issues.
          • Improved Liferay proxy portlet support.
            • The proxy portlet uses a better and more configurable HTTP client to connect to Form Runner. In particular, this means that gzip compression can be supported between Form Runner and the portlet.  (doc)
            • The WAR file is much smaller than before (1.7 MB).
            • The portlet supports dynamic form selection from URL parameters. Note that this is disabled by default for security reasons. (doc)
          • Form Builder improvements.
            • Itemsets in actions support internationalization. (docblog)
            • You can control required values with formulas. (docblog)
          • Multiple remote servers. The Form Runner Home page supports multiple remote servers. You can, for example, configure a staging and a production server. (docblog)
          • Flat view support with DB2. Previously, the flat view was supported with Oracle only. It is now also supported with DB2. (doc)
          • Improvements to simple processes.
            • New set-data-status action. (doc)
            • New setvalue action. (doc)
            • Support XVT in success-message and error-message actions. (doc)
            • Ability to set and clear fields with the duplicate action. (doc)
            • Ability to send form metadata to a service. (doc)
          • New APIs.
            • Service API to duplicate form data. (doc)
            • Service API to list form data attachments. (doc)
          Internationalization

          See Localizing Orbeon Forms for the latest localization support. Localization depends on volunteers, so please let us know if you want to help!

          Other new features and bug-fixes

          Including the major features and enhancements above, we closed about 40 issues since Orbeon Forms 4.6.2 (and over 60 since 4.6).
          Current browser support
          • Form Builder (creating forms):
            • Chrome 38 (latest stable version, plus the current dev channel)
            • Firefox 32 (latest  stable version, plus the current Firefox ESR)
            • IE 11
            • Safari 8
            • Additional support for Form Runner (accessing form):
              • IE 8, IE 9 and IE 10
              • Safari Mobile on iOS 6, iOS 7 and iOS 8
              • Chrome for Android (stable channel)
            Compatibility notes
            • Browser support
              • For end-users (when accessing forms, e.g. through Form Runner) – Support for IE7 was deprecated in Orbeon Forms 4.5, and isn't supported anymore started with Orbeon Forms 4.7.
              • For form authors (using Form Builder) – In Orbeon Forms 4.7, we used to support both IE10 and IE11; since in most cases IE10 auto-updates to IE11, the share of IE10 dropped very quickly after IE11 was released in October 2013. So, in Orbeon Forms 4.7, we decided to drop support for IE10. Going forward, for Form Builder, we plan to only support the latest release of IE.
            • Portlets. Support for loading portlet content asynchronously has been removed. This had been introduced to improve portlet loads with slower browsers such as IE 7, which is no longer supported.
            • Property deprecation. The oxf.fr.production-server-uri property is deprecated and replaced with the new oxf.fr.home.remote-servers property.  If oxf.fr.production-server-uri is set and not empty, it takes precedence over the new oxf.fr.home.remote-servers property, for backward compatibility.
            You can download the latest version of Orbeon Forms from the downloads page.
              Don't forget to grab a trial license for the PE version.

              Please send feedback:
              We hope you enjoy this release!