Friday, March 29, 2013

Orbeon Forms 4.1 M1

Today we released Orbeon Forms 4.1 M1, a milestone build for the upcoming Orbeon Forms 4.1 release. This is not a stable release.

Here is the list of changes since 4.0.1:
  • Form Builder
    • Ability to enter HTML labels and hints in the builder (#870)
    • Introduce a distinction between the 'read' and 'update' permissions (#806)
    • Edit Source must not send data to server incrementally (#848)
  • Form Runner
    • Proxy portlet: ability to configure forwarding of headers and parameters (#886)
  • XForms engine
    • Use a single property for input/textarea string replacement (#867)
    • Non-relevant group around <li> with AVT doesn't place xforms-disabled (#889)
    • REGRESSION: Uploading large files causing "Got unexpected request sequence number" error (#894)
    • xf:case: xforms-case-selected/deselected are not placed on first delimiter (#893)
    • Fix basepath of TinyMCE (#896)
  • Other
    • First automated client-side test using ScalaTest Selenium DSL
    • Upgrade to Scala 2.10.1
You can download the builds using these links:


Don't forget to grab a trial license for the PE version.

Please send feedback:

Thursday, March 28, 2013

Iterating

Photo by hide99
During the Orbeon Forms 4 development cycle, we released over 20 intermediate builds (including milestones, betas, and a release candidate - you can see them all here). More often than not, we released one build per week. We have found that doing so was extremely positive:

  • It helps maintain a great sense of progress and accomplishment.
  • As each release is documented in a blog post, the outside world can also get a better sense of what's going on.
  • Users can try out new features as they get implemented. (This is also possible with automated nightly builds, but those are not tested by humans).
  • Developers have to keep an eye on the current issues and features that need work.
  • It is harder to postpone tasks such as testing and documentation (designing features and writing code is so much more fun!).
So the product, developers, and users all benefit and we really would like to keep doing this as we work towards the next stable releases of Orbeon Forms.

But it goes further. As we are putting a bigger emphasis on automated testing, our ideal is that every feature that we implement and every bug we fix has an automated test, even if it involves some funky browser behavior. This will give us increased confidence that between two builds we don't have serious regressions.

So we hope that this will enable us to iterate stable builds much faster than in the past. And that would be really awesome!

Monday, March 25, 2013

Automated browser testing with Selenium on Travis-CI

We've written before about the importance of having a set of comprehensive unit tests for Orbeon Forms. For instance, we put in place the necessary infrastructure to test code in Scala back in 2011 and to test XML pipelines back in 2005. With this, every time a check-in is made, a build is automatically created and 687 tests (and counting) are executed. If any test fails, we get a notification, and fixing the build becomes our priority.

But until now, we couldn't automatically run tests that requires a browser, which means that a significant portion of Orbeon Forms couldn't be automatically tested, including all our client-side code (JavaScript and CoffeeScript), interactions between client-side and server-side code, and higher level components such as Form Builder and Form Runner.

We now have this infrastructure in place:

  • Running the tests automatically – Everytime we do a check-in to our Github repository, Travis-CI checks the code out, compiles the code, runs the tests, and sends us an email if any test fails.
  • Running Tomcat from Ant – We still use1 the venerable Ant as our build system, so it is up to Ant to compile the code and run the tests. The new tests require a server to run, and for that we use Tomcat. During the build, we install Tomcat in the build directory, and configure it to pick up its configuration files from test/catalina_base, by setting the catalina.base property. Tomcat needs to run in parallel with the tests, which we do with Ant's Parallel task. Before running the tests we need to make sure Tomcat is successfully started, which can be done very easily with Ant's Waitfor task. To see how this is done, check the target name="test" in our build.xml.
  • Starting a browser – Our tests run in a real browser: Firefox2. Since the test machine isn't connected to an actual screen, we run Xvfb through our .travis.yml.
  • Writing the tests – To drive Firefox, we use Selenium 2 with WebDriver. Our tests are written in Scala, with ScalaTest and its Selenium DSL.

  1. And in case reading that we are still building on Ant makes you worry about our sanity, rest assured that we are thinking of moving to a more modern build system, like sbt.
  2. ChromeDriver currently can't be used on Tavis-CI as Chrome does not work in an OpenVZ container, which is used by Travis-CI. But in the future, while still using Travis-CI, we might run the tests on more browsers through Sauce.

Monday, March 18, 2013

Orbeon Forms 4.0.1

Today we released Orbeon Forms 4.0.1, a bug-fix and performance release.

Here is the list of changes since 4.0.0:
  • Form Builder
    • Open/close state of existing sections is lost when inserting new sections (#177)
    • Itemset editor: fixup label/value if needed (#830)
    • Close large form sections upon load to improve performance (#838)
    • CodeMirror gets unusably slow with very large forms (#840)
    • Service body lost after hitting save (#859)
    • Validation Properties shows, but doesn't contain properties (#861)
  • Form Runner
    • fr:section shows closed arrow when becoming relevant (#862)
    • Empty, non-closable dialog upon PDF error (#864)
    • Property to allow PDF generation even if form is invalid (#871)
  • XForms engine
    • MIP evaluations are slow with many bind/@name (#834)
    • Failing constraint should default to false() (#835)
    • Support for XPath id() function (#836)
    • Support xxforms-replace event (#837)
    • Order XBL JS and CSS resources in consistent order (#844)
    • Can't access event('response-body') if debugging is disabled (#863)
    • Value of xf:textarea mediatype="text/html" not updated on "switch" (#868)
    • AssertionError upon xforms-submit-error (#876)
    • Configurable xf:input/xf:textarea string replacement (#855/#867)
    • Lightweight syntax for HTML LHHA (#874)
  • Other
    • Blank page-public-methods property causes default to GET HEAD (#865)
You can download the latest version of Orbeon Forms from the downloads page.

Please send feedback:

Wednesday, March 13, 2013

Dropdown populated based on values from another form

Say you created two forms with Form Builder: a contact form and a phone call form. In the phone call form, you use one of the fields to capture the name of the person you'll have a call with. Instead of using a text field, you'd like to have a dropdown, with the values in the dropdown coming from values previously captured with the contact form.

Form Builder doesn't come, out-of-the-box, with a way to link forms in the manner described above. However, this is something that you can do with Form Builder's Services & Actions. The idea goes as follows:

  1. Write a query retrieving the contact names captured with the contact form. The query will differ depending on the persistence layer you're using: it will be in XQuery if you're storing data in eXist, or SQL if you're storing data in Oracle or MySQL.
  2. Create a service that runs this query. It will be either an HTTP service (to run XQuery) or a database service (to run SQL).
  3. Add a dropdown to your form.
  4. Create an action that runs the service on form load, and use the result to populate the dropdown.

The query

For the purpose of this post, we'll assume data is stored in eXist, which is bundled with Orbeon Forms, and we'll query data in the contact form, which is one of the sample forms. Since we're querying eXist, we're using XQuery:

for $c in /form/contact return element contact {
    element label {concat($c/first-name, ' ', $c/last-name)},
    element value {tokenize(util:collection-name($c), '/')[last()]}
}

This query returns one <contact> element per entry in the contact form, with for each one, the person name (to be shown in the dropdown) and an id (stored in form data, so we can link a phone call with a person). The root element of data captured with forms you create with Form Builder is always <form> and in the contact form the first/last name are in a section named contact, so the query iterates over /form/contact. Also note how the query retrieves the document id in which a node $c is stored with tokenize(util:collection-name($c), '/')[last()]. With this, the query returns <contact> elements that look like:

<contact>
    <label>Homer Simpson</label>
    <value>9eff349bfd95aab8d4d5e048bd25a815</value>
</contact>
<contact>
    <label>Charles Burns</label>
    <value>138b27e80eefc3f0429a3d14cbc77df527630787</value>
</contact>
...

The service

You can pick a name for the service, say get-contacts. The URL should point to the contact form, as it is stored in eXist. With the embedded eXist, if Orbeon Forms is deployed on http://localhost:8080/orbeon then the URL pointing to the contact form will be (here the URL is split in two lines for formatting, when pasting it in Form Builder put it on one line with no spaces):

http://localhost:8080/orbeon
  /exist/rest/db/orbeon/fr/orbeon/contact/data

The service POSTs and XML document as follows, with the XQuery above within the <exist:text> element.

<exist:query xmlns:exist="http://exist.sourceforge.net/NS/exist" max="0">
  <exist:text> ... </exist:text>
</exist:query>

The action

Finally, the action calls the service on form load, and uses the result to populate the dropdown, as shown in the screenshot on the right (click it to see a larger version). Click the test button, and you'll see how the dropdown populates with values entered previously in your contact form.

Wrapping it up

Since populating a dropdown with values from another form at this point isn't supported out-of-the-box by Form Builder, to do so you need to write a query against the database and use services and actions to run the query and populate the dropdown. While this isn't as simple as we'd like it to be, and future versions of Orbeon Forms might make this process simpler, it is already something that you can do today, and it goes to show the flexibility of the services and actions feature, which allows you to query existing data, whether it has been captured with a form you created with Form Builder or by some other mean.

Tuesday, March 12, 2013

Announcing Orbeon Forms 4.0

After more than 2000 commits, 19 milestone builds, 3 betas, and one release candidate, we are proud to announce the final release of Orbeon Forms 4.0!

As a goal for 4.0 we wanted to create a release we care about, on the outside as well as the inside. So in this release, in addition to new features and bug-fixes, we have fixed countless UI issues, refactored a lot of code, and thought a lot about architecture.

So what's new since the last major version, Orbeon Forms 3.9? The answer is: a whole lot!

1. A brand new look

Let's start with one of the most obvious changes. Form Runner and Form Builder feature a new look based on Twitter Bootstrap. Forms are more pleasant out of the box, and chances are that more developers will be familiar with Bootstrap when customization is desired.

New look and feel with Bootstrap


2. Security and access control

Orbeon Forms 4.0 introduces many security improvements:
  • All services are protected by default.
  • The built-in eXist database is protected by default.
  • All uses of cryptographic algorithms have been reviewed and improved (blog post).
  • File uploads are now tamper-resistant (blog post).
  • Ajax requests are better protected against DOS attacks.
  • Processing and loading of external DTDs when XML validation is turned off (which is in most cases) is disabled by default.
  • The product version number is hidden by default.
  • You can protect your Form Runner forms with a captcha (reCAPTCHA or SimpleCaptcha) (documentationblog post).
  • Form Builder supports setting fine-grained CRUD permissions for individual forms (blog post).

3. Repeats in Form Builder

This is one of the most important features of the release: Form Builder now supports grids with repeated rows. See this blog post which introduces the feature.

Example of repeated content


4. Nested sections and live controls in Form Builder

Under the hood, Form Builder had major architecture changes. In 4.0, everything you see in the builder is dynamic. This allowed us to add support for nested sections, and also to let controls (including your own custom controls built with XBL) appear in the builder as they will when the form is deployed. This is also a strong foundation for future improvements to the builder.

Examples of XBL controls at design time

5. Multi-page "wizard" view

The wizard view is a nice new way to present a form to the user, as a sequence of multiple screens. In fact, chances are you will want this mode for most forms! See this blog post which introduces the feature.

6. Improved portal support

Both the full portlet and the Form Runner proxy portlet have been improved and tested with Liferay 6.1 GA2. Other improvements include:
  • Liferay role and user information (including email and full name) is available to the full portlet (documentation).
  • The workflow-review/workflow-send buttons work with the Form Runner proxy portlet.
  • The Orbeon Forms CSS is sandboxed so that the portal style is not impacted.
Form Runner in Liferay 6.1

7. New web site

We have simplified and modernized our web site for the 4.0 release. Check it out! From there you can find out more about Orbeon Forms 4.0, how to try it out, and how to get support.

8. And much more

There are too many new features to describe in this post. For more details, see the Orbeon Forms 4.0 release notes, which also covers backward compatibility.

We hope you enjoy this release!