Unification of the relational persistence layers

XForms by itself doesn't fully solve the question of how to persist data. It provides solid foundations that we can leverage to persist data, but intentionally doesn't answer questions such as "how is the data organized?" or "how is data saved, retrieved, or searched, this in an actual database, say Oracle?". From the start, we aimed at Form Builder and Form Runner to work as much as possible out-of-the-box, which meant that we had to answer those questions. We did this by:

• Defining an API for persistence, which Form Builder and Form Runner use, so both form definitions and form data are saved through this API.
• Providing built-in implementations of this API for some of the most popular databases.

On the implementation side, we started with the eXist XML database; it ships with Orbeon Forms, and enables you to have a solution that works out-of-the-box without any setup required. Then, we added implementations for two relational databases: Oracle and MySQL. Those two databases are almost opposites in terms of features: Oracle is very rich and provides tons of extensions, while MySQL is a much "simpler" database. This lead us to implement things quite differently on those two databases, so we went with two completely separate implementations, which didn't really share any code.

Then, with version 4.3, we added support for DB2, and continued having separate code for each database, but found this approach not to be sustainable. It was very much a maintenance problem as every fix, new feature, or performance improvement done in one version of the code had to be ported in two other versions. This meant that the 3 implementations were not fully in sync: for instance, in 4.3 some features were added and were only available in DB2 and MySQL, but not Oracle, while some performance improvements were available for Oracle but not DB2 and MySQL. Making any type of change felt like a huge task, let alone thinking of supporting more databases.

In the upcoming Orbeon Forms 4.4, we resolved this by creating a unified implementation, which works across relational databases. The difficulty was in finding ways of doing things that didn't rely on unique database features, but were efficient enough on all databases. There are still places in the code where we need to do something different depending on the database, but those are now the exception rather than the rule.

While having a unified implementation for relational databases doesn't provide any new user-facing feature, this will allow us to improve those implementation faster, and to add support for new relational databases (did I hear someone mention SQL Server and PostgreSQL?).