apache > db
Apache DB Project
 
Font size:      

Derby Versioning Scheme

This document describes the Derby Versioning Scheme. This is a work-in-progress derived from Javadoc comments and from explanations Dan Debrunner and others posted to the Derby lists. Please post questions, comments, and corrections to derby-dev@db.apache.org.

Basic Numbering Scheme

Note
This is an overview of how the version scheme is set up in the Derby code and was intended to work for Cloudscape. It was described by Dan Debrunner on 26 Aug 2004.

Externally the Derby code uses a four part version number

Major.minor.fixpack.point or M.m.f.p

The initial version (as in the code drop from IBM) is set to 10.0.2.0

Major.minor is a typical major and minor release numbering scheme and maps to the version scheme expected by JDBC (major and minor version numbers for Driver and Database). A change in these numbers indicated new functionality in Cloudscape, typically that which modified the on-disk format in some way that would not be understood by an older release, e.g. support of a new SQL catalog item such as triggers. For Cloudscape each M.m version had its own (and only one) branch in the Perforce SCM system.

The point (p) value was increased every time a "point release" of Cloudscape was made, which were releases with a bug fix or fixes for customers. The point releases were always cumulative, that is 10.0.2.7 would include the fixes of 10.0.2.6 as well as the new bug fix(es). Point releases did not contain changes that changed the on-disk format of databases. Testing on point releases was restricted to nightly functional testing as a point release was usually in response to a customer issue, sothe fix had to be released within a few days of the report. As a "closed-source" project IBM could control releases so that the point number was only bumped when the software was actually released to a customer, regardless of the number fixes between releases.

Fixpack was intended to indicate a release of a given Major.minor that had undergone more testing than a point release, e.g. a multi-week quality assurance (QA) testing cycle. A fix pack release would include all the fixes of the previous point release, e.g. 10.0.5.0 would include all the fixes from 10.0.4.12 if that was the most recent point release for 10.0.

Fixpack (f) of 0 (zero) is a special value that indicates alpha quality software and causes version string to be appended with the word "alpha".

Additionally a boolean beta flag exists, which when true causes the version string to be appended with the word "beta" to indicate beta quality software. The initial code drop has the beta flag as false.

A build number can also be set which corresponded to the Perforce (SCM) global change number for Cloudscape.

The single branch model for a M.m used by Cloudscape meant that customers never received special one-off releases. A customer that needed a new release for a newly discovered bug always received a point release with all the bugs fixed for that branch and the new bug fix. E.g. a customer that was using 10.0.2.3 and found a new bug, might receive a point release that was 10.0.5.2, thus included all the fixes in the 10.0 branch and had undergone 3 additional QA testing cycles.

Sysinfo and how it reports versions

This version value is reported by the sysinfo utility, in some messages and in the java.sql.DatabaseMetaData methods as follows

Method Return value
String getDatabaseProductVersion()
String getDriverVersion() 		both return M.m.f.p [ {alpha|beta} ]
int getDatabaseMajorVersion()
int getDriverMajorVersion() 		both return M
int getDatabaseMajorVersion()
int getDriverMajorVersion() 		both return M
int getDatabaseMinorVersion()
int getDriverMinorVersion() 		both return m

Version Upgrade Mechanism

Note
This is a description of how the mechanics of the code is set up to handle upgrade and how it was intended to work for Cloudscape.

The database engine has a version EM.Em.Ef.Ep.

An on disk database has a persistent current version that corresponds to the engine that created it, or last upgraded it, DM.Dm.Df.Dp.

The general rules for Cloudscape were (and are in the Derby code)

For a on-disk database at version DM.Dm.Df.Dp

  1. Any engine with EM == DM and Em == Dm can be used.
  2. Any engine with EM.Em < DM.Dm will not boot the database.
  3. Any engine with EM.Em > DM.Dm will boot the database in soft upgrade mode.
  4. Any engine with EM.Em > DM.DM can boot and upgrade the database to its own version (EM.Em.Ef.Ep).
  5. Exception that any database created by an alpha or beta engine can not be upgraded or run in soft upgrade mode.
  6. Exception that any alpha or beta engine may not run in soft upgrade mode or perform an upgrade.
  7. Requirement that EM <= 10 and DM >= 10.

Soft upgrade is a mode where a newer engine can run against an older database successfully, but does not support any of the engine's new features that would not be understood by an engine at the database's version of DM.Dm. An exception is thrown if such a feature is used. The idea here is to allow a single version of the engine in a JVM be upgraded without requiring all databases be upgraded. Soft upgrade may make changes to the on-disk database that can be safely consumed by older releases, e.g. fix incorrect system catalog entries.

Upgrade is enabled by the boot connection to the database using the upgrade=true attribute. This modifies the on-disk form of the database by making any required changes, such as adding a new system catalog to support some new SQL feature. Once a database is upgraded, it cannot be downgraded.

Example (all version numbers are made up, only the inital code at 10.0.2.0 exists at the moment)

Database SALESDB created by 10.0.2.4 engine.

Any 10.0.x.y engine can now run successfully against SALESDB, e.g. application can use 10.0.2.8, 10.0.4.23, revert to 10.0.2.0 with no issues.

A future 10.2.0.0 (alpha) release would not boot SALESDB as it is an alpha release.

A future 10.2.1.3 (beta) release would not boot SALESDB as it is an beta release.

A future 10.2.2.3 release would boot SALESDB in soft upgrade mode. If say 10.2.2.3 had added GRANT/REVOKE, then any attempt to use those SQL statements would result in an exception indicating the database must be upgraded to 10.2. Also any new features added in 10.1 would throw an exception indicating the database must be upgraded to 10.1.

Connecting to SALESDB with 10.2.2.3 with the attribute upgrade=true would upgrade the database to 10.2.2.3. Subsquently any attempt to boot SALESDB with 10.0.x.y or 10.1.x.y would fail.

Upgrading System Catalogs

Note
This was described by Dan Debrunner on 16 March 2005

Here is some more on the mechanics of the upgrade for the system catalogs. I will add this into code comments in the relevant classes. I working on a submission that adds more comments into this area and sets the system catalog version number correctly for 10.1. This will then allow people to code the upgrade mechanics for new features.

System catalog upgrade issues - there are various kinds of changes that affect the system catalogs with a new release. A release may not contain all of these types, but will typically include at least one.

  1. Change structure of system tables, examples are:
    • add new system table
    • add new column to existing system table
    • add index to existing system table
  2. Modify contents of system table(s), examples are
    • fix incorect information, e.g. inccorrect nullability of column
    • Change a value for a set of rows to support new functionality (rare, usually the code uses the old values)
  3. New version of engine will write information that will not be understood by a previous release, e.g.
    • new table type value in systables
    • new alias descriptor in sysaliases

When running in soft upgrade mode the upgrade code (in DD_Version) needs to:

  1. Check the current soft upgrade level (property derby.softDataDictionaryVersion) and determine which "safe changes" can be made.
  2. Apply "safe changes" from category 2) in a single transaction. An example would be fixing incorrect information.
  3. Set derby.softDataDictionaryVersion to represent the current engine version to indicate the changes in 2. have been applied. Otherwise in soft upgrade no other changes are made to the system catalogs

In full upgrade mode, initial database connection with upgrade=true these steps are made:

  1. Check the current hard upgrade level (property DataDictionaryVersion) and determine which changes are required.
  2. Make any changes from categories 1) and 2) in a single transaction.
  3. Set DataDictionaryVersion to represent the current engine version thus indicating the upgrade on the system catalogs has been completed. Thus new features can execute code that would result in changes in category 3

Code that will make changes in category 3) or requires other changes made by hard upgrade is required to check the version of the system catalogs using the DataDictionary.checkVersion method, or the corresponding utility method in the parser. This check will throw an exception if the system catalog has not been hard upgraded to the current level.

As examples, with the current development going on for Derby I know of these items that require upgrade thought:

  • Java signatures in method definitions, writes information into the system catalog in its current structure, but the signature would not be understood by 10.0. Would need to perform a checkVersion(10.1) if a signature is present. (10.1 is not the actual constant, just to give an idea).
  • Log record checksum to detect out of order writes. If in soft upgrade mode, need to ensure checksum records are not written. Though this is a store upgrade issue, not a system catalog one.
  • Synonym support, like java signatures, most likely would be implemented by new information in system catalogs that would not be understood by 10.0, thus a checkVersion(10.1) would be required.