Job Persistence

Introduction

GridSAM can utilise most JDBC compliant database to support job persistence and pipeline recovery. The default setup uses the embedded Hypersonic SQL database, which requires no administrative attention. However if you would like to use other database implementation, you will need to modify the configuration file $OMII_HOME/apache-tomcat-5.5.23/webapps/gridsam/WEB-INF/classes/database.xml referenced by the main jobmanager.xml file.

GridSAM has been tested with the following databases. You should consult the documentation for the MySQL Database to learn how to experiment with other systems.

GridSAM can survive server crashes given the database is intact. The container can simply be restarted and the failed job stages would recover from the point they have left off. For example, if the server has failed during the staging-out stage of a job, GridSAM will attempt to stage files out again without re-running the preceduing stages of the job. Therefore, it is important the database server is kept on high availability hardware and the vendor-specific tools be used to replicate the database servers in case of failure.

Supported Databases

Hypersonic SQL Database

GridSAM is preconfigured with an embedded Hypersonic SQL database that requires no administrative attention. It provides a fast in-VM JDBC compliant database to the JobStore abstraction.

The Hypersonic database files are stored in the directory $OMII_BASE/apache-tomcat-5.5.23/webapps/gridsam/WEB-INF/data by default.

WARNING: Redeploying GridSAM will overwrite the database files. Please ensure you have backed up the directory prior to redeployment if you want to retain the database content.

Task: Changing location of database files
  1. Create the persistence directory (i.e. mkdir /new/database/dir).
  2. Copy the preset data files from the original data directory into the new one.
    $> cp -R ${OMII_HOME}/apache-tomcat-5.5.23/webapps/gridsam/WEB-INF/data /new/database/dir
  3. Ensure jobmanager.xml imports the sub-module called database.xml available to GridSAM as a classpath resource.
        ..
    <!-- database.xml file available in the classpath -->
    <sub-module descriptor="database.xml"/>
    ..
  4. Edit the file database.xml in webapps/gridsam/WEB-INF/classes/, so that the contributions "gridsam.SchedulerConfig" and "gridsam.JobInstanceStoreConfig" have the database URL pointing to the new location. A high preference value will ensure the properties override the common values in the default contribution.
        ...
    <contribution configuration-id="gridsam.SchedulerConfig">
        <Properties preference="10">
            org.quartz.dataSource.QuartzDataSource.URL = jdbc:hsqldb:file:/new/database/dir
        </Properties>
    </contribution>

    <contribution configuration-id="gridsam.JobInstanceStoreConfig">
        <Properties preference="10">
            hibernate.connection.url=jdbc:hsqldb:file:/new/database/dir
        </Properties>
    </contribution>
    ...
Task: Using a networked Hypersonic server

By using a networked Hypersonic Server, the SQL processing can be performed on a host other than the container host. This mode of deployment is suitable for high-traffic service.

  1. Install Hypersonic SQL. Please consult the Hypersonic documentation for installation instruction.
  2. Copy the data files from the original data directory into the desired persistence directory (e.g. ~/hsqldb)
    $> cp -R ${OMII_HOME}/apache-tomcat-5.5.23/webapps/gridsam/WEB-INF/data ~/hsqldb/
  3. Start the Hypersonic server (please consult the Hypersonic documentation for other options). The database server can be distributed on a differ host as the GridSAM service.
    $> java -cp ${HSQLDB_HOME}/lib/hsqldb.jar org.hsqldb.Server -database.0 ~/hsqldb/gridsam-db -dbname.0 gridsam
  4. Ensure jobmanager.xml imports the sub-module called database.xml available to GridSAM as a classpath resource.
        ..
    <!-- database.xml file available in the classpath -->
    <sub-module descriptor="database.xml"/>
    ..
  5. Edit the file database.xml in webapps/gridsam/WEB-INF/classes/, so that the contributions "gridsam.SchedulerConfig" and "gridsam.JobInstanceStoreConfig" have the database URL pointing to the server location. A high preference value will ensure the properties override the common values in the default contribution.
        ...
    <contribution configuration-id="gridsam.SchedulerConfig">
        <Properties preference="10">
            org.quartz.dataSource.QuartzDataSource.URL = jdbc:hsqldb:hsql://your.db.host/gridsam
        </Properties>
    </contribution>

    <contribution configuration-id="gridsam.JobInstanceStoreConfig">
        <Properties preference="10">
            hibernate.connection.url=jdbc:hsqldb:hsql://your.db.host/gridsam
        </Properties>
    </contribution>
    ...
  6. You must ensure the URL jdbc:hsqldb:hsql://$HOST[:$PORT]/$DBALIAS is correctly specified where $HOST is the hostname of the database server, $PORT is the optional port number and $DBALIAS is the name of the database specified by the -dbname.0 flag when you start the server.

MySQL Database

To use MySQL as the database implementation, you must ensure the necessary tables are created. The database scripts required by GridSAM can be found in the directory config/ddl/tables_[DB_VENDOR].sql in the server distribution.

  1. Find the script that best matches your database type ( tables_mysql.sql in this case) and execute the script using the tools supplied by your database vendor. The SQL script should create all the necessary tables required for the submission pipeline.
  2. Next, ensure jobmanager.xml imports the sub-module called database.xml available to GridSAM as a classpath resource.
        ..
    <!-- database.xml file available in the classpath -->
    <sub-module descriptor="database.xml"/>
    ..
  3. Edit the file database.xml in webapps/gridsam/WEB-INF/classes/, so that the contributions "gridsam.SchedulerConfig" and "gridsam.JobInstanceStoreConfig" have the database URL pointing to the server location. A high preference value will ensure the properties override the common values in the default contribution.
        ...
    <contribution configuration-id="gridsam.SchedulerConfig">
        <Properties>
        org.quartz.jobStore.driverDelegateClass=[SchedulerDBDelegateClass]

        # Default setting: Use HSQL as the database
        org.quartz.dataSource.QuartzDataSource.driver = [JDBCDriverClassName]
        org.quartz.dataSource.QuartzDataSource.URL = [JDBCURL]
        org.quartz.dataSource.QuartzDataSource.user = [JDBCUser]
        org.quartz.dataSource.QuartzDataSource.password = [JDBCPassword]

        # OR
        # Use the following instead if the data source is configured in the application server (such as the OMII data source)
        # org.quartz.dataSource.NAME.jndiURL = DB_JNDI_URL

        </Properties>
    </contribution>

    <contribution configuration-id="gridsam.JobInstanceStoreConfig">
        <Properties>
hibernate.connection.driver_class=[JDBCDriverClassName]
hibernate.connection.url=[JDBCURL]
hibernate.dialect=[DBVendorDialectClass]
        </Properties>
    </contribution>
    ...
  4. Replace values labelled as [] with your database vendor specific values. The Scheduler and JobStore can point to different databases for added performance. You also need to make available the .jar file of the JDBC driver to the OMII container by putting it in to the $OMII_BASE/apache-tomcat-5.5.23/common/lib directory. Both the Scheduler and JobStore would attempt to match closely with your vendor according to the database dialect classes chosen. Please find the delegate classes that closely match your database vendor here for the scheduler, and the dialect here for the JobStore. In the case of MySQL, the values are
    SchedulerDBDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
DBVendorDialectClass: org.hibernate.dialect.MySQLDialect
JDBCDriverClassName: com.mysql.jdbc.Driver
JDBCURL: jdbc:mysql://${HOST}:${PORT}/${DBALIAS}
JDBCUser: your database username
JDBCPassword: your database password