Running a Parallel Standby Environment

To run a parallel standby environment, your current environment must be backed-up and then restored to the parallel one.

Restoring an environment

The following scenarios are addressed with restoration of a product environment:

Scenarios
Cold Standby A method of redundancy in which the secondary (i.e., backup) system is restored and brought online when the primary system fails. The system on cold standby receives scheduled data backups, but less frequently than a warm standby. Cold standby systems are used for non-critical applications or in cases where data is changed infrequently. The Disaster Recovery server names are the same as the server names currently in use.
Hot Standby A method of redundancy in which the primary and secondary (i.e., backup) systems run simultaneously. The data is mirrored to the secondary server in real time so that both systems contain identical information. The Disaster Recovery server names are different than the non-standby server names. See Database Disaster Recovery Options for issues when using SQL Log Shipping and Database Mirroring.
Hot Standby is also known as Active-Active replication.
See the appropriate vendors’ product documentation for information on implementing Hot Standby for supporting systems such as Operating Systems, Databases or other underlying systems required by your environment.
Warm Standby A method of redundancy in which the secondary (i.e., backup) system runs in the background of the primary system. Data is mirrored to the secondary server at regular intervals, which means that there are times when both servers do not contain the exact same data. See Database Disaster Recovery Options for issues when using SQL Log Shipping and Database Mirroring.
Warm Standby is also known as Active-Passive replication.
See the appropriate vendors’ product documentation for information on implementing Hot Standby for supporting systems such as Operating Systems, Databases or other underlying systems required by your environment.
If loss of data is a concern, another option would be to perform SAN (system area network) mirroring, i.e. everything that is written to a SAN in the main site is replicated via the SAN link to the disaster recovery site. The downside of this option is that it can be cost-prohibitive.

Restoring to a Cold Standby environment

Follow the steps below to restore to a cold standby:

  1. Ensure that the Windows machine is in working order.
  2. Install the Server and Site components.
  3. Apply service packs and / or updates.
  4. Restore the installation directory from backup.
  5. Restore databases from backup.
  6. Restore servers and IIS servers from backup.
  7. Restore any other files for your processes.
  8. Run the Setup Manager to update the new license key.
  9. Start your server.
  10. Test your processes and data.
  11. Check the Host Server log and the server event log for any errors.

Preparing a Hot or Warm Standby environment for restoration

It is important to note that preparation of the Hot or Warm Standby environment is required prior to the disaster occurring, as the system stores configuration and licensing information within the configuration database.

Follow the steps below to prepare the environment for a Hot or Warm Standby restoration:

  1. Install the product on the Disaster Recovery servers.
  2. Perform a full system backup of the Disaster Recovery servers.
  3. Perform a backup of the data in the following two tables in the database:
    • Server.Server
    • HostServer.LicenseKey
  4. Take the Hot or Warm Standby servers offline.

Restoration of a Hot or Warm Standby environment

Follow the steps below to restore a Hot or Warm Standby environment:

  1. Bring the Hot or Warm Standby environment online.
  2. Restore the the product ServiceBroker directory ("%ProgramFiles%\K2\ServiceBroker") from backup.
  3. Restore any other custom assemblies that your processes may require to the machine.
  4. Restore the backed up database.
  5. Restore the Disaster Recovery Servers.
  6. Restore the data from the two tables backed up earlier.
  7. Update all references to the old machine such as ASPX pages, links to the Workspace, Worklist Web parts in SharePoint etc to point to the new machine name.
  8. Start the Service.
  9. Test your processes and data.
  10. Check the Host Server log and the server event log for any errors.

This topic covers the recommended procedures to enable Disaster Recovery (DR) within the infrastructure that leverage physically different data centers for production and disaster scenarios.

The following parameters are assumed:

  • Accepted data loss of 4 hours.
  • System data is consistent upon a cut over to the DR site.

Should the window of accepted data loss need to be adjusted, the scheduled jobs laid out below should be recalculated accordingly.

The procedure below leverage an initial set of database backups and then transaction log shipping with a point-in-time recovery to allow for consistent product data in the event of a disaster event.

Prerequisite

The following should be done:

  • Within the DR site:
    1. Record the licensing information in the following SQL tables:
      • Server.Server
      • HostServer.LicenseKey
        • The data should be similar in both locations.
    2. Create a SQL Script that will reapply this information during an actual disaster recovery event.
      This is required because the licensing is stored within the database and is bound to a specific environment. As such restoring databases across environments as in a DR scenario will replace the DR license with the PRODUCTION license. Creating a SQL script to restore the original DR license, while not always necessary, will make this a more repeatable procedure should it be desired.
    3. Keep the host server nodes turned off until a DR cut over is required. This ensures that the DR site does not try to process transactions based upon its copy of the database.

Initial Setup

The following setup is required:

  • Within the PRODUCTION site:
    1. Turn off the the system service on all nodes within the farm. This insures no data manipulation.
    2. Create Full Database Backups of all product databases as mentioned in databases Backup and Restore. Alternatively, incremental backups may also be used . Either way, compression of databases is advised for efficiency.
    3. Script out all external database server artifacts, e.g. database logins, users, etc.
    4. Backup additional database(s) leveraged by the solution.
    5. Turn on the service on all nodes within the farm.
    6. Move the database backups to the DR site.
  • Within the DR site:
    1. Restore the PRODUCTION database backups.
    2. Reapply external artifacts (logins, etc).
    3. Re-apply licensing, as identified in the Prerequisite step.

Ongoing

  • Within the PRODUCTION site:
    1. Create a scheduled SQL Agent job (recommended hourly) within the SQL Server that backs up the transaction logs for the following:
      • All product databases.
      • Solution database(s).
    2. Schedule a Log Shipping job to transmit these logs to the DR site.
  • Within the DR site:
    1. Within the DR site create a scheduled job (recommended hourly) that will apply any transaction log backups for the databases that are greater than 3 hours hold.
  • Archive the just-applied Transaction Log files.

Disaster Recovery Cutover Procedure

  1. If possible, backup the ‘tail end’ of the log(s) from the primary site and transmit them to the secondary site.
  2. Determine the time of the disaster at the production site and subtract 2 hours. Choose an exact time of recovery that makes sense, for instance 13:35:00:000.
  3. Recover queued transaction logs for all databases to that same specific point in time (the chosen time may or may not include the ‘tail end’ backups obtained in 1). In this example, all databases should be recovered until 13:35:00:000.
  4. Reapply the appropriate server licenses within the DR environment (preferably leveraging the script file recommended in the Prerequisite section, point number 2).
As a result of the time difference, there will always be 1 (or 2 or 3 as need be) log difference. This allows a restore to a point in time operation on the final log in the event of a DR event. After this restore to a point in time operation all databases will be consistent.

Testing

A manual failover process should be executed during development / QA to ensure that the procedure works as expected. All operations should be tested in order to confirm that the primary data files and external artifacts (such as logins) are valid.