Knowledgebase: MarkLogic Server
MarkLogic Database Restore Across Clusters
16 July 2020 04:49 PM

Summary

There are scenarios where you may want to restore a database from a MarkLogic Server backup that was taken from a database on a different cluster. 

Examples

Two example scenarios where this may be appropriate:

- For development or testing purposes - you may want to take the content from one system to perform development of testing on a different cluster.

- A system failed, and you need to recreate a cluster and restore the database to the last known good state.

Constraints

There are constraints on performing a database restore from a MarkLogic database backup across clusters

  1. The source and target servers must be the same Operating System.  More specifically, they must be able to use the same MarkLogic Server installation package.
  2. The backups must be accessible from all servers on which a forest in the target database resides.   
  3. The path to the backups must be identical on all of the servers.
  4. The MarkLogic process must have sufficient access credentials to read the files in the backup.
  5. If the number of hosts and/or forests is different, see Restoring a Reconfigured Database.

If running MarkLogic versions prior to 9.0-4 then the following conditions must also be met

  1. The forest names must be identical in both the source database and the target database.
  2. The number of forests in both the source and target databases should be the same.  If the source database has a forest that does not reside on the target, then that forest data will not be included in the target after the database restore is complete.

Note: Differences in index configuration and/or forest order may result in reindexing or rebalancing after the restore is complete

Debugging Problems

If you are experiencing difficulties restoring a database backup, you can validate the backup using xdmp:database-backup-validate, or xdmp:database-incremental-backup-validate:

1. In the Query Console, execute a simple script that validates restoring the backup.  Something like

xquery version "1.0-ml";

let $db-name := "Documents"

let $db-backup-path := "/my-backup-dir/test"

return xdmp:database-restore-validate(

    xdmp:database-forests( xdmp:database($db-name)),

    $db-backup-path)

But with the $db-name and $db-backup-dir set appropriately.  The results will be a backup plan in xml format. Look at both the ‘forest-status’ and ‘directory-status’ for each of the forests.  Both should have the “okay” value.

A common error for the ‘directory-status’ is “non-existent”.  If you get this error, check the following.

- Verify that the backup directory exists on each server in the cluster that has a forest in the database;

- Verify that the backup directory has a “Forests” subdirectory, and the “Forests” directory contains subdirectories for each of the forests that reside on the Server.

- For the above directories, subdirectories and file contents, verify that the MarkLogic process has the proper credentials to access them.

2. If xdmp:database-backup-validate, or xdmp:database-incremental-backup-validate does not indicate any errors, then look in the MarkLogic Server’s ErrorLog.txt for entries during the time of the restore for any errors reported.  It is a good idea to set the MarkLogic Server group’s ‘File log level’ to ‘debug’ in order to get detailed error messages.

Helpful Commands:  

On Unix Systems, the following commands may be useful in troubleshooting:

  • Check the 'file system access user ID' for the MarkLogic process
    • ps -A -o fuser,pid,comm | grep MarkLogic
  • View file/directory permissions, owner and group
    • ls -l
  • Change ownership recursively.  In a default installation this should be daemon
    • chown -R daemon.daemon /path/to/Backup
  • Add read and write permissions recursively
    • chmod -R +rw /path/to/Backup

Further Reading

Transporting Resources to a New Cluster

Phases of Backup or Restore Operation

Restoring a Reconfigured Database

(12 vote(s))
Helpful
Not helpful

Comments (0)