Background:

Due to concerns around high availability, every node in a MarkLogic Server instance has its own copy of configuration files like databases.xml, hosts.xml, etc. Consequently, it's important for each node to make sure it's working off the correct version of the various configuration files.

This is accomplished by using the Admin/config-lock.xqy library to make sure the appropriate configuration file is locked before they are updated to make sure that multiple file requests through the Admin API or REST Management API don’t corrupt the configuration files. As a consistency check, before locking, the Admin/config-lock.xql library makes sure that the timestamps of the configuration files are more recent than the timestamps in the lock file. This is done to make sure that if someone else has acquired the lock and updated the files, the lock won’t be returned until the files are consistent.

Problem Statement:

If you restore the "App-Services" database from another cluster, you can wind up with timestamps in the lock file that bear no useful relationship to the actual configuration files. This is because the "App-Services" database is where the lock file /config-lock/config-timestamps.xml is located, which contains the timestamps of the configuration files from the last time they were locked - for the restore target cluster. Restoring the "App-Services" database from a different source cluster will overwrite the restore target's /config-lock/config-timestamps.xml file. This causes the error "MANAGE-TIMESTAMPOLD: Config files out of date on host" - which is triggered when you try to perform any subsequent update on the restore target cluster's MarkLogic Server configuration files.

Note: In general, this error is triggered by any PUT or POST operation to the REST Management API, with the exception of some of the security endpoints, which actually update the security database instead of the configuration files.

Workaround:

In the restore target cluster showing the "MANAGE-TIMESTAMPOLD" error, deleting the lock file “/config-lock/config-timestamps.xml” should fix the problem because that same file will be re-created with the correct timestamps.

Note: it is very important to make sure no one is running Admin updates before deleting the lock file, as the timestamps corresponding to those updates will be lost when you delete the lock file.

Best Practice:

The "App-Services" database is one of the default MarkLogic databases, used to track configuration file timestamp information local to the cluster on which it resides. It is not recommended to restore "App-Services" databases across MarkLogic clusters.