Scripting Failover: "flipping" replica forests back to their masters using XQuery
19 June 2020 02:19 AM
If you have forest level failover configured on your MarkLogic cluster, in the event that a single host in the cluster loses contact with the other hosts, the forests will fail over to the backup set of forests: the replica forests
What should I do in the event of a failover?
Failover shifts the responsibility for a given set of forests over to other hosts in the cluster; if the failing host "loses" control of its' forests, control is not automatically given back when the master becomes available; failing forests has to happen manually.
To fail a forest back (to "flip" control back to the master), if both the replica and master forests are in sync with each other, all that's needed is to restart the replica forest. This can be done using the admin API (Configure > Forests > Forest Name > Status > Restart), or XQuery (xdmp:forest-restart):
The above code is intended as a sample for something that could be used as a scheduled task that will automatically check for failed over forests and to flip them back where possible.
It's also worth noting that this may not be something you'd want to do; in many cases, a failover event might be a warning of a problem that occurred that needs to be investigated (for example: a disk error), so if you are planning on managing failing back forests automatically, you may want to ensure that you are monitoring the ErrorLogs for evidence of failover events so you know that they're happening.