Replacing a D-node with local disk failover configured
05 December 2016 08:25 PM
In a multiple node cluster with local disk failover configured, there may be a need to replace a server with new hardware. This article explains how to do that while preserving the failover configuration.
Consider a 3-node cluster with local disk failover for database Test, and the forest assignment for the hosts looks like this: (all forests ending with 'p' are primary and those ending with 'r' are replica)
With this configuration under normal operations, each host will have the two primary forests "open" and the replica forests "sync replicating".
In the event of a node failure of say, Host B, primary forests on Host B will failover to Hosts A & C as expected. The forests a-3r and c-4r are now "open" and acting as master forests.
When Host B comes back online, the replica forests a-3r and c-4r will continue as acting masters, and forests b-3p & b-4p on Host B will now act as replicas; This state will persist until another failover event occurs or the forests are manually restarted.
Replacing a Host
In the case where a node in the cluster needs to be physically replaced with another node, it is important to preserve the original master-replica configuration of the forests, so that there is no performance burden on a single node hosting all the primary forests.
Example: replacing Host-B with a new Host-D
The steps listed below show how to replacing a node (old Host-B with new Host-D) without affecting the failover configuration:
This will replace Host B with Host D, as well as preserve the previously existing primary-replica configuration among the hosts.
It is important to make sure that the database is quiesced before taking the forest backups. The idea is to disallow ingestion/updates on the database: One technique is to quiesce a database by making all of its forests 'read-only' - http://docs.marklogic.com/guide/admin/forests#id_72520 during the process and revert once complete.
Note: This example assumes a distributed master-replica configuration of a 3-node cluster. However, the same procedure works with other configurations with some careful attention to the number of forests on each host and breaking replication between the right set of hosts.