Replacing a D-node with local disk failover configured
05 December 2016 08:25 PM


In a multiple node cluster with local disk failover configured, there may be a need to replace a server with new hardware. This article explains how to do that while preserving the failover configuration.

Sample configuration

Consider a 3-node cluster with local disk failover for database Test, and the forest assignment for the hosts looks like this:  (all forests ending with 'p' are primary and those ending with 'r' are replica)

Host A Host B Host C
forest a-1p forest b-3p forest c-5p
forest a-2p forest b-4p forest c-6p
forest a-3r forest b-1r forest c-2r
forest a-6r forest b-5r forest c-4r

With this configuration under normal operations, each host will have the two primary forests "open" and the replica forests "sync replicating".

Failover Example

In the event of a node failure of say, Host B, primary forests on Host B will failover to Hosts A & C as expected. The forests a-3r and c-4r are now "open" and acting as master forests. 

When Host B comes back online, the replica forests a-3r and c-4r will continue as acting masters, and forests b-3p & b-4p on Host B will now act as replicas; This state will persist until another failover event occurs or the forests are manually restarted.

Replacing a Host 

In the case where a node in the cluster needs to be physically replaced with another node, it is important to preserve the original master-replica configuration of the forests, so that there is no performance burden on a single node hosting all the primary forests.

Example: replacing Host-B with a new Host-D

The steps listed below show how to replacing a node (old Host-B with new Host-D) without affecting the failover configuration:

  1. Shut down Host B and make sure forest failover successful - Forests c-4r & a-3r are "open" (acting masters).
  2. Add Host D as a node to the cluster; 
  3. Create new replica forests (d-1r and d-5r) on Host D and make them replicas of the corresponding primary forests on Host A & C. 
  4. Create new primary forests 'd-3p' and 'd-4p' on Host D  (These will replace b-3p and b-4p); 
  5. Break replication between a-1p and b-1r, and between c-5p and b-5r by updating the forest configuration for the primary forests.
  6. Take forest level backup of the failed over forests ('forest a-3r' and 'forest c-4r')
  7. Restore the backups from step 6 to the new primary forests 'forest d-3p' and 'forest d-4p' on Host D
  8. Attach forests 'forest d-3p' and 'forest d-4p' to the database and make forests 'forest a-3r' and 'forest c-4r'  their replicas.

This will replace Host B with Host D, as well as preserve the previously existing primary-replica configuration among the hosts.

Host A Host D Host C
forest a-1p forest a-2p forest a-3r
forest d-3p forest d-4p forest d-1r
forest c-5p forest c-6p forest c-2r
forest a-6r forest d-5r forest c-4r

Additional Notes

It is important to make sure that the database is quiesced before taking the forest backups. The idea is to disallow ingestion/updates on the database: One technique is to quiesce a database by making all of its forests 'read-only' - during the process and revert once complete.

Note: This example assumes a distributed master-replica configuration of a 3-node cluster. However, the same procedure works with other configurations with some careful attention to the number of forests on each host and breaking replication between the right set of hosts.







(3 vote(s))
Not helpful

Comments (0)