Removing hosts from a MarkLogic cluster minimizing downtime
16 February 2021 03:32 PM
Removing Hosts From a MarkLogic Cluster Minimizing Downtime
This is a procedure to remove hosts from a MarkLogic cluster while minimizing unavailability. It is assumed that High availability is configured using local disk failover and all primary forests have at least one replica forest configured.
Typically when a host is removed from a cluster, it will trigger a restart of the cluster to apply the new configuration. In some environments, this could be burdensome if it is a large cluster and a number of hosts are being removed.
For hosts to be removed from the cluster, they must meet the following criteria:
We also recommend scheduling cluster maintenance during low usage periods.
Removing more than one host
Hosts can only be removed one at a time, but you can remove multiple hosts without restarting the cluster by using the Management API DELETE /admin/v1/host-config, with the remote-host parameter.
We recommend that the hosts being removed be offline, either by shutting down the MarkLogic service, or by shutting down host at the OS level, but this can also be done while a host is online.
State of removed hosts
When a host is removed using the remote-host parameter, the host information is removed from the cluster, but the host itself still retains the cluster configuration and will continue to attempt to connect to the cluster unsuccessfully. This means you may see errors in the logs until the MarkLogic service on the removed host is shutdown.
If the removed host will be added to another cluster, the existing MarkLogic configuration will need to be reset. This can be done by stopping the MarkLogic service, removing the contents of /var/opt/MarkLogic, and starting the MarkLogic service. The removed host will now be in an uninitialized state, and can be added to a new cluster.
Once the all the hosts have been removed from the cluster, we do recommend performing a restart of the cluster to ensure the configuration change has been fully committed.