Solutions

MarkLogic Data Hub Service

Fast data integration + improved data governance and security, with no infrastructure to buy or manage.

Learn More

Learn

Stay On Top Of Everything MarkLogic

Be the first to know! News, product information, and events delivered straight to your inbox.

Sign Me Up

Community

Stay On Top Of Everything MarkLogic

Be the first to know! News, product information, and events delivered straight to your inbox.

Sign Me Up

Company

Stay On Top Of Everything MarkLogic

Be the first to know! News, product information, and events delivered straight to your inbox.

Sign Me Up

 
Knowledgebase:
Recreating a Node into an Existing Cluster
26 June 2017 03:15 PM

Introduction

In some situations an existing cluster node needs to be replaced. There are multiple reasons for this activity like hardware failure or hardware replacement.

In this Knowledgebase article we will outline the steps necessary to replace the node by reusing the existing cluster configuration without registering it again.

Important notes:

  • The replacement node must have the same architecture as all other nodes of the cluster (e.g., Windows, Linux, Solaris). The CPUs must also have the same number of bits (e.g., 64, 32).
  • The replacement node must have the same (or higher) count of CPU cores
  • The replacement node must have the same (or higher) allocated disk space and mount points as the old node
  • The replacement node must have the same hostname as the old node

Preparation steps for re-joining a node into the cluster

  • Install and configure the operating system
    • make sure the mount points are matching the old setup
    • in case the previous storage is healthy it can be reused (forests located on it will be mounted)
  • For any non-MarkLogic data (such as XQuery modules, Deployment scripts etc.) required to run on this node, ensure these are manually zipped and copied over as part of the staging process
  • Copy over MarkLogic configuration files (/var/opt/MarkLogic/*.xml) from a backup of the old node

If you don't have your current MarkLogic version available for installation you can download the install binary using curl; this will also allow you to download older versions if required - for example: to download the binary for MarkLogic Server 6.0-4, you could use the following call:

curl -O -XPOST -d'email=you@company.com&pass=yoursecret' 'https://developer.marklogic.com/download/binaries/6.0/MarkLogic-6.0-4.x86_64.rpm'

Steps for installation of the new node to rejoin the cluster

There are two procedures to rejoin a host into the cluster, using an older set of configuration files or creating a new set from one of the other nodes in the cluster. Both are explained in the next sections:

Steps to rejoin by using older configuration files

This procedure can be only performed if an old configuration file set from /var/opt/MarkLogic/*.xml is available from the lost/old node other wise it will fail causes a lot of problems.

  • Perform a standard MarkLogic server installation on the new target node
    • rpm -Uvh /path/to/MarkLogic-<version>.x86_64.rpm
    • apply your changes to /etc/sysconfig/MarkLogic (optional)
    • do not start MarkLogic server
  • Create a new data directory
    • mkdir /var/opt/MarkLogic (default location; might exists if separate mount point)
    • verify permissions (default chown daemon:daemon /var/opt/MarkLogic)
  • Copy an old set of configuration files into the data directory
    • cp /path/to/old/config/*.xml /var/opt/MarkLogic
    • verify permissions (default chown daemon:daemon /var/opt/MarkLogic/*.xml)
  • Perform a last sanity check
    • hostname must be the same as the old node
    • firewall rules are correct
    • mountpoints and permissions are correct
  • Start MarkLogic
  • Monitor the start up process
    • by following ErrorLog.txt (tail -f /var/opt/MarkLogic/Logs/ErrorLog.txt)
    • by checking status in Admin-UI (http://<hostname>:8001)

After starting the node it will reuse the old configuration settings like the other node did. 

 

Steps to rejoin by creating a new set of configuration files from another cluster node

This procedure is required if there is no older configuration file set available. For example no file backup was made from /var/opt/MarkLogic/*.xml. It requires manual editing of a configuration file.  

  • Perform a standard MarkLogic server installation on the new target node
    • rpm -Uvh /path/to/MarkLogic-<version>.x86_64.rpm
    • apply your changes to /etc/sysconfig/MarkLogic (optional)
  • start MarkLogic
    • service MarkLogic start
  • Perform a normal server setup as single node. Do NOT join the cluster now 
    • perform only a basic setup
    • NOT joining the cluster is important!
  • stop MarkLogic
    • service stop MarkLogic
  • Move all current configuration files away
    • mv /var/opt/MarkLogic/*.xml /secure/place (using default settings; destination can be adjusted)
  • Copy a configuration files set from one of the other nodes over
    • scp <othernode>:/var/opt/MarkLogic/*.xml /var/opt/MarkLogic
    • verify permissions (default chown daemon:daemon /var/opt/MarkLogic/*.xml)
  • Edit copied server.xml [/var/opt/MArkLogic/server.xml] (this is important as the copied version matches the other node)
    • use an editor which is UTF-8 safe like nano or vi
    • update <license-key> with the value from /secure/place/server.xml (value from initial installation)
    • update <licensee> with the value from /secure/place/server.xml (value from initial installation)
    • update <host-id> with the value found in /var/opt/MarkLogic/hosts.xml (open hosts.xml that is copied from the other node, it contains multiple <host> nodes from all configured hosts of the cluster, search for the hostname of this node; the <host-id> is usually locate above)
  • After saving the file review that changes
    • important the <host-id> must match the <host> defined in hosts.xml otherwise the host won't start 
  • Perform a last sanity check
    • hostname must be the same like the old node
    • firewall rules are correct
    • mountpoints and permissions are correct
  • Start MarkLogic
  • Monitor the start up process
    • following ErrorLog.txt (tail -f /var/opt/MarkLogic/Logs/ErrorLog.txt)
    • check status in Admin-UI (http://<hostname>:8001)

The important step is to update the server.xml to change the <host-id> to match the value defined in hosts.xml and apply the right license information. Without this changes the node either won't start up or it will confuse the other nodes. 

After placing the right configuration files on the new node the startup of both prodecures is the same. It will reuse the configuration that was used by the old host and rejoin the cluster. Existing forests (if mounted or copied at the right location) will be mounted like before or automatically recreated in case they got lost. Any configured replication will start sync replicating the forests after the node is started. In case there is no replication setup and forests are recreated they can be restored by using the latest available backup.

Additional notes

This article explains how to directly replace a node in a cluster by using the same host name. Another way is to add a new node to the cluster and transfer the forests which is explained in the following knowledge base article "Replacing a D-Node with local disk failover".

Some of these steps may differ, such as operating system calls or file system locations. On a different OS, the specific commands will need to be adjusted to match the environment.

 

 

(5 vote(s))
Helpful
Not helpful

Comments (0)