Introduction

Rolling upgrades are used to upgrade a large cluster with many hosts to a newer version of MarkLogic Server without incurring any downtime in availability or interruption of transactions. 
This article acts as a supplement to our Rolling upgrade documentation. It discusses the preconditions and assumptions that our feature documentation makes for a successful no-downtime Rolling Upgrade. It also makes a few suggestions to plan the overall approach.

Assumptions:

There are a few basic assumptions that our Rolling upgrade documentation makes, and they include ALL of the following:

1. as suggested in the feature documentation, 'The security database and the schemas database must be on the same host, and that host should be the first host you upgrade when upgrading a cluster.'
2. fast failover works, and it should be in the order of seconds. Use a xdmp:shutdown with the failover flag set to true.
   
3. the MarkLogic node, taken down as part of Rolling Upgrade, does not have any in-flight transactions
4. The load balancer should have all the requests drained for the node going down and it should not send any more requests to that node.
   
5. all failed transactions will automatically retry and should succeed as soon as fast failover is complete.

Suggestions and approaches:

1. To avoid breaking open network connections while taking a node down for service during a Rolling upgrade, you must drain your requests from the load balancer before you shut down a particular MarkLogic node. Redirecting all new requests to the remaining available nodes allows you not to lose connections. Most modern-day load balancers (such as F5) should be able to perform such kinds of operations. So include these steps in your overall plan as there may be a need to trigger them manually. So that, while a Rolling upgrade is underway, your Load Balancer is accepting incoming requests and routing them to healthy instances only.

2. While taking down a node, for a faster failover, use xdmp:shutdown with the failover flag set to true. In the case of REST manage call for the node, use the 'failover=true' URI parameter for faster failover.