|
XML Data Query Protocol (XDQP)
24 October 2023 02:58 PM
|
|
SummaryMarkLogic uses a proprietary protocol for internal communication between nodes in a cluster. This article is describes the XML Data Query Protocol (XDQP) and related configuration settings. XDQP Version:Each MarkLogic Server release communicates with at a specific XDQP version. The following XDQP version mismatch warning may be logged if there is a node in your cluster that is running at a different MarkLogic Server version and those instances run at different incompatible XDQP versions.
In MarkLogic 8 and earlier versions, this is an error condition that must be corrected. To learn more about the XDQP version mismatch, please refer to our KB article 'What is XDMP-XDQPVER, and what should be done about it? ' XDQP Connections:XDQP is Application protocol that runs at the TCP Layer, using Port 7999 and 7998. Port 7999 is used for Heartbeat check as well as E-node/D-node data communication. Port 7998 is mainly used for communication between clusters for Replication. This article mainly will focus on port 7999 XDQP communication. XDQP Connections are created when MarkLogic Server starts up and, barring any exception condition, remains persistent until the Server is shutdown. During steady state cluster operations, no XDQP connections are being opened or closed. During startup, For each node in cluster, MarkLogic Server will create 3 connections to other Node's on their port 7999. Symmetrically other node will also create 3 connections to first Node's 7999 port. Therefore, each node in the cluster will have 6x(N-1) connections created for Cluster with N number nodes. Multiple connection channels (3) for XDQP port 7999 are intended for Performance reasons and number of channels are NOT configurable. MarkLogic Server sends heartbeat message to each node in the cluster every second (NOT configurable). The heartbeat message synchronizes all servers to the same clock (transaction timestamp), keeps a consistent state of the 'quorum', propagates configuration changes, and may carry Query Data. If the heartbeat messages are interrupted for an extended period of time, a host may become disconnected from the cluster (Host Timeout). Heartbeat messages are cycled across all 3 connection channels; Since MarkLogic sends heartbeat messages to each specific node every second, we would expect to see traffic at least once on each connection channel every 3 seconds. Heartbeats from other hosts are dropped if clock skew is detected beyond acceptable limit, causing hosts to disconnect.
New connections are also rejected if clock skew beyond acceptable limit are detected (host timeout). Other Host is declared down if the problem persists with new connect.
XDQP Maximum Connections:MarkLogic will not handle incoming XDQP Connection request from more then 255 peer nodes, which effectively makes the MarkLogic cluster upper limit 256 nodes. Connections are made as follows: - all nodes in a cluster will connect to each other - bootstrap nodes will connect to bootstrap nodes in another cluster when the clusters are connected - d-nodes in one cluster will also connect to d-nodes in another cluster when their forests are connected for database replication - in addition, bootstrap nodes will be connected to all the d-nodes in a connected cluster when those d-nodes are involved in database replication If a node reaches the maximum connections it will throw an XDMP-XDQPMAX error. Timeouts Configuration ItemsXDQP Timeout:Is a Group level configuration item that "specifies the time, in seconds, before a request between a MarkLogic Server evaluator node (the node from which the query is issued) and a MarkLogic Server data node (the node from which the forest data is retrieved) times out." A timeout on the XDQP channel is at the application level, meaning XDQP timeout does NOT necessarily close the connection channel underneath for fist XDQP timeout, like you would imagine for an HTTP Timeout. MarkLogic will deem XDQP connection to be broken : (1) If a Send operation takes longer than XDQP timeout the XDQP connection. (2) If no traffic has been Received from another host over XDQP for two times the XDQP timeout (default XDQP timeout 10 seconds x 2 = 20 Seconds). (3) If there is clock skew greater then Host timeout between Hosts. Once XDQP connection to a host is deemed broke, MarkLogic will throw Retry Exception and trigger XDQP Restart, closing and re-opening all 3 XDQP connections to a host. An XDQP Restart for a given host will cause all running requests accessing that host to be retried. XDQP Restart does not affect 3 connections channel *from* other host on local port 7999, as they are separate group. Generally, An XDQP Restart fixes a lot of networking issues and avoids Host timeout. Host timeout is a lot severe, as it will removes other host from Cluster based on Quorum and will trigger failover. By default XDQP timeout are set to 10 seconds. To learn more about issues that may affect how you configure the XDQP timeout, refer to our KB article 'How to handle XDQP-TIMEOUT on a busy cluster' Host Timeout:Is a Group level configuration item that "specifies the time, in seconds, before a MarkLogic Server host-to-host request times out. The host-to-host requests are used for communication between nodes in a MarkLogic Server cluster." Host timeout pertains to how long MarkLogic node can go without seeing any response from peer node/host on the XDQP channel, before MarkLogic closes the connection. Other then non responding host, Network issue, Host timeout also limits the acceptable clock skew between nodes, not allowing clock skew to be greater then Host timeout. By default, the host timeout is set to 3 times the XDQP Timeout = 30 seconds. The reason for default value is to allow XDQP connection issues to be resolved and fixed before a host is judged offline, triggering failover. If there is a networking problem it usually can be resolved with an XDQP restart, preventing a failover. Host Initial timeoutIs a Group level configuration item that "specifies the time, in seconds, that an instance of MarkLogic Server will wait for another node to come online when the cluster first starts up before deciding that the node is down, and initiating failover for any forests that are assigned to that offline host." Host Initial timeout pertains to establishment of initial 3 TCP channels on XDQP port 7999. By default, MarkLogic will try for 4 minutes (240 seconds) to establish a connection before giving up on a peer node. Further Reading
| |
 (5 vote(s)) |