Introduction

MarkLogic uses two different Certificates for different objective. 1) Host certificate, and 2) AppServer Certificate

Host Certificates are used for inter cluster communication between nodes. Certificates are stored locally only individual nodes hosts.xml file. While most other configuration files are common and have same content on all nodes, hosts.xml file having local certificate are specific to that node only and have different certificate on each node.

App Certificate through Certificate Template used by Application Server running on specific application port.  are different than SSL certificates configured for app server,SSL certificates are configured for secure connection between server and client.

There is a third intra cluster certificates that are used mainly for communication between a cluster and foreign cluster, which are stored in clusters.xml. All nodes in cluster have common clusters.xml file and hence common certificate to communicate with foreign cluster.  This cluster certificates can be replaced using 'admin:cluster-set-xdqp-ssl-certificate' api. 

Issue

MarkLogic installation comes with Host certificate valid for 10 years. MarkLogic currently does not have any Admin GUI based path to replace host certificate which has expired after 10 years, which results in inter cluster XDQP SSL communication failure when Certificate expires after 10 years.

Solution

Admin can generate host certificate through query in Query console and replace certificate and key in individual node hosts.xml file.

Objective is to generate a new public and private key for a node. Private key is stored in server.xml and public key is stored in hosts.xml. Private key stored in server.xml is specific to that node, whereas hosts.xml that has public key stores the cert information of all the hosts in that cluster.

Admin can generate the certificate for host by running a query with host-id for which certificate is expired, which would give us both Private and Public key for that specific host. Private key is copied to server.xml of a node whereas public key needs to be updated on hosts.xml of all nodes in a cluster for that particular host-id. 

Steps to generate host certificate and private key file.

1. If cert needs to be generated for Node A, then replace host-id for the Node A at <host-id of your node>. This query will generate public and private key for Node A.

xquery version "1.0-ml";
import module namespace pki = "http://marklogic.com/xdmp/pki" at "/MarkLogic/pki.xqy";
declare namespace x509="http://marklogic.com/xdmp/x509";

let $host-id as xs:unsignedLong := <host-id of your node>

let $keys as xs:string* := 
  xdmp:rsa-generate(
    <options xmlns="ssl:options">
      <key-length>2048</key-length>
    </options>)

let $cert :=
  xdmp:x509-certificate-generate(
    element x509:cert {
      element x509:version {2},
      element x509:serialNumber {pki:integer-to-hex(xdmp:random())},
      element x509:issuer {
        element x509:commonName {$host-id}
      },
      element x509:validity {
        element x509:notBefore {fn:current-dateTime()},
        element x509:notAfter {fn:current-dateTime() + xs:dayTimeDuration("P3650D")}
      },
      element x509:subject {
        element x509:commonName {$host-id}
      },
      element x509:publicKey {$keys[2]},
      element x509:v3ext {
        element x509:basicConstraints {
          attribute critical {"false"},
          "CA:FALSE"
        }
      }
    },
    $keys[1]
  )

return ($cert, $keys)

2. Stop the service on all nodes.

3. Copy the generated private key to server.xml of Node A (Default location: /var/opt/MarkLogic/server.xml) on the that specific node and save under tag - <ssl-private-key>. server.xml is a node specific file

4. Copy the public key that was generated to the hosts.xml of Node A (Default location: /var/opt/MarkLogic/hosts.xml)and save it under the <host><ssl-certificate> for our specific host-id. hosts.xml is cluster wide common file, hence Admin will need to copy updated hosts.xml file on all nodes in cluster when changing certificate for Node A.

5. Repeat above steps for all 7 Nodes requiring new certificate

6. Start MarkLogic service for all nodes and enable XDQP SSL.

Further Reading

• admin:cluster-set-xdqp-ssl-certificate
• https://docs.marklogic.com/xdmp:x509-certificate-generate
• MarkLogic Host ID