Introduction

A common use case in many business applications is to find if an element exists in any document or not. This article provide ways to find such documents and explain points that should be taken care of while designing a solution.

Solution

In general, existence of an element in a document can checked by using below XQuery.

cts:element-query(xs:QName('myElement'),cts:and-query(()))

Note the empty cts:and-query construct here. An empty cts:and-query is used to fetch all fragments.

Hence running below search query will bring back all the documents having element "myElement".

Wrapping the query in cts:not-query will bring back all the documents *not* having element "myElement" 

As a search using cts:not-query is only guaranteed to be accurate if the underlying query that is being negated is accurate from its index resolution, hence to check existence of a specific XPath, we need to index that XPath.
e.g. if you want to find documents having /path/1/A (and not /path/2/A) then you can create a field index for path /path/1/A and then use it in your query instead.

Things to remember

1.) Have unique element name in a single document i.e. try not to use same element name at multiple places within a document if they have different meaning for your use case. Either give them different element names or put them under different namespaces to remove any ambiguity. e.g. if you have element "table" at two places in a single document then you can put them both under different namespaces such as html:table & furniture:table or you can name them differently such as html_table & furniture_table.

2.) If element names are unique within a document then you don't need to create additional indexes. If element names are not unique within a document and you are interested in only a specific XPath then create path(field) indexes on those XPaths and use the same in your not-query.

Introduction

MarkLogic Server has shipped with full support for the W3C XML Schema specification and schema validation capabilities since version 4.1 (released in 2009).

These features allow for the validation of complete XML documents or elements within documents against an existing XML Schema (or group of Schemas), whose purpose is to define the structure, content, and typing of elements within XML documents.

You can read more about the concepts behind XML Schemas and MarkLogic's support for schema based validation in our documentation:

https://docs.marklogic.com/guide/admin/schemas

Caching XML Schema data

In order to ensure the best possible performance at scale, all user created XML Schemas are cached in memory on each individual node within the cluster using a portion of that node's Expanded Tree Cache.

Best practices when making changes to pre-existing XML Schemas: clearing the Expanded Tree Cache

In some cases, when you are redeploying a revised XML Schema to an existing schema database, MarkLogic can sometimes refer to an older, cached version of the schema data associated with a given document.

Therefore, it's important to note that whenever you plan to deploy a new or revised version of a Schema that you maintain, as a best practice, it may be necessary to clear the cache in order to ensure that you have evicted all cached data stored for older versions of your schemas.

If you don't clear the cache, you may sometimes get references to the old, cached schema references and as result, you may get errors like:

XDMP-LEXVAL (...) Invalid lexical value

You can clear all data stored in the Expanded Tree Cache in two ways:

1. By restarting MarkLogic service on every host in the cluster. This will automatically clear the cache, but it may not be practical on production clusters.
2. By issuing a call to xdmp:expanded-tree-cache-clear() command on each host in the cluster. You can run the function in query console or via REST endpoint and you will need a user with admin rights to actually clear the cache.

An example script has been provided that demonstrates the use of XQuery to execute the call to clear the Expanded Tree Cache against each host in the cluster:

Please contact MarkLogic Support if you encounter any issues with this process.

Related KB articles and links:

• Performance implications of updating Module and Schema databases
• Best Practices for schema definition
• MarkLogic Blog: Using XML Schema with MarkLogic Server
• MarkLogic Blog: Schema Agnosticism: What It Is and Why You Should Care
• Techniques for dividing tasks between hosts in a cluster

Summary

XDMP-ODBCRCVMSGTOOBIG can occur when a non-ODBC process attempts to connect to an ODBC application server.  A couple of reasons that this can happen is that there is an http application that has been accidentally configured to point to the ODBC port, or a load balancer is sending http health checks to an ODBC port. There are a number of common error messages that can indicate whether this is the case.

Identifying Errors and Causes

One method of determining the cause of an XDMP-ODBCRCVMSGTOOBIG error is to take the size value and convert it to Characters.  For example, given the following error message:

2019-01-01 01:01:25.014 Error: ODBCConnectionTask::run: XDMP-ODBCRCVMSGTOOBIG size=1195725856, conn=10.0.0.101:8110-10.0.0.103:54736

The size, 1195725856, can be converted to the hexadecimal value 47 45 54 20, which can be converted to the ASCII value "GET ".  So what we see is a GET request being run against the ODBC application server.

Common Errors and Values

Conclusion

XDMP-ODBCRCVMSGTOOBIG errors, do not affect the operation of MarkLogic Server, but can cause error logs to fill up with clutter.  Determining that the errors are caused by an http request to an ODBC port can help to identify the root cause, so the issue can be resolved.

Summary

Meters data can be a good resource for getting an approximation of the number of requests being managed by the server at a given time. It's also important to understand how Meters data is generated, should there be a discrepancy between the Meters samples, and the entries in the access log.

Meters Request Data

The Meters data is designed to record a sampling of activity, every few seconds. Meters data is not designed to accurately record server request rates much lower than every few seconds. Request rates are 15-second moving averages, recalculated every second and available in real time through the xdmp:host-status, xdmp:server-status and xdmp:forest-status built-in functions.

Meters Samples

The metering subsystem samples these real-time rates on the minute and saves the samples in the Meters database. Meters sampled data of events that occur less frequently than the moving average period will be lower than the number of access log entries. The difference between the two will depend on when the last event happened and when the sample was taken.

This mean that if an event happens once a minute, the request rate will rise when an event happens, but then decay away within a few seconds. If the sample is taken after the event has decayed, the saved meters data will be lower than the actual number of requests

Conclusion

The result of the Meters sampling method means that it is not unusual for Meters to under report the number of requests in certain circumstances.

Summary

In MarkLogic Server v7.0-2, the tokenizer keys, for languages where MarkLogic provides generic language support, were removed so that they now all use the same key. For example, Greek falls into this class of languages. This change was made as part of an optimization for languages in which MarkLogic Server has advanced stemming and tokenization support.  

Stemmed searches that include characters from languages that do not have advanced language support, performed on MarkLogic Server v7.0-2 or later releases, against content loaded on a version previous to v7.0-2, may not return the expected results.

Resolution

In order to successfully run these stemmed searches, you can either:

• Reindexing the database ; or
• Reinsert the affected documents (i.e. the documents that contain characters in languages for which MarkLogic Server only has generic language support).

If these are not possible in your environment, you can always run the query unstemmed.

An Example

The following example demonstrates the issue

1. On MarkLogic Server version 7.0-1, insert a document (test.xml) that contains the Greek character 'ε'.
2. Run this query 
   xdmp:estimate( cts:search( doc('test.xml'), 'ε')),
cts:contains( doc('test.xml'), 'ε')
3. The query will return the correct results: 1, true
4. Upgrade MarkLogic Server to version 7.0-3 or later and run the query again
5. The query will return incorrect results: 0, false 
6. Reindex the database and re-run the query
7. The query will return the correct result once again.
    

Introduction

HAProxy (http://www.haproxy.org/) is a free, fast and reliable solution offering high availability, load balancing and proxying for TCP and HTTP-based applications.

MarkLogic 8 (8.0-8 and above) and MarkLogic 9 (9.0-4 and above) include improvements to allow you to use HAProxy to connect to MarkLogic Server.

MarkLogic Server supports balancing application requests using both the HAProxy TCP and HTTP balancing modes depending on the transaction mode being used by the MarkLogic application as detailed below:

1. For single-statement auto-commit transactions running on MarkLogic version 8.0.7 and earlier or MarkLogic version 9.0.3 and earlier, only TCP mode balancing is supported. This is due to the fact that the SessionID cookie and transaction id (txid) are only generated as part of a multi-statement transaction.
2. For multi-statement transactions or for single-statement auto-commit transactions running on MarkLogic version 8.0.8 and later or MarkLogic version 9.0.4 and later both TCP and HTTP balancing modes can be configured.

The Understanding Transactions in MarkLogic Server and Single vs. Multi-statement Transactions in the MarkLogic documentation should be referenced to determine whether your application is using single or multi-statement transactions.

Note: Attempting to use HAProxy in HTTP mode with Single-statement transactions prior to MarkLogic versions 8.0.8 or 9.0.4 can lead to unpredictable results.

Example configurations

The following example configurations detail only the parameters relevant to enabling load balancing of a MarkLogic application, for details of all parameters that can be used please refer to the HAProxy documentation.

TCP mode balancing

The following configuration is an example of how to balance requests to a 3-node MarkLogic application using the "roundrobin" balance algorithm based on the source IP address. The health of each node is checked by a TCP probe to the application server every 1 second.

backend app
mode tcp
balance roundrobin
stick-table type ip size 200k expire 30m
stick on src
default-server inter 1s
server app1 ml-node-1:8012 check id 1
server app2 ml-node-2:8012 check id 2
server app3 ml-node-3:8012 check id 3

HTTP mode balancing

The following configuration is an example of how to balance requests to a 3-node MarkLogic application using the "roundrobin" balance algorithm based on the "SessionID" cookie inserted by the MarkLogic server.

The health of each node is checked by issuing an HTTP GET request to the MarkLogic health check port and checking for the "Healthy" response.

backend app
mode http
balance roundrobin
cookie SessionID prefix nocache
option httpchk GET / HTTP/1.1\r\nHost:\ monitoring\r\nConnection:\ close
http-check expect string Healthy
server app1 ml-node-1:8012 check port 7997 cookie app1
server app2 ml-node-2:8012 check port 7997 cookie app2
server app3 ml-node-3:8012 check port 7997 cookie app3

Summary

MarkLogic Server organizes Trusted Certificate Authorities (CA) by Organization Name.  Trusted Certificate Authorities are the issuers of digital certificates, which in turn are used to certify the public key on behalf of the named subject as given in the certificate.  These certificates are used in the authentication process by:

1. A MarkLogic Application Server configured to use SSL (HTTPS).
2. Any Web Client which is making a connection to a MarkLogic Application Server over HTTPS (in the case of SSL Client Authentication).

Example Scenarios

Consider the following example:

$openssl x509 -in CA.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 18345409437988140316 (0xfe97fcaf8a61b51c)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=CA, L=San Carlos, O=MarkLogic Corporation, OU=Engineering, CN=MarkLogic CA
        Validity
            Not Before: Nov 30 04:08:31 2015 GMT
            Not After : Nov 29 04:08:31 2020 GMT
        Subject: C=US, ST=CA, L=San Carlos, O=MarkLogic Corporation, OU=Engineering, CN=MarkLogic CA

In this example, From viewing the Trusted CA Subject field, the CA Certificate name will be listed with the organisation name of "MarkLogic Corporation" (O=MarkLogic Corporation) in MarkLogic's list of Certificate Authorities.

You can view the full list of currently configured Trusted Certificate Authorities by logging into the MarkLogic administration Application Server (on port 8001) and viewing the status page: Configure -> Security -> Certificate Authorities

Trusted CA Certificate without Organization name (O=)

In some cases, there are legitimate Trusted CA Certificates which do not contain any further information about the Organization responsible for the certificate.

The example below shows a sample self signed root CA (DemoLab CA) which highlights this scenario:

$openssl x509 -in DemoLabCA.pem  -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12836463831212471403 (0xb22447d80f91b46b)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=DemoLab CA
        Validity
            Not Before: Nov 30 05:23:13 2015 GMT
            Not After : Nov 29 05:23:13 2020 GMT
        Subject: CN=DemoLab CA

If this Certificate were to be loaded into the MarkLogic, no name would appear under the list of <em>Certificate Authorities</em>in the list provided through the administration Application Server at Configure -> Security -> Certificate Authorities

In the case of the above example, it would be difficult to use the certificate validated by DemoLab CA (and to use DemoLab CA as our Trusted Certificate Authority) as MarkLogic will only list certificates that are associated with an Organization.

Solution

To workaround this issue, we can configure MarkLogic to use the certificate through some scripting with Query Console.

1) Loading the CA using Query Console

Start by using a call to pki:insert-trusted-certificates to load the Trusted CA into MarkLogic.  The sample Query Console code below demonstrates this process (Please ensure this query is executed against the Security database)

Make a note of value of the id returned by MarkLogic. It will return an unsigned long (xs:unsignedLong) which is the id value that can be used later to retrieve that certificate

2) Attach Trusted CA with "SSL Client Certificate Authorities" using Query Console

The next step is to associate the certificate that we just inserted from our filesystem (DemoLabCA.pem) with a given MarkLogic Application Server. Once this is done, any client connecting to that application server over SSL will be presented with the cerificate and DemoLab CA will be used to match the certificate using the Common Name value (Common Name eq "DemoLab CA")

3) Verify attached Trusted CA for Client Cetificate Authorities

Executing the above code should return the same identifier (for the Trusted CA) as returned as result of the code executed in step 1. Additionally, we can see that our Application Server (DemoAppServer) is now configured to expect an SSL Client Certificate Authority signed by DemoLab CA.

Further Reading

• Definition of a CA (on Wikipedia)
• Definition of a digital certificate (on Wikipedia)
• MarkLogic API documentation for pki:insert-trusted-certificates
• MarkLogic API documentation for admin:appserver-get-ssl-client-certificate-authorities
• MarkLogic API documentation for admin:appserver-set-ssl-client-certificate-authorities

Introduction

MarkLogic Server is engineered to scale out horizontally by easily adding forests and nodes. Be aware, however, that when adding resources horizontally, you may also be introducing additional demand on the underlying resources.

Details

On a single node, you will see some performance improvement in adding additional forests, due to increased parallelization. This is a point of diminishing returns, though, where the number of forests can overwhelm the available resources such as CPU, RAM, or I/O bandwidth. Internal MarkLogic research (as of April 2014) shows the sweet spot to be around six forests per host (assuming modern hardware). Note that there is a hard limit of 1024 primary forests per database, and it is a general recommendation that the total number of forests should not grow beyond 1024 per cluster.

At cluster level, you should see performance improvements in adding additional hosts, but attention should be paid to any potentially shared resources. For example, since resources such as CPU, RAM, and I/O bandwidth would now be split across multiple nodes, overall performance is likely to decrease if additional nodes are provisioned virtually on a single underlying server. Similarly, when adding additional nodes to the same underlying SAN storage, you'll want to pay careful attention to making sure there's enough I/O bandwidth to accommodate the number of nodes you want to connect.

More generally, additional capacity above a bottleneck generally exacerbates performance issues. If you find your performance has actually decreased after horizontally scaling out some part of your stack, it is likely that a part of your infrastructure below the part at which you made changes is being overwhelmed by the additional demand introduced by the added capacity.

Summary

MarkLogic Application Servers will keep a connection open after completing and responding to a request, waiting for another new request, until the Keep Alive timeout expires. However, there is an exception scenario where the connection will close regardless of timeout settings when the content is larger then 1 MB. This article is intended to provide further insight into connection close with respect to Payload size.

HTTP Header

Connection-Length

In general, Application Servers communicating in HTTP send the Content-Length header as part of their response HTTP Headers to indicate how many bytes of data the client application should expect to receive. For example

HTTP/1.1 200 OK

Content-type: application/sparql-results+json; charset=UTF-8

Server: MarkLogic

Content-Length: 1264

Connection: Keep-Alive

Keep-Alive: timeout=5

This requires Application Servers to know the length of the entire response data before the very first bytes (Response HTTP Headers) are put on to the wire. For small amounts of data, the time to calculate the content-length is fast; For large amounts of content, the calculation may be time consuming with the extreme being that the client finds the server unresponsive due to the delay in calculating the entire response length. Additionally, the server may need to bring the entire content into Memory Buffer, putting further burden on server resources.

Chunked-encoding

To allow servers to begin transmitting dynamically-generated content before knowing the total size of that content, HTTP 1.1 supports chunked encoding. This technique is widely used in music & video streaming and other industries. Chunked encoding eliminates the need of knowing the entire content length before sending a portion of the data, thus making the server looks more responsive.

At the time of this writing, MarkLogic Server (v8.0-6 and earlier releases) does not support chunked encoding. However, do look for this feature in future releases of MarkLogic Server.

Connection Close

In MarkLogic Server v7 and v8, MarkLogic Server closes the connection after transmitting content greater 1MB, which allows MarkLogic to avoid calculating content length in advance. The client will not see Content-Length Header for Larger (>1MB) content in HTTP Response from MarkLogic. Instead it will receive a Connection Close header in HTTP Response. After sending the entire content, MarkLogic Server will terminate the connection, to indicate to Client that the end of content has been reached.

Closing the existing connection for content larger then 1MB is an exception to the Keep-Alive configuration. This may result in unexpected behavior on clients that relying on MarkLogic Server respecting the Keep-Alive configuration, so this behavior should be accounted while designing Client Application Connection Pool.

Client Applications may have to send TCP SYN again to establish new connection to send subsequent request, which will add overhead of TCP 3 way handshake before sending next request. However, in the context of the data transfer for larger payload (>1MB), where many more round trips are added in overall communication, overhead of TCP 3 way handshake is very nominal.

Further Reading

• Chunked transfer encoding (Wikipedia)
• KB - "Notes on Using the Java Client API efficiently"

Summary

CSV files are a very common data exchange format. It is often used as an export format for spreadsheets, databases or any other application. Depending on the application, you might be able to change the delimiter character to a #hash or *asterix etc. One of the default delimiter definitions is a tab character. Content Pump supports reading and loading such CSV files.

Detail

The Content Pump -delimiter option defines which delimiter will be used to split the columns. Defining a tab as a value for the delimiter option on the command line isn't straight forward.

Loading tab delimited data files with content pump can result in an error massage like the following:

mlcp>bin/mlcp.sh IMPORT -host localhost -port 9000 -username admin -password secret -input_file_path sample.csv -input_file_type delimited_text -delimiter '    ' -mode local
13/08/21 15:10:20 ERROR contentpump.ContentPump: Error parsing command arguments: 
13/08/21 15:10:20 ERROR contentpump.ContentPump: Missing argument for option: delimiter
usage: IMPORT [-aggregate_record_element <QName>]
... 

Depending on the command line shell, a tab needs to be escaped to be understand from the shell script: 

On bash shell, this should work: -delimiter $'\t'
On Bourne shell, this should work: -delimiter 'Ctrl+V followed by tab' 
Alternative way would be to use: -delimiter \x09 

If none of these work, another approach you can try is to use the -options_file /path/to/options-file parameter. The options file can contains all of the same parameters as the command line does. The benefit of using an option file is that the command line is simpler and characters are interpreted as intended. The options file will contain multiple lines where the first line is always the action like IMPORT,  EXPORT etc. followed by a pair of lines. The first line is the option parameter and second the value for the option.

A sample could look like the following:

Make sure the file is saved in UTF-8 format to avoid any parsing problems. To define a tab as delimiter, place a real tab between single quotes (i.e. '<tab>')

To use this option file with mlcp execute the following command:

Linux, Mac, Solaris:

mlcp>bin/mlcp.sh -options_file /path/to/sample.options

Windows:

mlcp>bin/mlcp.bat -options_file /path/to/sample.options

The options file can take any paramter which mlcp understands. It is important that the action command is defined on the first line. It is also possible to use both command line parameters and the option file. Command line parameters take precedence over those defined in the options file.

Introduction:

MarkLogic Server allows you to set-up an alerting application to notify users when new content is available that matches a predefined query. This can be achieved through the Alerting API with the Content Processing Framework (CPF). CPF is designed to keep state for documents, so it is easy to use CPF to keep track of when a document in a particular scope is created or updated, and then perform some action on that document. However, although alerting works for document updates and inserting, it does not occur for document deletes. You will have to create a custom CPF pipeline to catch the delete through an appropriate status transition.

Details

To achieve alerting for document delete, you will have to write your own custom pipeline with status transition to handle deletes. For example:

The higher 'priority' value and 'always' = true indicates that the custom pipeline has precedence over the default status change handling pipeline to handle document deletes.  Similarly, in the action module, you can write your custom code for alerting.

Note: By default, when a document is deleted, the on-delete pre-commit trigger is fired and it calls the action in the Status Change Handling pipeline (if enabled) for ‘delete’ status transition. It is recommended that you do not modify this pipeline as it can cause compatibility problems in future upgrades and releases of MarkLogic server.

Introduction

If you're looking at the MarkLogic Admin UI on port 8001, you may have noticed that the status page for a given database displays the last backup dateTime for a given database.

We have been asked in the past how this gets computed so the same check can be performed using your own code.

This Knowledgebase article will show examples that utilise XQuery to get this information and will explore the possibility of retrieving this using the MarkLogic ReST API

XQuery: How does the code work?

The simple answer is in the forest status for each of the forests in the database (note these values only appear if you have created a backup already).  For the sake of these examples, let's say we have a database (called "test") which contains 12 forests (test-1 to test-12).  We can get the backup status for these using a call to our ReST API:

http://localhost:8002/manage/LATEST/forests/test-1?view=status&format=html

In the results returned, you should see something like this:

last-backup : 2016-02-12T12:30:39.916Z datetime
last-incr-backup : 2016-02-12T12:37:29.085Z datetime

In generating that status page, what the MarkLogic code does is to create an aggregate: a database doesn't contain documents in MarkLogic; it contains forests and those forests contain documents.

Continuing the example above (with a database called "test" containing 12 forests) if I run the following:

This will return the forest status(es) for all forests in the database "test" and return the forest names using XPath, so in this case, we would see:

<forest-name xmlns="http://marklogic.com/xdmp/status/forest">test-1</forest-name>
[...]
<forest-name xmlns="http://marklogic.com/xdmp/status/forest">test-12</forest-name>

Our admin UI is interrogating each forest in turn for that database and finding out the metrics for the last backup.  So to put that into context, if we ran the following:

This gives us:

<last-backup xmlns="http://marklogic.com/xdmp/status/forest">2016-02-12T12:30:39.946Z</last-backup>
[...]
<last-backup xmlns="http://marklogic.com/xdmp/status/forest">2016-02-12T12:30:39.925Z</last-backup>

The code (or the status report) doesn't want values for all 12 forests, it just wants the time the last forest completed the backup (because that's the real time the backup completed), so our code is running a call to fn:max:

Which gives us the max value (as these are all xs:dateTimes, it's finding the most recent date), which in the case of this example is:

2016-02-12T12:30:39.993Z

The same is true for the last incremental backup (note all that we're changing here is the XPath to get to the correct element:

So we can get the max value for this by getting the most recent time across all forests:

This would give us 2016-02-12T12:37:29.161Z

Using the ReST API

The ReST API also allows you to get this information but you'd need to jump through a few hoops to get to it; the ReST API status for a given database would give you the names of all the forests attached to that database:

http://localhost:8002/manage/LATEST/databases/test

And from there you could GET the information for all of those forests:

http://localhost:8002/manage/LATEST/forests/test-1?view=status&format=html
[...]
http://localhost:8002/manage/LATEST/forests/test-12?view=status&format=html

Once you'd got all those values, you could do what MarkLogic's admin code does and get the max values for them - although at this stage, it might make more sense to write a custom endpoint that returns this information, something like:

Where you could make a call to that module to get the aggregates (e.g.):

http://[server]:[port]/[modulename.xqy]?db=test

This would return the database status for any given parameter-name that is passed in.

Problem:

When searching for matches using OR'ed word-queries, and in the case where there are overlapping matches, (i.e. one query contains the text of another query), the results of a cts:highlight query are not as desired.

For example:

 The desired outcome of this would be:

               <p>From the <m>memoirs of an accomplished artist</m> </p>

 Whereas, the actual results are:

                <p>From the <m>memoirs of an </m> <m>accomplished artist</m></p>

This behavior is by design and the results are expected. It is because cts:highlight  breaks up overlapping  areas into separate matches.

The cts:highlight built-in variables – $cts:queries and $cts:action help in understanding how this works, as well as to work-around this problem.

  $cts:queries --> returns the matching queries for each of the matched texts.

  $cts:action --> can be used with xdmp:set to specify what should happen next

• "continue" - (default) Walk the next match. If there are no more matches, return all evaluation results.
• "skip" - Skip walking any more matches and return all evaluation results
• "break" - Stop walking matches and return all evaluation results

   For eg., replacing the return statement with the following in the original query:

These results give us a better understanding of how the text is being matched. We can see that " accomplished artist" is matched by both the word-queries 'accomplished artist' and 'memoirs of an accomplished artist'; hence the results of cts:highlight seem different.

To work around this problem, we can insert a small piece of code: 

Please note that this solution relies on assumptions about what's inside the or-query, but this example could be modified to handle other overlapping situations.

Introduction

In the Scalability, Availabilty & Failover Guide, the node communication section describes a quorum as >50% of the nodes in a cluster.

Is it possible for a database to be available for reads and writes, even if a quorum of nodes is not available in the cluster?

The answer is yes, there are configurations and sequences of events that can lead to forests remaining online when there are fewer than 50% of the hosts being online.

Details

If a single forest in a database is not available, the database is not be accessible. It is also true that as long as all of a database's forests are available in the cluster, the database will be available for reads and writes regardless of any quorum issues.

Of course, the Security database must also be available in the cluster for the cluster to function.

Forest Availability: Simple Case

In the simplest case, if you have a forest that is not configured with either local disk failover or shared disk failover and as long as the forest's host is online and exists in the cluster, the forest will be available regardless of any quorum issues.

To explain this case in more detail: if we have a 3-node MarkLogic cluster containing 3 hosts (let's call them host-a, host-b and host-c); if we were to then initialize host-a as the primary host (so this is the first host is set up in the cluster and is the host containing the master security database) and we then join host-b and host-c to host-a to complete the cluster. 

Shortly after that, if we shut both the joiner hosts (host-b and host-c) down, so only host host-a remained online, we would see a chain of messages in the primary host's ErrorLog that indicated there was no longer quorum within the cluster:

2020-05-21 01:19:14.632 Info: Detected quorum (3 online, 1 suspect, 0 offline)
2020-05-21 01:19:18.570 Warning: Detected suspect quorum (3 online, 2 suspect, 0 offline)
2020-05-21 01:19:29.715 Info: Disconnecting from domestic host host-b.example.marklogic.com because it has not responded for 30 seconds.
2020-05-21 01:19:29.715 Info: Disconnected from domestic host host-b.example.marklogic.com
2020-05-21 01:19:29.715 Info: Detected suspect quorum (2 online, 1 suspect, 1 offline)
2020-05-21 01:19:33.668 Info: Disconnecting from domestic host host-c.example.marklogic.com because it has not responded for 30 seconds.
2020-05-21 01:19:33.668 Info: Disconnected from domestic host host-c.example.marklogic.com
2020-05-21 01:19:33.668 Warning: Detected no quorum (1 online, 0 suspect, 2 offline)

Under these circumstances, we would be able to access the host's admin GUI on port 8001 and it would respond without issue.  We would be able to access Query Console on that host on port 8000 and would be able to inspect the primary host's databases.  We would also be able to access the Monitoring History on port 8002 - all directly from the primary host.

In this scenario, because the primary host remains online and the joining hosts are offline; and because we have not yet set up failover anywhere, there is no requirement for quorum, so host-a remains accessible.

If host-a also happened to have a database with forests that only resided on that host, these would be available for queries at this time.  However, this is a fairly limited use case because in general, if you have a 3-node cluster, you would have a database whose forests reside on all three hosts in the cluster with failover forests configured on alternating hosts. 

As soon as you do this, if you lose one host and you don't have failover configured, the database would now become unavailable (due to a crucial forest being offline) and if you had failover forests configured, you would still be able to access the database on the remaining two hosts.

However, if you then shut down another host, you would lose quorum (which is a requirement for failover).

Forest Availability: Local Disk Failover

For forests configured for local disk failover, the sequence of events is important:

In response to a host failure that makes an "open" forest inaccessible, the forest will failover to the configured forest replica as long as a quorum exists and the configured replica forest was in the "sync replicating" state. In this case, the configured replica forest will transition to the "open" state; the configured replica forest becomes the acting master forest and is available to the database for both reads and writes.

Additionally, an "open" forest will not go offline in response to another host being evicted from the cluster.

However, once cluster quorum is lost, forest failovers will no longer occur.

Conclusion

Depending on how your forests are distributed in the cluster and depending of the order of host failures, it is possible that a database can remain online even when there is no longer a quorum of hosts in the cluster.

Of course, databases with many forests spread across many hosts typically can't stay online if you lose quorum because some forest(s) will become unavailable.

Recommendation

Even though it is possible to have a functioning cluster with less than a quorum of hosts online, you should not architect your high availability solution to depend on it.

Summary

This article discusses what happens when you backup or restore your database after a local disk failover event on one of the database forests.

Introduction

MarkLogic Server provides high availability in the event of a data node failure. Data node failures can include operating system crashes, MarkLogic Server restarts, power failures, or persistent system failures; for example hardware failures. With Forest level failover enabled and configured, a machine that hosts a forest can go down and the MarkLogic Server cluster automatically recovers from the outage and keep continuing to process queries without any immediate action needed by an administrator. In MarkLogic Server, if a forest becomes unavailable then the entire database to which this forest is attached becomes unavailable for further query operations. Without failover, such a failure requires a manual intervention (such as administrator) to either reconfigure the forest to another host or to remove this forest from the configuration (cluster). With failover, you can configure the forest to automatically switch to a replica forest on a different host. MarkLogic Server Failover provides for high availability and maintains data and transactional integrity in the event of a data node failure.

The failover scenarios are well documented on our developer web site.

Local Disk Failover

You to configure a forest on another host to serve as a replica forest which will take over when a primary master forest's host goes offline. Local-disk failover allows you to create one or more replica forests for each primary forest. Replica forests contain the exact same data as the primary forest and are kept consistent transactionally. 

It is helpful to use the following terms to refer to the forest configurations and states:

• Configured Master is the forest which is originally configured as the primary forest.
• Configured Replica is a forest on another host that is configured as a replica forest of the primary. 
• Acting Master is the forest that is server as the master forest, regardless of the configuration.
• Acting Replica is the forest that is server as the replica forest, regardless of the configuration.

Database Backup when a forest is failed over

If you attempt to take a Database back up or perform a database restore when One of the forests of the database had failed over to the replica (i.e. Configured Replica is serving as Acting Master), it may result in XDMP-FORESTNOTOPEN or XDMP-HOSTDOWN errors.

When a database backup takes place, by default, everything associated with database gets backed up. You can also choose to backup any individual forests (only the forests selected while configuring backup are backed up). T

Replica Forest will only be backed up when the 'Include replica forests' are enabled.  If you have not configured the backup to include replica forests, then the replica forests will not be backed up even if it is the acting master. If the Configured Master is also not available, then neither forest will be backed up. In this circumstance, you may see a message in the error logs similar to "Warning: Not backing up database test because first forest master is not available, and replica backups aren't enabled."

Restore when a forest is failed over

Restore's will fail if executed when a forest is failed over (i.e. Configured Replica is serving as Acting Master). In this circumstance, you may see a message in the error logs similar to "Operation failed with error message. Check server logs." or "XDMP:HOSTDOWN".

How to detect if a forest is failed over

In the Admin UI:

1. Click the Forests icon in the left tree menu;
2. Click the Summary tab;
3. You see the configured replica in open state; (This indicates that the Configured Replica is serving as Acting Master).

At the time of the failover event, you may see messages in the Error Log similar to:
2013-10-03 12:49:53.873 Info: Disconnecting from domestic host rh6v-intel64-9.marklogic.com in cluster 16599165797432706248 because it has not responded for 30 seconds.
2013-10-03 12:49:53.873 Info: Disconnected from host rh6v-intel64-9.marklogic.com
2013-10-03 12:49:53.873 Info: Unmounted forest test_P
2013-10-03 12:49:53.875 Info: Forest test_R assuming the role of master with new precise time 13808297938747190
2013-10-03 12:49:53.875 Debug: Recovering undo on forest test_R
2013-10-03 12:49:53.875 Debug: Recovered undo at endTimestamp 13807844927734200 minQueryTimestamp 0 on forest test_R

Revert back from the failover state:

When the configured master is the acting replica, this is considered the "failover state".  In order to revert back, you must either restart the acting master forest or restart the host in which the acting master forest is locally mounted. After restarting, the forest will automatically revert to Configured Master if it's host is online. To check the status of the forests, see the Forests Summary tab in the Admin Interface. 

Conclusion 

For backup and restore to work correctly, clusters configured with local disk failover must have no forests in a failed over state. If a cluster is configured with local disk failover, and if some of its forests are failed over to their local disk replicas, the conditions causing the fail over must be resolved, and the cluster must be returned to the original forest configuration before backup and restore operations may resume.

INTRODUCTION

From the documentation:

Queries on a Replica database must run at a timestamp that lags the current cluster commit timestamp due to replication lag. Each forest in a Replica database maintains a special timestamp, called a Non-blocking Timestamp, that indicates the most current time at which it has complete state to answer a query. As the Replica forest receives journal frames from its Master, it acknowledges receipt of each frame and advances its nonblocking timestamp to ensure that queries on the local Replica run at an appropriate timestamp. Replication lag is the difference between the current time on the Master and the time at which the oldest unacknowledged journal frame was queued to be sent to the Replica.

To read more:

http://tinyurl.com/7zwq4l2

SCENARIO

Consider the following customer scenario:

• The storage the database resides on at one site fails.
• This requires the customer to run for a period of time on a single site.
• The storage / MarkLogic server are recovered at the site where the failure occurred.
• The customer needs to re-establish replication between the two sites

QUESTIONS AND ANSWERS

Q: Should we tune the lag limit to suit our application?

A: We have found in our own performance testing that increasing the lag limit beyond the default is typically not helpful.

When the master has a sustained rate of updates, a large lag limit causes it to run quickly ahead of the replica, then stall for an extended period of time until the replica catches up. This pattern repeats over and over and gives inconsistent performance on the master.

A smaller lag limit causes the master to suspend updates more frequently but for shorter periods of time, resulting in more consistent perceived performance.

Q: Is there any option to restore the replica database to a point in time from a backup of the master database & re-initiate replication from that point onwards?

A: It's fine to restore a backup to the failed system when it comes back online and before configuring replication in the reverse direction.

Q: Is there a limit to how old a backup of the replica database can be (e.g. can a replica be restored from months back in comparison to the master) and will it still sync back to the master without issue? And does this depend on what journal data is available?

A: There is no limit to how old a backup can be; the system will calculate all the deltas and apply them.

Q: Are there any documented API built-ins for any of these things?

A: Indeed; all the replication information is available through a call to xdmp:forest-status()

xdmp:forest-status( 
  xdmp:database-forests( 
    xdmp:database("MyDatabase"), 
    fn:true()))

For further information:

http://tinyurl.com/d6vbpk4

Q: Can you also advise if the replication lag limit mentioned in section 1.2.5 and the related possibility of transactions stalling on the master database applies during the bulk replication phase?

A: As long as the replica's forests are in "open replica" state, the replica will respond to queries at any commit timestamp it is able to support irrespective of whether replication is lagged.

A new feature in MarkLogic 5 is an application server setting for multi-version concurrency control (by default this is set to contemporaneous - meaning it will run from the latest timestamp that any query has committed - irrespective of whether there are still transactions in-flight).

Conversely, if nonblocking is chosen (i.e. if you create an application server to query a replica database and you set multi-version concurrency control to nonblocking), the server will choose the last timestamp where all pending transactions are known to have successfully committed.

If you wish to evaluate a query against a replica database you can use xdmp:database-nonblocking-timestamp() to determine the most current query timestamp that will not block.

Introduction

Database Replication replicates fragments/documents from a source database to a target database. You may see different database sizes (even when active fragment counts are then same) between Master and Replica Databases. This article provides overview of variables and reasons behind such observation.

Database Replication:

Database Replication operates at the forest level by copying journal frames from a forest in the Master database and replaying them on a corresponding forest in the foreign replica database. In other words, this means that when Journal frames are replayed in the replica database, the same group of documents in a single stand of the master database, does not necessarily reside in the same stand on the replica database - i.e. the distribution of fragments within stands are different between the master and replicas. 

Also, Note that Master and Replica forests can be distributed differently across hosts in each cluster. Even when they are distributed identically (Master DB forest name to Replica DB forest name) you could still see a different number stand between them.

Database Size, Deleted Fragment and Merge:

Current Database Size depends on number of factors like number of documents, index, deleted fragments in Stand etc. Deleted Fragments in any stand itself depends on Merge Policy, Background Merge process, Processing Cycle available, Linux Memory Config, Memory Usage at any given time, and application usage pattern.

Conclusion:

Master Cluster and Replica Cluster are separate entities. Although connected, they operate independently. Replica Database on target cluster provides data consistency. However how data can be spread across different stands than the corresponding master, including the retention of deleted fragments, will differ between Master and Replica Cluster. Hence you may see different sizes between Master and Replica Databases, even where the active fragments are the same.

Further Reading

• Database Replication in MarkLogic Server
• KB - Rebalancing, replication and forest reordering
• KB - Disaster Recovery
• KB - Unable to Merge all Deleted Fragments

Introduction

If your MarkLogic Server has it's logging level set to "Debug", it's common to see a chain of 'Detecting' and 'Detected' messages that look like this in your ErrorLogs:

2015-01-27 11:11:04.407 Debug: Detected indexes for database Documents: ss, fp, fcs, fds, few, fep, sln
2015-01-27 11:11:04.407 Debug: Detecting compatibility for database Documents
2015-01-27 11:11:04.407 Debug: Detected compatibility for database Documents

This message will appear immediately after forests are unmounted and subsequently remounted by MarkLogic Server.

What would cause the forests to be unmounted and remounted

• Heavy network activity leading to a cluster (XDQP) "Heartbeat" timeout
• Changes made to forest configuration or indexes
• Any incident that may cause a "Hung" message

What are "Hung" messages?

Whenever you see a "Hung" message it's very often indicative of a loss of connection to the IO subsystem (especially the case when forests are mounted on network attached storage rather than local disk). Hung messages are explained in a little more detail in this Knowledgebase article:
https://help.marklogic.com/Knowledgebase/Article/View/35/0/hung-messages-in-the-errorlog

What do the "Detected" messages mean and what can I do about them?

Whenever you see a group of "Detecting" messages:

2015-01-14 13:06:26.016 Debug: Detecting indexes for database XYZ

There was an event where MarkLogic chose to (or was required to) attempt to unmount and remount forests (and the event may also be evident in your ErrorLogs).

The detecting index message will occur soon after a remount, indicating that MarkLogic Server is examining forest data to check whether any reindexing work is required for all databases available to the node which have Forests attached:

2015-01-14 13:06:26.687 Debug: Detected indexes for database XYZ: ss, wp, fp, fcs, fds, ewp, evp, few, fep

The line immediately below indicates that the scan has been completed and the database has been identified as having been configured with a number of indexes. For the line above, these are:

ss

stemmed searches

wp

word positions

fp

fast phrase searches

fcs

fast case sensitive searches

fds

fast diacritic sensitive searches

ewp

element word positions

evp

element value positions

few

fast element word searches

fep

fast element phrase searches

From this list, we are able to determine which indexes were detected.  These messages will occur after every remount if you have index detection set to automatic in the database configuration.

Every time the forest is remounted, in addition to a recovery process (where the Journals are scanned to ensure that all transactions logged were safely committed to on-disk stands), there are a number of other tests the server will do. These are configured with three options at database level:

• format compatibility
• index detection
• expunge locks

By default, these three settings are configured with the "automatic" setting (in MarkLogic 7), so if you have logging set to "Debug" level, you'll know that these options are being worked through on remount:

2015-01-14 13:06:26.016 Debug: Detecting indexes for database XYZ (represents the task for "automatic" index detection where the reindexer checks for configuration changes)
2015-01-14 13:06:26.687 Debug: Detecting compatibility for database XYZ (represents the task for "automatic" format compatibility where the on-disk stand format is detected)

These default values may change in accross releases of MarkLogic Server. In MarkLogic 8, expunge locks is set to none but the other two are still set to automatic.

Can these values be changed safely and what happens if I change these?

Unmounting / remounting times can be made much shorter by configuring these settings away from automatic but there are some caveats involved; if you need to upgrade to a future release of the product, it's likely that the on-disk stand format may change (it's still 5.0 even when MarkLogic 8 is released) and so setting format compatibility to 5.0 should cause the "Detecting compatibility" messages to disappear and speed up remount times.

The same is true for disabling index detection but it's important to note that changing index settings on the database will no longer cause the reindexer to perform any checks on remount; in this case you would need to enable this for changes to database index settings to be reindexed.

Summary

This article will provide steps to debug applications using the Alerting API that are not triggering an alert.

Details

1) Check that all required components are present in the database where alerting is setup: config, actions, rules.   Run the attached script 'getalertconfigs.xqy' through the Query Console and review the output.  

2) As documented in our Search Developer's Guide, Test the alert manually with alert:invoke-matching-actions(). 

Example:

alert:invoke-matching-actions("my-alert-config-uri", 
      <doc>hello world</doc>, <options/>)

3) Use the rule's query to test against the database to check that the expected documents are returned by the query.

Take the query text from the rule and run it through Query Console using a cts:search() on the database.  This will confirm whether the expected documents are a positive match.  If the documents are returned and no alert is triggered, then further debugging will be needed on the configuration or the query may need to be modified.

Introduction 

Division operations involving integer or long datatypes may generate XDMP-DECOVRFLW in MarkLogic 7. This is the expected behavior but it may not be obvious upon initial inspection.  

For example, similar queries with similar but different input values executed in Query Console on Linux/Mac machine running MarkLogic 7 gives the following results

1. This query returns correct results

let $estimate := xs:unsignedLong("220")

let $total := xs:unsignedLong("1600")

return $estimate div $total * 100

==> 13.75

2. This query returns the XDMP-DECOVRFLOW Error

let $estimate := xs:unsignedLong("227")

let $total := xs:unsignedLong("1661")

return $estimate div $total * 100

==> ERROR : XDMP-DECOVRFLW: (err:FOAR0002)

Details

The following defines relevant behaviors in MarkLogic 7 and previous releases.

• In MarkLogic 7, if all the operands involved in div operations are integer, long or integer sub-types in XML, then the resulting value of the div operation are stored as xs:decimal.
• In versions previous to MarkLogic 7, if an xs:decimal value is large and occupies all digits then it was implicitly cast into an xs:double for further operations - i.e. beginning with MarkLogic, implict casting no longer occurs in this situation .
• xs:decimal can accomodate 18 digits as a datatype.
• In MarkLogic 7 on Linux & Mac, xs:decimal can occupy all digits depending upon actual value ( 227 div 1661 = 0.1366646598434677905 ), all 18 digits occupied in xs:decimal
• MarkLogic 7 on Windows does not perform division with full decimal precision ( 227 div 1661 produces 0.136664659843468 ); as a result, not all 18 digits occupied in xs:decimal
• MarkLogic 7 will generates Overflow Exception : FOAR0002, when an operation is performed on an xs:decimal that is already at full decimal precision

In the example above, multiplying the result with 100 gives an error in Linux/Mac, while its OK on Windows.

Recommendations:

We recommend xs:double be used for all division related operations in order to explicitly cast resulting value to larger data-type.

For example: These will return results

xs:double($estimate) div $total * 100


$estimate div $total * xs:double(100)

.

Context:

There are options 'maintain last modified' and 'maintain directory last modified' on the Admin UI for a database, which when turned on add properties to every document inserted in the database.  There may be a need to remove all the property fragments of all the documents in the database when the properties no longer need to be retained.

Problem:

Turning these options off for a database ensure that properties will not be created for new documents. However, existing document properties will not be removed by turning these settings off.

Solution:

To delete existing document properties, the following query can be used:

xdmp:node-delete(xdmp:document-properties(“your-document-uri”))

Please make sure that 'maintain last modified' and 'maintain directory last modified' options are turned off for the database, so that the property fragment does not get recreated for the document.

Summary

Terraform from HashiCorp is a deployment tool that many organizations use to manage their infrastructure as code. It is platform agnostic, allowing for the deployment and configuration of on-site physical infrastructure, as well as cloud infrastructure such as AWS, Azure, VSphere and more.

Terraform uses the Hashicorp Configuration Language (HCL) to allow for concise descriptions of infrastructure. HCL is JSON compatible language, and was designed to be both human and machine friendly.

This powerful tool can be used to deploy a MarkLogic Cluster to AWS using the MarkLogic CloudFormation Template. The MarkLogic CloudFormation Template is the preferred method recommended by MarkLogic for building out MarkLogic clusters within AWS.

Setting Up Terraform

For the purpose of this example, I will assume that you have already installed Terraform, the AWS CLI and you have configured the credentials. You will also need to have a working directory that has been initialized using terraform init.

Terraform Providers

Terraform uses Providers to provide access to different resources. The Provider is responsible for understanding API interactions and exposing resources. The AWS Provider is used to provide access to AWS resources.

Terraform Resources

Resources are the most important part of the Terraform language. Resource blocks describe one or more infrastructure objects, like compute instances and virtual networks.

The aws_cloudformation_stack resource, allows Terraform to create a stack from a CloudFormation template.

Choosing a Template

MarkLogic provides two templates for creating a managed cluster in AWS.

• MarkLogic cluster in new VPC
• MarkLogic cluster in an existing VPC

Defining Variables

The MarkLogic CF Template takes a number of input variables, including the region, availability zones, instance types, EC2 keys, encryption keys, licenses and more. We have to define our variables so they can be used as part of the resource.

Variables in HCL can be declared in a separate file, which allows for deployment flexibility. For instance, you can create a Development resource and a Production resource, but using different variable files.

Here is a snippet from our variables file:

variable "cloudform_resource_name" {
type = string
default = "Dev-Cluster-CF"
}
variable "stack_name" {
type = string
default = "Dev-Cluster"
}
variable "ml_version" {
type = string
default = "10.0-4"
}
variable "availability_zone_names" {
type = list(string)
default = ["us-east-1a","us-east-1b","us-east-1c"]
}
...

In the snippet above, you'll notice that we've defined the availability_zone_names as a list. The MarkLogic CloudFormation template won't take a list as an input, so later we will join the list items into a string for the template to use.

This also applies to any of the other lists defined in the variable files.

Using the CloudFormation Resource

So now we need to define the resource in HCL, that will allow us to deploy a CloudFormation template to create a new stack.

The first thing we need to do, is tell Terraform which provider we will be using, defining some default options:

    provider "aws" {
    profile = "default"
    #access_key = var.access_key
    secret_key = var.secret_key
    region = var.aws_region
    }

Next, we need to define the `aws_cloudformation_stack` configuration options, setting the variables that will be passed in when the stack is created:

    resource "aws_cloudformation_stack" "marklogic" {
    name = var.cloudform_resource_name
    capabilities = ["CAPABILITY_IAM"]


    parameters = {
    IAMRole = var.iam_role
    AdminUser = var.ml_admin_user
    AdminPass = var.ml_admin_password
    Licensee = "My User - Development"
    LicenseKey = "B581-REST-OF-LICENSE-KEY"
    VolumeSize = var.volume_size
    VolumeType = var.volume_type
    VolumeEncryption = var.volume_encryption
    VolumeEncryptionKey = var.volume_encryption_key
    InstanceType = var.instance_type
    SpotPrice = var.spot_price
    KeyName = var.secret_key
    AZ = join(",","${var.avail_zone}")
    LogSNS = var.log_sns
    NumberOfZones = var.number_of_zones
    NodesPerZone = var.nodes_per_zone
    VPC = var.vpc_id
    PublicSubnets = join(",","${var.public_subnets}")
    PrivateSubnets = join(",","${var.private_subnets}")
    }
    template_url = "${var.template_base_url}${var.ml_version}/${var.template_file_name}"
    }

Deploying the Cluster

Now that we have defined our variables and our resources, it's time for the actual deployment.

$> terraform apply

This will show us the work that Terraform is going to attempt to perform, along with the settings that have been defined so far.

Once we confirm that things look correct, we can go ahead and apply the resource.

Now we can check the AWS Console to see our stack

And we can also use the ELB to login to the Admin UI

Wrapping Up

We have now deployed a 3 node cluster to an existing VPC using Terraform. The cluster is now ready to have our Data Hub, or other application installed.

Introduction

This KB article lists some available tools for continuous integration and automatically deploying the MarkLogic Server

Deployment

Roxy is an open source utility for configuring and deploying MarkLogic applications. Using Roxy you can define your app servers, databases, forests, groups, tasks, etc in local configuration files. Roxy can then remotely create, update, and remove those settings from the command line.
https://github.com/marklogic/roxy

MarkLogic Admin API and REST API can be used to script server configurations
https://docs.marklogic.com/guide/admin-api
https://docs.marklogic.com/REST/management

MLSound is an open source tool for deploying and bootstraping projects into MarkLogic 8.
https://github.com/miguelrgonzalez/mlsound

Integration Testing

Roxy includes a unit testing component that allows tests to be written in XQuery and run from the UI or from the command line.

Another useful open source unit testing tool that is available for writing XQuery unit tests is ‘xray’ - https://github.com/robwhitby/xray

Implementation Specific Tools

There are also implementation specific tools to help with check-in build/deployment:

Jenkins
Jenkins is often used with MarkLogic Server for building deployable artifacts, staging build artifacts, running automated tests, and deploying said artifacts. Jenkins has great REST endpoints that make it easy to get / put job configurations, and enable / disable jobs from scripts.

Jenkins provides a driver to the continuous integration / continuous delivery process that can integrate with other tools. In combination with Roxy, it can be used to run a bootstrap/deploy module/unit test on code check-in.

One pipeline example used with Jenkins is to:
1) Pull from Git
2) Deploy to DEV with Roxy
3) Run xray unit testing and Sonar vulnerability scans
4) Email a report of the success/failure
5) Kick off job to deploy to another environment

Also noted that the most important best practice here would be to make sure Jenkins runs primarily off of a host other than a MarkLogic host.

SUMMARY

This article will help MarkLogic Administrators to monitor the health of their MarkLogic cluster. By studying the attached scripts, you will learn how to find out which hosts are down and which forests have failed over, enabling you to take the necessary recovery actions.

Initial Setup

On a separate Linux host (not a member of the cluster), download the file attachments from this article, making sure that they all reside within the same directory.

Here is a general description of each file:

cluster-name.conf - Example configuration file used by script. Configures information for monitoring one ML cluster. 

ml-ck-for-life.sh - A very simple, low-load check that all the nodes of a cluster are up and running.

ml-ck-for-health.sh - A more detailed check for essential cluster functionality with alerting (paging and/or emails to DBAs) if warranted. This script relies on at least one external XQuery file (mon-report-failed-over-forests.xqy) and makes use of the REST MGMT API as well as REST XQuery requests.

mon-report-failed-over-forests.xqy - External XQuery file used by ml-ck-for-health.sh

Preparing the CONF File for Use on Your Cluster

Before running the scripts, the cluster-name.conf needs to be customized for your specific cluster. Start by changing the file name to match the name of your cluster, e.g.,

$ mv cluster-name.conf some-other-name.conf

Where "some-other-name" is the actual name of the cluster, or of the application that is hosted on that cluster.

Next, you will need to customize some of the internal variables inside the CONF file itself. Here is the contents of the cluster-name.conf file, as downloaded:

CLUSTER_NAME="CLUSTER-NAME"
CLUSTER_NODES=( node1.my-company.com node2.my-company.com node3.my-company.com )
# MarkLogic Credentials for the REST Management port - 8002
USER_PW_MGMT=rest-manager-user:re-manager-password
# MarkLogic Credentials for the XQuery eval port - 8000
USER_PW_XQ=user-name:user-password
UNIX_USER=unix-user-name
PAGE_ADDRESSES=ml.alert.page@my-company.com
MAIL_ADDRESSES=ml.alert.mail@my-company.com

---------  end of listing ---------

For CLUSTER_NAME, provide the cluster-name listed in the cluster's /var/log/MarkLogic/clusters.xml file.

For CLUSTER_NODES, write in the host-names for each node in your cluster.

For USER_PW_MGMT, provide the user-name and password for the REST MANAGEMENT user, the format is name:password.

For USER_PW_XQ, provide the user-name and password for the user who will execute the XQuery scripts, the format is name:password.

The UNIX_USER is a local Unix username with the correct rwx access rights for this directory.

The PAGE_ADDRESSES & MAIL_ADDRESSES are alert email addresses who will be notified whenever there is a failover event.

Periodicity

The script ml-ck-for-health.sh was created with the idea it would be run repeatedly at a certain interval to keep tabs on system health. For example, it can be configured to be invoked with a cron job. A frequency of 5 to 120 minutes is a good candidate range. Ten minutes is a good time if you would like to be woken up (on average) within 5 minutes of a failover event.

Setting up SSH Passwordless Login

In monitoring script ml-ck-for-health.sh, section (6) FOREST STATUS CHANGE, requires ssh access to the cluster hosts. That is because this section greps through MarkLogic server ErrorLogs. To enable this part of the script to run without prompting the user, "ssh passwordless login" should be setup between the monitoring host and all the cluster hosts.There are many examples of how to do this on the internet, for example: http://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/ Alternatively, this monitoring section can be commented out.

Also regarding section (6), the “grep” command is setup up to grep the latest 10 minutes from the ErrorLog. If this script is configured to be run less often then every 10 minutes, the “grep” command line should be adapted to cover the desired period between script runs.

Example Usage

You are now ready to execute the failover monitoring scripts! Here is how you would execute them:

$ ./ml-ck-for-health.sh some-other-name.conf MY-CLUSTER-NAME

$ ./ml-ck-for-life.sh some-other-name.conf

[where "some-other-name" and MY-CLUSTER-NAME are your actual CONF and cluster-name, as described above]

Monitoring Multiple Clusters

So, given a monitoring machine with a directory of cluster configuration files in the style of cluster-name.conf, those configuration files could be iterated through to monitor a suite of clusters from a single monitoring machine. It should be fairly easy to build a custom shell script to iterate through various cluster CONF files.

Final thought and Limitations

Please be aware that the ml-ck-for-health.sh script is only partially implemented. In particular, the Replication Lag and Replication Failure sections are left as exercises for the user.

This script is being presented as a backup, lowest common denominator monitoring solution. For a more complete solution, you should explore other options, such as Splunk or Nagios.

Summary:

After adding or removing a forest and correspond replica forest in a database, we have seen instances where the Rebalancer does not properly distribute the documents amongst existing and newly added forests.

For this particular instance, XDMP-HASHLOCKINGRETRY debug level error message reported repeatedly in the error logs.  The messages would look something like: 

2016-02-11 18:22:54.044 Debug: Retrying HTTPRequestTask::handleXDBCRequest 83 because XDMP-HASHLOCKINGRETRY: Retry hash locking. Forests config hash does not match.

2016-02-11 18:22:54.198 Debug: Retrying ForestRebalancerTask::run P_initial_p2_01 50 because XDMP-HASHLOCKINGRETRY: Retry hash locking. Forests config hash does not match.

Diagnosing

Gather statistics about the rebalancer to see the number of documents being scheduled. If you run attached script “rebalancer-preview.xqy” in the query console of your MarkLogic Server cluster, it will produce rebalancer statistics in tabular format.

• Note that you must first change the database name (YourDatabaseNameOnWhichNewForestsHaveBeenAdded) on the 3^rd line of the XQuery script “rebalancer-preview.xqy”:

declare variable $DATABASE as xs:string := xdmp:get-request-field("db", "YourDatabaseNameOnWhichNewForestsHaveBeenAdded");

If experiencing this issue, the newly added forests will show zero in the “Total to be moved” column in the generated html page.

Resolving

Perform a cluster wide restart in order to get past this issue.  The restart is required to reload all of the configuration files across the cluster.  The rebalancer will also check to see if additional rebalancing work needs to occur. The rebalancer should work as expected now and the  XDMP-HASHLOCKINGRETRY messages should no longer appear in the logs. If you run the rebalancer-preview.xqy script again, the statistics should now show the the number of documents being scheduled to be moved.

You can also validate the rebalancer status from the Database Status page in the Admin UI.

The XDMP-HASHLOCKINGRETRY rebalancer issue has fixed in the latest MarkLogic Server releases.  However, the rebalancer-preview.xqy script can be used to help diagnose other perceived issues with the Rebalancer.

Search fundamentals

Difference between cts:contains and fn:contains

 1) fn:contains is a substring match, where as cts:contains performs query matching

 2) cts:contains therefore can utilize general queries and stemming, where fn:contains does not

For example:-

Example.xml

<test>daily running makes you fit</test>

•         fn:contains(fn:doc(“Example.xml”),”ning”)

          True

•          cts:contains(fn:doc(“Example.xml”),”ning”)

         False

•         fn:contains(fn:doc(“Example.xml”),”ran”)

           False

•         cts:contains(fn:doc(“Example.xml”),”ran”)

            True

Note:-

The cts:contains examples are checking the document against cts:word-querys.  Stemming reduces words down to their root, allowing for smaller term lists.

1) Words from different languages are treated differently, and will not stem to the same root word entry from another language.

2) Note: Nouns will not stem to verbs and vice versa. For example, the word “runner” will not stem to “run”.

References

• fn:contains
• cts:contains

Introduction

MarkLogic Server provides a variety of  disaster recovery (DR) facilities including full backup, incremental backup, and journal archiving that when combined with other ML features can create a complete disaster recovery strategy. This paper shows some examples of how these features can be combined. It is not comprehensive nor does it reflect features offered only in the latest releases.

Details

This article will cover three perspectives. First, a quick overview of the metrics used by businesses to measure the quality of their Disaster Recovery strategies will be covered. Next, an overview of how to combine the features that MarkLogic offers in various categories will be given.

More?: High Availability and Disaster Recovery features ,  High Availability & Disaster Recovery datasheet, Scalability, Availability, and Failover Guide 

Disaster Recovery Criteria

In order to configure MarkLogic Server to perform well in Disaster Recovery situations, we should first define what parameters we will use to measure each possible approach. For most situations, these four measures are used: 

Long Term Retention Policy (LTR): Long Term Retention Policy can be driven by any number of business, regulatory and other criteria. It is included here because MarkLogic's backup files are often a key part of an LTR strategy. 

Recovery Point Objective (RPO): The requirement for how up-to-date the database has to be post-recovery with respect to its state immediately before the incident that required recover.

Recovery Time Objective (RTO): The requirement for the time elapsed between the incident and the recovery to the RPO.

Cost: The storage cost, the computational resource cost and  the operations cost of the overall deployment strategy.

Flexible Replication Features

Flexible replication can be used to support LTR objectives but is generally not useful for Disaster Recovery

More? Flexible Replication Guide

Platform Support Features

Flash backup provides a way to leverage backup features of your deployment platform while maintaining transaction integrity. Platform specific solutions can often achieve RPO and RTO targets that would be impossible through other means.

More? Flash Backup

High Availability Features

Forest replication provides recovery from host failures.

More? Scalability, Availability, and Failover Guide

Disaster Recovery Features

Database Replication

Database Replication is the process of maintaining copies of forests on databases in multiple MarkLogic Server clusters.

More? Understanding Database Replication

Backups

Of all your backup options, full backups restore the quickest, but take the most time to backup and possibly the most storage space. Each full backup is a backup set in that it contains everything you need to restore to the point of the backup.

Full backups with journal archiving allow restores to a point after the backup, but the journal archive grows in an unbounded way with the number of transactions, and replaying the journals to get to your recovery point takes time proportional to the number of transactions in the journal archive, so over time, this becomes less efficient.

With full + incremental backups, a backup set is a full backup, plus the incremental backups taken after that full backup. Incremental backups are quick to backup, but take longer to restore, and over time the backup set gets larger and larger, so it may end up consuming more backup space than a full backup alone (depending on your backup retention policy).

Full + incremental backups with journal archiving have the same characteristics as incremental backups, except that you can roll forward from the most recent incremental. With this strategy, the journal archive doesn't grow in an unbounded way because the archive is purged when you take the next incremental backup. Note that if your RPO is between incremental backups, you must also enable a merge timestamp by setting the merge timestamp to a negative value (see below).

More?: Administrator’s Guide to Backing Up and Restoring a Database  How does "point-in-time" recovery work with Journal Archiving? 

Forest Merge Configurations

Forest merges recover the disk space occupied by deleted documents. A negative merge timestamp delays that permanent deletion. If we want incremental backups to contain all the fragments that were deleted since the last incremental backup then we want to set the delay to a period greater than the incremental backup period. This requires more disk space for the incremental backups and also requires additional space in the live database, but provides the most flexibility.

Setting retain-until-backup on a given database (thru the Admin UI or thru an API call) has a similar effect by telling the server to keep the deleted fragments until a full backup or an incremental backup completes. Many clients choose to use both the negative merge timestamp and retain until backup options together.

More?: admin:database-set-merge-timestamp  admin:database-set-retain-until-backup

Other Features

The need for a negative merge timestamp can be understood by remembering that forest merges recover the disk space occupied by deleted documents. A negative merge timestamp delays that permanent deletion. If we want incremental backups to contain all the fragments that were deleted since the last incremental backup then we want to set the delay to a period greater than the incremental backup period. This requires more disk space for the incremental backups and also requires additional space in the live database, but provides the most flexibility.

Setting retain-until-backup on a given database (thru the Admin UI or thru an API call) has a similar effect by telling the server to  keep the deleted fragments until a full backup or an incremental backup. Many clients choose to use both the negative merge timestamp and retain until backup options together.

More?: admin:database-set-merge-timestamp,  admin:database-set-retain-until-backup 

Conclusion

Planning to meet a Long Term Retention (LTR) policy, a Recovery Point Objective (RPO) and a Recovery Time Objective (RTO) and a Cost goal is a key part of developing an overall MarkLogic deployment plan. MarkLogic offers a wealth of tools that can complement each other when they are properly coordinated. As is clear from this article, the choices are many, broad, and interrelated.

Introduction

In the more recent versions of MarkLogic Server, there are checks in place to prevent the loading of invalid documents (such as documents with multiple root nodes).  However, documents loaded in earlier versions of MarkLogic Server can now result in duplicate URI or duplicate document errors being reported.

Additionally, under normal operating conditions, a document/URI is saved in a single forest. If somehow the load process gets compromised, then user may see issues like duplicate URI (i.e. same URI in different forests) and duplicate documents (i.e. same document/URI in same forest).

Resolution

If the XDMP-DBDUPURI (duplicate URI) error is encountered, refer to our KB article "Handling XDMP-DBDUPURI errors" for procedures to resolve.

If one doesn't see XDMP-DBDUPURI errors but running fn:doc() on a document returns multiple nodes then it could be a case of duplicate document in same forest. To check that the problem is actually duplicate documents, one can either do an xdmp:describe(fn:doc(...)) or fn:count(fn:doc((...)). If these commands return more than 1 e.g. xdmp:describe(fn:doc("/testdoc.xml")) returns (fn:doc("/testdoc.xml"), fn:doc("/testdoc.xml")) or fn:count(fn:doc("/testdoc.xml")) returns 2 then the problem is of duplicate documents in the same forest (and not duplicate URIs).

To fix duplicate documents, the document will need to be reloaded.

Introduction

This article talks about effects of case sensitivity of search term on search score and thus on final order of search results for a secondary query which is using cts:boost-query and weight. The case-insensitive word term is treated as the lower case word term, so there can be no difference in the frequencies and scores of results for any-case/case-insensitive search term and lowercase search term with “case-sensitive” option or when neither "case-sensitive" nor "case-insensitive" is present. If neither "case-sensitive" nor "case-insensitive" is present, text of search term is used to determine case sensitivity.

Understanding relevance score

In MarkLogic Search results are returned in a relevance order. The most relevant results are first in result sequence and least relevant are last.
More details on relevance score and its calculation are available at, https://docs.marklogic.com/guide/search-dev/relevance

Of many ways to control this relevance score one way is to use a secondary query to boost relevance score, https://docs.marklogic.com/guide/search-dev/relevance#id_30927 . This article takes advantage of examples using secondary query to boost relevance scores and impact of text case (upper, lower or unspecifed) of search terms on relevance score on order of results returned.

A few examples to understand this scenario

Consider a few scenarios where below mentioned queries are trying to boost certain search results up using cts:boost-query and weight for word "washington" in returned results.

Example 1: Search with lowercase search term and option for case not specified

Query1:
xquery version "1.0-ml";
declare namespace html = "http://www.w3.org/1999/xhtml";

for $hit in
( cts:search(
fn:doc()/test,

cts:boost-query(cts:element-word-query(xs:QName("test"),"George" ),
cts:element-word-query(xs:QName("test"),"washington",(), 10.0) )
)
)

return element hit {
attribute score { cts:score($hit) },
attribute fit { cts:fitness($hit) },
attribute conf { cts:confidence($hit) },
$hit
}

Results for Query1:
<hit score="28276" fit="0.9393904" conf="0.2769644">
<test>Washington, George... </test>
</hit>
...
...
<hit score="16268" fit="0.7125317" conf="0.2100787">
<test>George washington was the first President of the United States of America...</test>
</hit>
...

Example 2: Search with lowercase search term and case-sensitive option

Query2:
xquery version "1.0-ml";
declare namespace html = "http://www.w3.org/1999/xhtml";

for $hit in
( cts:search(
fn:doc()/test,

cts:boost-query(cts:element-word-query(xs:QName("test"),"George" ),
cts:element-word-query(xs:QName("test"),"washington",("case-sensitive"), 10.0) )
)
)

return element hit {
attribute score { cts:score($hit) },
attribute fit { cts:fitness($hit) },
attribute conf { cts:confidence($hit) },
$hit
}

Results for Query2:
<hit score="28276" fit="0.9393904" conf="0.2769644">
<test>Washington, George... </test>
</hit>
...
...
<hit score="16268" fit="0.7125317" conf="0.2100787">
<test>George washington was the first President of the United States of America...</test>
</hit>
...

Example 3: Search with uppercase search term and option case-insensitive, in cts:boost-query like below with rest of query similar to above queries

Query3:

cts:boost-query(cts:element-word-query(xs:QName("test"),"George" ),
cts:element-word-query(xs:QName("test"),"Washington",("case-insensitive"), 10.0) )

Results for Query3:
<hit score="28276" fit="0.9393904" conf="0.2769644">
<test>Washington, George... </test>
</hit>
...
...
<hit score="16268" fit="0.7125317" conf="0.2100787">
<test>George washington was the first President of the United States of America...</test>
</hit>
...

Clearly above queries are producing the same scores with same fitness and confidence scores. This is because the case-insensitive word term is treated as the lower case word term, so there can therefore be no difference in the frequencies of those two terms (any-case/case-insensitive and lowercase/case-sensitive), and therefore no difference in scoring. Thus no difference in scores of results for Query3 and Query2.
And for cases where case sensitivity is not specified, text of search term is used to determine case sensitivity. For Query3 text of search term contains no uppercase hence it treated as "case-insensitive".

Now let us now take look at a query with a word with uppercase and case-sensitive option in query.

Example 4: Search with uppercase search term and option case-sensitive, in cts:boost-query like below with rest of query similar to above queries

Query4:

cts:boost-query(cts:element-word-query(xs:QName("test"),"George" ),
cts:element-word-query(xs:QName("test"),"Washington",("case-sensitive"), 10.0) )

Results for Query4:
<hit score="44893" fit="0.9172696" conf="0.3489831">
<test>Washington, George was the first... </test>
</hit>
...
...
<hit score="256" fit="0.0692672" conf="0.0263533">
<test>George washington was the first President of the United States of America...</test>
</hit>
...

As we can clearly see the scores are changed for results for Query4 and thus final order of results is also updated.

Conclusion:

While using a secondary query having cts:boost-query and weight, to boost certain search results up, it is important to understand the impact of case of search text on result sequence. A case-insensitive word term is treated as the lower case word term, so there can therefore be no difference in the frequencies of any-case/case-insensitive and lowercase/case-sensitive search terms, and therefore no difference in scoring. For search term with upper case alphabets in text and with “case-sensitive” option scores are boosted up as expected in comparison with a “case-insensitive search”. If neither "case-sensitive" nor "case-insensitive" is present, text of search term is used to determine case sensitivity. If text of search term contains no uppercase, it specifies "case-insensitive". If text of search term contains uppercase, it specifies "case-sensitive".

Background

MarkLogic Server includes element level security (ELS), an addition to the security model that allows you to specify security rules on specific elements within documents. Using ELS, parts of a document may be concealed from users who do not have the appropriate roles to view them. ELS can conceal the XML element (along with properties and attributes) or JSON property so that it does not appear in any searches, query plans, or indexes - unless accessed by a user with appropriate permissions.

ELS protects XML elements or JSON properties in a document using a protected path, where the path to an element or property within the document is protected so that only roles belonging to a specific query roleset can view the contents of that element or property. You specify that an element is part of a protected path by adding the path to the Security database. You also then add the appropriate role to a query roleset, which is also added to the Security database.

ELS uses query rolesets to determine which elements will appear in query results. If a query roleset does not exist with the associated role that has permissions on the path, the role cannot view the contents of that path.

Notes:

1. A user with admin privileges can access documents with protected elements by using fn:doc to retrieve documents (instead of using a query). However, to see protected elements as part of query results, even a user with admin privileges will need to have the appropriate role(s).
2. ELS applies to both XML elements and JSON properties; so unless spelled out explicitly, 'element' refers to both XML elements and JSON properties throughout this article.

You can read more about how to configure Element Level Security here, and can see how this all works at this Element Level Security Example.

Node-update

One of the commonly used document level capabilities is 'update'. Be aware, however, that document level update is too powerful to be used with ELS permissions as someone with document level update privileges could update not only a node, but also delete the whole document. Consequently, a new document-level capability - 'node-update' - has been introduced. 'node-update' offers finer control when combined with ELS through xdmp:node-replace and xdmp:node-delete functions as they can be used to update/delete only the specified nodes of a document (and not the document itself in its entirety).

Document-level vs Element-level security

Unlike at the document-level:

• 'update' and 'node-update' capabilities are equivalent at the element-level. However, at the document-level, if a user only has a 'node-update' privilege to a document, you cannot delete the document. In contrast, 'update' privileges allows that user to delete the document
• 'Read', 'insert' and 'update' are checked separately at the element level i.e.:
  • read operations - only permissions with 'read' capability are checked
  • node update operations - only permissions with 'node-update' (update) capability are checked
  • node insert operations - only permissions with  'insert' capability are checked

Note: read, insert, update and node-update can all be used at the element-level i.e., they can be part of the protected path definition.

Permissions:

Document-level:

1. update: A node can be updated by any user that has an 'update' capability at the document-level
2. node-update:  A node can be updated by any user with a 'node-update' capability as long as they have sufficient privileges at the element-level

Element-level:

1. If a node is protected but no 'update/node-update' capabilities are explicitly granted to any user, that node can be updated by any user as long as they have 'update/node-update' capabilities at the document-level
2. If any user is explicitly granted 'update/node-update' capabilities to that node at the element level, only that specific user is allowed to update/delete that node. Other users who are expected to have that capability must be explicitly granted that permission at the element level

How does node-replace/node-delete work?

When a node-replace/node-delete is called on a specific node:

1. The user trying to update that node must have at least a 'node-update' (or 'update') capability to all the nodes up until (and including) the root node
2. None of the descendant nodes of the node being replaced/deleted can be protected by a different roles. If they are protected:
   1. 'node-delete' isn’t allowed as deleting this node would also delete the descendant node which is supposed to be protected
   2. 'node-replace' can be used to update the value (text node) of the node but replacing the node itself isn’t allowed

Note: If a caller has the 'update' capability at the document level, there is no need to do element-level permission checks since such a caller can delete or overwrite the whole document anyway.

Takeaways:

1. 'node-update' was introduced to offer finer control with ELS, in contrast to the document level 'update'
2. 'update' and 'node-update' permissions behave the same at element-level, but differently at the document-level
   1. At document-level, 'update' is more powerful as it gives the user the permission to delete the entire document
   2. All permissions talk to each other at document-level. In contrast, permissions are checked independently at the element-level
      1. At the document level, an update permission allows you to read, insert to and update the document
      2. At the element level, however, read, insert and update (node-update) are checked separately
         1. For read operations, only permissions with the read capability are checked
         2. For node update operations, only permissions with the node-update capability are checked
         3. For node insert operations, only permissions with the insert capability are checked (this is true even when compartments are used).
3. Can I use ELS without document level security (DLS)?
   1. ELS cannot be used without DLS
   2. Consider DLS the outer layer of defense, whereas ELS is the inner layer - you cannot get to the inner layer without passing through the outer layer
4. When to use DLS vs ELS?
   1. ELS offers finer control on the nodes of a document and whether to use it or not depends on your use-case. We recommend not using ELS unless it is absolutely necessary as its usage comes with serious performance implications
   2. In contrast, DLS offers better performance and works better at scale - but is not an ideal choice when you need finer control as it doesn’t allow node-level operations 
5. How does ELS performance scale with respect to different operations?
   1. Ingestion - depends on the number of protected paths
      1. During ingestion, the server inspects every node for ELS to do a hash lookup against the names of the last steps from all protected paths
      2. For every protected path that matches the hash, the server does a full test of the node against the path - the higher the number of protected paths, the higher the performance penalty
      3. While the hash lookup is very fast, the full test it comparatively much slower - and the corresponding performance penalty increases when there are a large number of nodes that match the last steps of the protected paths
         1. Consequently, we strongly recommend avoiding the use of wildcards at the leaf-level in protected paths
         2. For example: /foo/bar/* has a huge performance penalty compared to /foo/*/bar
   2. Updates - as with ingestion, ELS performance depends on the number of protected paths
   3. Query/Search - in contrast to ELS ingestion or update, ELS query performance depends on the number of query rolesets
      1. Because ELS query performance depends on the number of query rolesets, the concept of Protected PathSet was introduced in 9.0-4
      2. A Protected PathSet allows OR relationships between permissions on multiple protected paths that cover the same element
      3. Because query performance depends on the number of relevant query rolesets, it is highly recommended to use helper functions to obtain the query rolesets of nodes configured with element-level security

Further Reading

• Security Guide: Element Level Security
• Element Level Security and Redaction
• A security model for data integration

Introduction

Some customers have reported problems when attempting to access the Configuration Manager application. In the past, this has been attributed to part of the upgrade process failing for some reason (for example: a port required by MarkLogic already being used) or in some cases it was due to a default databases being removed by the customer at some previous stage.

XDMP-ARGTYPE Error

If you see this error when you attempt to access the Configuration Manager:

500 Internal Server Error XDMP-ARGTYPE XDMP-ARGTYPE: (err:XPTY0004) fn:concat( "could not initialize management plugins with scope: ", $reut:PLUGIN-SCOPE, ": ", xdmp:quote($e)) -- arg1 is not of type xs:anyAtomicType?

Resolving the error

Ensure you have an Extensions database configured by doing the following:

• Log into the MarkLogic Admin interface on port 8001 - http://[your-host]:8001/
• Under "Databases" box, ensure a database called Extensions is listed

If it does not exist, download and run the script attached to this article (create-extensions-db.xqy).

Summary

Does MarkLogic provide encryption at rest?

MarkLogic 9

MarkLogic 9 introduces the ability to encrypt 'data at rest' - data that is on media (on disk or in the cloud), as opposed to data that is being used in a process. Encryption can be applied to newly created files, configuration files, or log files. Existing data files can be encrypted by triggering a merge or re-index of the data.

For more information about using Encryption at Rest, see Encryption at Rest in the MarkLogic Security Guide.

MarkLogic 8 and Earlier releases

MarkLogic 8 does not provide support for encryption at rest for its own forests.

Using Amazon S3 Encryption For Backups

If you are hosting your data locally, would like to back up to S3 remotely, and your goal is that there cannot possibly exist unencrypted copies of your data outside your local environment, then you could backup locally and store the backups to S3 with AWS Client-Side encryption. MarkLogic does not support AWS Client-Side encryption, so this would need to be a solution outside MarkLogic.

See also: MarkLogic documentation: S3 Storage.

See also: AWS: Protecting Data Using Encryption.

Introduction

Here we compare XDBC servers and the Enhanced HTTP server in MarkLogic 8.

Details

XDBC servers are still fully supported in MarkLogic Server version 8. You can upgrade existing XDBC servers without making any changes and you can create new XDBC servers as you did in previous releases.

The Enhanced HTTP Server is an additional feature on HTTP servers which is protocol and binary transport compatible with XCC clients, as long as you use the xcc.httpcompliant=true system property.

The XCC protocol is actually just HTTP, but the details of how to handle body, headers, responses, etc., are "built in" to the XCC client libraries and the XDBC server. The HTTP server in MarkLogic 8 now shares the same low-level code and can dispatch XCC-like requests.

Introduction

This article talks about best practices for use of external proxies vs using rewriter rules in the Enhanced HTTP server.

Details

Whether to use external proxies versus using rewriter rules in the Enhanced HTTP application server is an application design tradeoff not dissimilar to using a single HTTP application server and a XQuery rewriter or endpoint that can dynamically dispatch to different databases and modules (using eval-in).  The Enhanced HTTP server does this type of dispatching much more efficiently, but the concept is similar, with the same pros and cons.

It is mostly an application and business management issue—by sharing the same port you share the same server configuration (authentication, server settings) and the "outside world" only sees one port, so configuring port-based security on firewalls, routers, or load balancers is more difficult.

Summary

A forest reindex timeout error may occur when there are transactions holding update locks on documents for an extended period of time. A reindexer process is started as a result of a database index change or a major MarkLogic Server upgrade.  The reindexer process will not complete until after update locks are released.

Example error text seen in the MarkLogic Server ErrorLog.txt file:

XDMP-FORESTERR: Error in reindex of forest Documents: SVC-EXTIME: Time limit exceeded

Detail

Long running transactions can occur if MarkLogic Server is participating in a distributed transaction environment. In this case transactions are managed through a Resource Manager. Each transaction is executed in a two phase commit. In the first phase, the transaction will be prepared for a commit or a rollback. The actual commit or rollback will occur in the second phase. More details about XA transactions can be found in the Applicactions Developer Guide - Understanding Transactions in MarkLogic Server

In order for a XA transaction to take place, it needs to prepare the execution for the commit. If updates are being made to pre-existing documents, update locks are held against the URIs for those documents. When reindexing is occuring during this process, the reindexer will wait for these locks to be released before it can successfully reindex the new documents.   Because the reindexer is unable to complete due to these pending XA transactions, the hosts in the cluster are unable to completely finish the reindexing task and will eventually throw a timeout error.

Mitigation

To avoid these kind of reindexer timeouts, it is recommended that the database is checked for outstanding XA transactions in "prepare" state before starting a reindexing process. There are two ways to verify if the database has outstanding transactions in "prepare" state:

• In the Admin UI, navigate  to each forest of the database and review the status page; or
• Run the following XQuery code (in Query Console):
  
  xquery version "1.0-ml"; 
  declare namespace fo = "http://marklogic.com/xdmp/status/forest";   
  
  for $f in xdmp:database-forests(xdmp:database()) 
  return    
    xdmp:forest-status($f)//fo:transaction-coordinator[fo:decision-state = 'prepare']

In the case where there are transactions in the "prepare" state, a roll-back can be executed:

• In the Admin UI, click on the "rollback" link for each transaction; or
• Run the following XQuery code (in Query Console):
  
  xquery version "1.0-ml"; 
  declare namespace fo = "http://marklogic.com/xdmp/status/forest";
  
  for $f in xdmp:database-forests(xdmp:database()) 
  return    
    for $id in xdmp:forest-status($f)//fo:transaction-coordinator[fo:decision-state = 'prepare']/fo:transaction-id/fn:string()
    return
      xdmp:xa-complete($f, $id, fn:false(), fn:false())

Introduction

Query Console is an interactive web-based query development tool for writing and executing ad-hoc queries in XQuery, Server-Side JavaScript, SQL and SPARQL. Query Console enables you to quickly test code snippets, debug problems, profile queries, and run administrative XQuery scripts.  Query Console uses workspaces to assist users with organizing queries.  A user can have multiple workspaces, and each workspace can have multiple queries.

Issue

In MarkLogic Server v9.0-11, v10.0-3 and earlier releases, users may experience delays, lag or latency between when a key is pressed on the keyboard, and when it appears in the Query Console query window.  This typically happens when there are a large number of queries in one of the users workspaces.

Workaround

A workaround to improve performance is to reduce the number of queries in each workspace.  The same number of queries can be managed by increasing the number of workspaces and reducing the number of queries in each workspace.  We suggest keeping no more than 30 queries in a workspace to avoid these latency issues.  

The MarkLogic Development team is looking to improve the performance of Query Console, but at the time of this writing, this performance issue has not yet been resolved. 

Further Reading

Query Console User Guide

Introduction

Users of Java based batch processing applications, such as CoRB, XQSync, mlcp and the hadoop connector may have seen an error message containing "Premature EOF, partial header line read". Depending on how exceptions are managed, this may cause the Java application to exit with a stacktrace or to simply output the exception (and trace) into a log and continue.

What does it mean?

The premature EOF exception generally occurs in situations where a connection to a particular application server connection was lost while the XCC driver was in the process of reading a result set. This can happen in a few possible scenarios:

• The host became unavailable due to a hardware issue, segfault or similar issue;
• The query timeout expired (although this is much more likely to yield an XDMP-EXTIME exception with a "Time limit exceeded" message);
• Network interruption - a possible indicator of a network reliability problem such as a misconfigured load balancer or a fault in some other network hardware.

What does the full error message look like?

An example:

INFO: completed 5063408/14048060, 103 tps, 32 active threads
 Feb 14, 2013 7:04:19 AM com.marklogic.developer.SimpleLogger logException
 SEVERE: fatal error
 com.marklogic.xcc.exceptions.ServerConnectionException: Error parsing HTTP
 headers: Premature EOF, partial header line read: ''
 [Session: user=admin, cb={default} [ContentSource: user=admin,
 cb={none} [provider: address=localhost/127.0.0.1:8223, pool=0/64]]]
 [Client: XCC/4.2-8]
 at
 com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:116)
 at com.marklogic.xcc.impl.SessionImpl.submitRequest(SessionImpl.java:268)
 at com.marklogic.developer.corb.Transform.call(Unknown Source)
 at com.marklogic.developer.corb.Transform.call(Unknown Source)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:679)
 Caused by: java.io.IOException: Error parsing HTTP headers: Premature EOF,
 partial header line read: ''
 at com.marklogic.http.HttpHeaders.nextHeaderLine(HttpHeaders.java:283)
 at com.marklogic.http.HttpHeaders.parseResponseHeaders(HttpHeaders.java:248)
 at com.marklogic.http.HttpChannel.parseHeaders(HttpChannel.java:297)
 at com.marklogic.http.HttpChannel.receiveMode(HttpChannel.java:270)
 at com.marklogic.http.HttpChannel.getResponseCode(HttpChannel.java:174)
 at
 com.marklogic.xcc.impl.handlers.EvalRequestController.serverDialog(EvalRequestController.java:68)
 at
 com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:78)
 ... 11 more
 2013-02-14 07:04:19.271 WARNING [12] (AbstractRequestController.runRequest):
 Cannot obtain connection: Connection refused

Configuration / Code: things to try when you first see this message

A possible cause of errors like this may be due to the JVM starting garbage collection and this process taking long enough as to exceed the server timeout setting. If this is the case, try adding the -XX:+UseConcMarkSweepGC java option

Setting the "keep-alive" value to zero for the affected XDBC application server will disable socket pooling and may help to prevent this condition from arising; with keep-alive set to zero, sockets will not be re-used. With this approach, it is understood that disabling keep-alive should not be expected to have a significant negative impact on performance, although thorough testing is nevertheless advised.

Summary

Here we discuss various methods for sharing metering data with Support:  telemetry in MarkLogic 9 and exporting monitoring data.

Discussion

Telemetry

In MarkLogic 9, enabling telemetry collects, encrypts, packages, and sends diagnostic and system-level usage information about MarkLogic clusters, including metering, with minimal impact to performance. Telemetry sends information about your MarkLogic Servers to a protected and secure location where it can be accessed by the MarkLogic Technical Support Team to facilitate troubleshooting and monitor performance.  For more information see Telemetry.

Meters database

If telemetry is not enabled, make sure that monitoring history is enabled and data has been collected covering the time of the incident.  See Enabling Monitoring History on a Group for more details.  

Backup of Meters database

A backup of the full Meters database will provide all the available raw data and is very useful, but is often very large and difficult to transfer, so an export of a defined time range is often requested.

Exporting data

One of the attached scripts can be used in lieu of a Meters database backup. They will provide the raw metering XML files from a defined period of time and can be reloaded into MarkLogic and used with the standard tools.

exportMeters.xqy

This XQuery export script needs to be executed in Query Console against the Meters database and will generate zip files stored in the defined folder for the defined period of time.

Variables for start and end times, batch size, and output directory are set at the top of the script.

get-raw.sh

This bash version will use MLCP to perform a similar export but requires an XDBC server and MLCP installed. By default the script creates output in a subdirectory called meters-export. See the attached script for details. An example command line is

./get-raw.sh localhost admin admin "2018-04-12T00:00:00" "2018-04-14T00:00:00"

Introduction

To avoid index bloat, MarkLogic only records positions in its indexes for words once for word-query fields. When word positions are necessary to accurately match element-word queries, they are normally used from the word-query field. When elements are excluded from the word query field, words under those elements are not indexed - so their positions are not recorded. In MarkLogic 7.0-5 and 8.0-1, a code change was included to avoid false negatives resulting from an element-word query expecting positions from words in elements descended from excluded elements. This code change was to not use positions from the word-query field for element-word searches if the word-query field has exclusions.

Implications

Unfortunately, this solution can sometimes result in false positives - which is captured in 7.0-5 bug #33207 and 8.0-1 bug #32686 (you can read more about both of these bugs in our Fixed Bugs Report). Consequently, a follow-up refinement was shipped in 7.0-5.1 & 8.0-2 to allow for the affected queries to be fully resolveable via indexes. To take advantage of this update, three changes are required:

1) Upgrade to 7.0-5.1 or later, or 8.0-2 or later

2) Database index settings must be updated to tell MarkLogic Server to use positions in this scenario and therefore avoid the previously seen false positives. There are two changes that could be made. Either:

2a. The element in the element-word query must be explicitly included in the word-query field

...or:

2b. All the word-query excluded elements must be configured as phrase-around elements.

3) After the relevant database index settings are updated and the upgrade has been applied, a reindex must be performed

If these changes are made, positions in the word-query field should then be used, which should then ultimately result in the elimination of false positives.

Introduction

 A "fast data directory" is configurable for each forest, and can be set to a directory built on a fast file system, such as one using SSDs. Refer to Using a mix of SSD and spinning drives. If configured MarkLogic Server will try to put as many writes and seeks to the Fast Data Directory (FDD) as it can. As such, it will try to put as many on disk stands as possible onto the FDD. Frequently updated documents tend to reside in the smaller stands and thus are more likely to reside on the FDD.

This article attempts to explain how you should account for the FDD when sizing disk space for your MarkLogic Server.

Details

Fo'est journals will be placed on the fast data directory. 

Each time an automatic merge is performed, MarkLogic Server will attempt to save the results onto the forest's fast data directory. If there is not sufficient space on the FDD, MarkLogic Server will use the forest's primary data directory. To preserve space for future small stands, MarkLogic Server is conservative in deciding whether to put the merge destination stands on the FDD, which means that even if there is enough available space, it may store the result to the forests regular data directory. For more details, refer to the fundamental of resource consumption white paper. 

It is also important to know when the Fast Data Directory is not used: Stands created from a manually triggered merges do not get stored on the fast data directory, but in the forest's primary data directory. Manual merges can be executed by calling the xdmp:merge function or from within the Admin UI; Forest-migrate  and Restoring backups do not put stands in the fast data directory.

Conclusion

MarkLogic Server maintains some disk space in the FDD for checkpoints and journaling. However, since the Fast Data Directory is not used in some procedures, we should not count the size of the FDD when sizing the disk space needed for forest data.

Introduction

The Performance Considerations section of the Loading Content Into MarkLogic Server documentation states 

"When you load content, MarkLogic Server performs updates transactionally, locking documents as needed and saving the content to disk in the journal before the transaction commits. By default, all documents are locked during an update and the journal is set to preserve committed transactions, even if the MarkLogic Server process ends unexpectedly."

There are two types of locking which are specified at the database level:

• Fast locking employs a hashed locking scheme (based on the URI) where each fragment URI has a designated forest, so the lock created during the insert is restricted only to that forest.
• Setting up a database with "strict" locking will force the coordination of an update lock across all forests in the database (and across the cluster) until the insert has taken place.

Fast locking has been the default setting for newly created MarkLogic databases since MarkLogic 5 (released October 2011)

When should I use strict locking?

If at any point in your code, you are specifying the forest to insert document or fragment into (using a technique commonly referred to as in-forest evaluation), configuring the setting for that database at "strict" is definitely the safest choice. If your code always allows the server to determine the target forest for the document/fragment, you're perfectly safe using fast locking.

In the situation where two different people create the same document (with the same URI) and where fast locking was taking place, this would result in:

• A transaction culminating in an insert into a given forest (as assigned by the ML node servicing the request) for the first fragment
• An "update" transaction (in the same forest) where the first fragment is then marked as deleted
• A new fragment takes place of the first fragment to complete the second transaction

Subsequent merges would then remove the stand entry for the first fragment (now deleted/replaced by the subsequent transaction)

The fast option would not create a dangerous race condition unless your application would allow two different people to insert a document with the same URI into two different forests as two separate transactions and where URI assignment is handled by your XQuery/application layer; if the code responsible for making those transactions were to inadvertently assign the same URI to two different forests in a cluster, this could cause a problem that strict locking would guard against. If your application always allows MarkLogic to assign the forest for the document, there is no danger whatsoever in keeping to the server default of "fast" locking.

Additionally - consider what kind of failover you system is using. When using fast journaling with local disk replication, the journal disk write needs to fail on both master and replica nodes in order for data loss to occur - so there's no need for strict in this scenario. In contrast, strict journaling should be used with shared-disk failover, as data loss is possible if using fast journaling and a single node fails before the OS flushes the buffer to disk.

Is there a performance implication in switching to strict locking?

Fast locking will be faster than strict locking, but the performance penalty is largely going to be dependent on a number of factors; the number of forests in a given database, the number of nodes across which the database forests are spread and the speed at which all nodes in the cluster can coordinate a transaction across the cluster (Network/IO) will all have some (potentially minimal) impact.

If the conditions of your application suit, we recommend staying with the default of fast locking on all your databases.

There may be reasons for using 'strict' locking - especially if you are considering loading documents using in-forest-evaluation in your code.

Further reading

https://docs.marklogic.com/guide/ingestion/performance

Summary

There are situations where the SVC-DIRREM, SVC-DIROPEN and SVC-FILRD errors occur on backups to an NFS mounted drive. This article explains how this condition can occur and describes a number of recommendations to avoid such errors.

Under normal operating conditions, with proper mounting options for a remote drive, MarkLogic Server does not expect to report SVC-xxxx errors.  Most likely, these errors are a result of improper nfs disk mounting or other IO issues.

We will begin by exploring methods to narrow down the server which has the disk issue and then list some things to look into in order to identify the cause.

Error Log and Sys Log Observation

The following errors are typical MarkLogic Error Log entries seen during an NFS Backup that indicate an IO subsystem error.   The System Log files may include similar messages.

        Error: SVC-DIRREM: Directory removal error: rmdir '/Backup/directory/path': {OS level error message}

        Error: SVC-DIROPEN: Directory open error: opendir '/Backup/directory/path': {OS level error message}

        Error: Backup of forest 'forest-name' to 'Bakup path' SVC-FILRD: File read error: open '/Backup/directory/path': {OS level error message}

These SVC- error messages include the {OS level error message} retrieved from the underlying OS platform using generic C runtime strerror() system call.  These messages are typically something like "Stale NFS file handle" or "No such file or directory".

If only a subset of hosts in the cluster are generating these types of errrors ...

You should compare the problem host's NFS configuration with rest of the hosts in the cluster to make sure all of the configurations are consistent.

• Compare nfs versions (rpm -qa | grep -i nfs)
• Compare nfs configurations (mount -l -t nfs, cat /etc/mtab, nfsstat)
• Compare platform version (uname -mrs, lsb_release -a) 

NFS mount options 

MarkLogic recommends the NFS Mount settings - 'rw,bg,hard,nointr,noac,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0'

• Vers=3 :  Must have NFS client version v3 or above
• TCP : NFS must be configured to use TCP instead of default UDP
• NOAC : To improve performance, NFS clients cache file attributes. Every few seconds, an NFS client checks the server's version of each file's attributes for updates. Changes that occur on the server in those small intervals remain undetected until the client checks the server again. The noac option prevents clients from caching file attributes so that applications can more quickly detect file changes on the server.
  • In addition to preventing the client from caching file attributes, the noac option forces application writes to become synchronous so that local changes to a file become visible on the server immediately. That way, other clients can quickly detect recent writes when they check the file's attributes.
  • Using the noac option provides greater cache coherence among NFS clients accessing the same files, but it extracts a significant performance penalty. As such, judicious use of file locking is encouraged instead. The DATA AND METADATA COHERENCE section contains a detailed discussion of these trade-offs.
  • NOTE: The noac option is a combination of the generic option sync, and the NFS-specific option actimeo=0.

• ACTIME=0 : Using actimeo sets all of acregmin, acregmax, acdirmin, and acdirmax to the same "0" value. If this option is not specified, the NFS client uses the defaults for each of these options listed above.
• NOINTR : Selects whether to allow signals to interrupt file operations on this mount point. If neither option is specified (or if nointr is specified), signals do not interrupt NFS file operations. If intr is specified, system calls return EINTR if an in-progress NFS operation is interrupted by a signal.
  • Using the intr option is preferred to using the soft option because it is significantly less likely to result in data corruption.
  • The intr / nointr mount option is deprecated after kernel 2.6.25. Only SIGKILL can interrupt a pending NFS operation on these kernels, and if specified, this mount option is ignored to provide backwards compatibility with older kernels.
• BG : If the bg option is specified, a timeout or failure causes the mount command to fork a child which continues to attempt to mount the export. The parent immediately returns with a zero exit code. This is known as a "background" mount.
• HARD (vs soft) : Determines the recovery behavior of the NFS client after an NFS request times out. If neither option is specified (or if the hard option is specified), NFS requests are retried indefinitely. If the soft option is specified, then the NFS client fails an NFS request after retrans retransmissions have been sent, causing the NFS client to return an error to the calling application.
  • Note: A so-called "soft" timeout can cause silent data corruption in certain cases. As such, use the soft option only when client responsiveness is more important than data integrity. Using NFS over TCP or increasing the value of the retrans option may mitigate some of the risks of using the soft option. 

Issue persists => Further debugging 

If after checking NFS configuration and after implementing the MarkLogic recommended NFS mount settings, the issue persists, then you will need to debug the NFS connection during an issue period.    You should enable rpcdebug for NFS on the hosts showing the NFS errors, and then analyze the resulting syslogs during a period that is experiencing the issues

        rpcdebug -m nfs -s all

 The resulting logs may give you additional information to help understand what the source of the failures are.

Introduction

It has long been possible to store binary files in MarkLogic. In the MarkLogic 5 release in 2011, binary support was enhanced to allow for even more control over binary files.

The purpose of this Knowledgebase article is not to cover MarkLogic's binary support in depth but to demonstrate a technique for retrieving a list of URIs for binary files which are managed in a MarkLogic Database.

Retrieving a list of binary document URIs from MarkLogic Server

The following code will use a call to cts:uris to get back a list of all URIs pointing to binary documents for a given MarkLogic database; note that this example assumes that you have the uri lexicon enabled in your database:

Further reading

• https://docs.marklogic.com/guide/app-dev/binaries

People often want fine-grained entitlement control in the applications they build on top of MarkLogic Server. This article discusses two options and their performance implications.

Best Practice

Often, we'll see people attempt an implementation using MarkLogic users and roles. While MarkLogic Server can easily handle a large number of roles in total, you'll run into scalability and performance issues if you have a large number of roles per user. Additionally, you'll want to minimize the number of updates to documents in your Security database as every update requires Security caches to be re-validated, thus incurring a performance penalty.

Instead, for a more scalable and performant solution, you will want to build your entitlements into your documents at the application level, then query those entitlement values with element range indexes on the elements containing those entitlement values.

Summary

When attempting to start MarkLogic Server on older versions of Linux (Non-supported platforms), a "Floating Point Exception" may prevent the server from starting.

Example of the error text from system messages:

kernel: MarkLogic[29472] trap divide error rip:2ae0d9eaa80f rsp:7fffd8ae7690 error:0

Detail

Older Linux kernels will, by default, utilize older libraries.  When a software product such as MarkLogic Server is built using a newer version of gcc, it is possible that it will fail to execute correctly on older systems.  We have seen it in cases where the glibc library is out of date, and not containing certain symbols that were added in newer versions. Refer to the RedHat bug that explains this issue: https://bugzilla.redhat.com/show_bug.cgi?id=482848

The recommended solution is to upgrade to a newer version of your Linux distribution.  While you may be able to resolve the immediate issue by only upgrading the glibc library, it is not recommended.

Introduction

Attached to this article is an XQuery module: "appserver-status.xqy", which will generate a report on all requests currently "in-flight" across all application servers in your cluster

Usage

Run this in Query Console (be sure to display results as html output), it will generate an html table showing all requests currently "in-flight" across all application servers in your cluster. For any transaction taking over 60 seconds, it provides extra detail to help understand and identify bottlenecks where specific modules (or tasks) may be having an adverse effect on the overall performance of the cluster.

The information generated by this module can be used in conjunction with any ticket opened with the support team where assistance is required to better understand and resolve performance issues relating to specific modules. This module could also be used in a situation where DBAs want to perform routine health checks on their cluster to find and identify slow running queries.

Introduction

At the time of this writing (MarkLogic 9), MarkLogic Server cannot perform spherical queries, as the geospatial indexes do not support a true 3D coordinate system.  In situations where cylindrical queries are sufficient, you can create a 2D geospatial index and a separate range index on an altitude value. An "and-query" with these indexes would result in a cylindrical query.

Example

Consider the following sample document structure:

Configure these 2 indexes for your content database:

1. Geospatial Element Pair index specifying latitude localname as ‘lat’ , longitude localname ‘long’ and ‘parent localname’ as ‘location’ in configuration
2. Range element index with localname as ‘alt’ with int scalar type

Assuming you have data in your content database matching above document structure, this query:

will return all the documents with location i.e., points falling in the cylinder with center at 37.655983, -122.425525 having a radius of 1000 miles and with an altitude of less than 4 miles.

Note that in MarkLogic Server 9 geospatial region match was introduced, so the above technique can be extended beyond cylinders.

Introduction

The MarkLogic Monitoring History dashboard (http://localhost:8002/history/) is probably the easiest way to gather monitoring history data, but almost all of this information available within the monitoring dashboard is also available over our ReST APIs:

Application Server Status details

Information on Application Severs can be found at https://docs.marklogic.com/REST/GET/manage/v2/servers and here's an example for getting detailed metrics - http://localhost:8002/manage/v2/servers?group-id=Default&view=metrics&format=xml

For Application Server status information - https://docs.marklogic.com/REST/GET/manage/v2/servers@view=status and here's an example with detailed metrics http://localhost:8002/manage/v2/servers?view=status&group-id=Default&format=xml&fullrefs=true

To access status information for a specific Application Server (for example, the TaskServer), you can get the current status by adding the name to the URI - http://localhost:8002/manage/v2/servers/TaskServer?group-id=Default&view=status&format=xml

You can also get the configuration information for a given application server (for example: "Admin") over the ReST API - http://localhost:8002/manage/v2/servers/Admin/properties?group-id=Default&format=xml

Database and Forest status details

For databases and forests, you can similarly use the endpoints for /databases or /forests:

• https://docs.marklogic.com/REST/management/databases
• https://docs.marklogic.com/REST/management/forests

Database level examples include:

• http://localhost:8002/manage/v2/databases/Meters?view=status&format=xml
• http://localhost:8002/manage/v2/databases/Meters?view=counts&format=xml
• http://localhost:8002/manage/v2/databases/Meters?view=config&format=xml

Forest level examples include:

• http://localhost:8002/manage/v2/forests/Documents?view=status&format=xml
• http://localhost:8002/manage/v2/forests/Documents?view=counts&format=xml
• http://localhost:8002/manage/v2/forests/Documents?view=config&format=xml

Introduction

When configuring database replication, it is important to note that the Connect Forests by Name field is true by default. This works great because, when new forests of the same name are later added to the Master and Replica databases, they will be automatically configured for Database Replication.

The issue

The problem arises when you use replica forest names that do not match the original Master forest names. In that case, you may find that failover events cause forests to get stuck in the Wait Replication state. The usual methods of failing back to the designated masters will not work - restarting the replicas will not work, and neither will shutting down cluster/removing labels/restarting cluster.

Resolution

In this case, the way to fix the issue is to set Connect Forests by Name to false, and then you must manually connect the Master forests on the local cluster to the Replica forests on the foreign cluster, as described in the documentation: Connecting Master and Replica Forests with Different Names.

it is worth noting that, starting MarkLogic 7, you are also allowed to rename the replica forests. Once you rename the replica forests to the same name as the forest name of the designated master database (e.g., the Security database should have a Security forest in both the master and replica), then they will be automatically configured for Database Replication, as expected.

MarkLogic default Group Level Cache and Huge Pages settings

The table below shows the default (and recommended) group level cache settings based on a few common RAM configurations for the 9.0-9.1 release of MarkLogic Server:

Note that these values are safe to use for MarkLogic 7 and above.

For all the databases that ship with MarkLogic Server, the Huge Pages ranges on this table will cover the out-of-the box configuration. Note that adding more forests will cause the second value in the range to increase.

From MarkLogic Server 9.0-7 and above

In the 9.0-7 release and above (and all versions of MarkLogic 10), automatic cache sizing was introduced; this setting is usually recommended.

Maximum group level cache settings

Assuming a Server configured with 256GB RAM (and above), these are the maximum sizes for the three main group level caches and will utilise 180GB (184320MB) per host for the Group Level Caches:

• Expanded Tree Cache - 73728 (72GB) (with 9 8GB partitions)
• List Cache - 73728 (72GB) (with 9 8GB partitions)
• Compressed Tree Cache - 36864 (36GB) (with 11 3 GB partitions)

We have found that configuring 4GB partitions for the Expanded Tree Cache and the List Cache generally works well in most cases; for this you would set the number of partitions to 18

For the Compressed Tree Cache the number of partitions can be set to 22.

Important note

The maximum number of configurable partitions is 32

Each cache partition should be no more than 8192 MB

Introduction

MarkLogic Server has a notion of groups, which are sets of similarly configured hosts within a cluster.

Application servers (and their respective ports) are scoped to their parent group.

Therefore, you need to make sure that the host and its exposed port to which you're trying to connect both exist in the group where the appropriate application server is defined. For example, if you attempt to connect to a host defined in a group made up of d-nodes, you'll only see application servers and ports defined in the d-nodes group. If the application server you actually want is in a different group (say, e-nodes), you'll get a connection error, instead.

Questions

Can I use any xdmp builtins to show which application servers are linked to particular groups?

The code example below should help with this:

Problem:

The errors 'XDMP-MODNOTFOUND - Module not found' and 'XDMP-NOPROGRAM - Server unable to build program from request' may occur when the requested XQuery document does not exist or the user does not have the right permissions on the module.

Solution:

When either of these errors is encountered, the first step would be to check if the requested XQuery module is actually present in the modules database. Make sure the the document uri matches the 'root' of the relevant app-server.

'Modules' field of the app-server configuration specifies the name of the database in which this app-server locates the XQuery application code (if it is not set to 'File-system'). When it is set to a specific database, then only documents in that database whose URI begin with the specified root directory are executable. For example, if 'root'  of the database is set to "/codebase/xquery/", then only documents in the database which start with this uri "/codebase/xquery/" are executable.

If set to 'File-system' make sure the requested module exists in the location specified in the 'root' directory of the app-server. 

Defining a 'File-system' location is often used on single node DEV systems but not recommended on a clustered environment. To keep the deployment of code simple it is recommended to use a Modules database in clustered production system.

Once you made sure that the module does exist, the next step is to check if the user has the right permissions to execute the database. More often, it is likely that the error is caused because of a permissions issue.

(i) Check app-server privileges

The 'privilege' field in the app-server configuration, when set, specified the execute privilege required to access the server. Only users who are assigned this privilege can access the server and the application code. Absence of this privilege may cause the XDMP-NOPROGRAM error.

Make sure the user accessing the app-server has the specified priveleges. This can be checked by using sec:user-privileges() (Should be run against the Security database).

The documentation here - http://docs.marklogic.com/guide/admin/security#id_63953 contains more detailed information about privileges.

(ii) Check permission on the requested module

The user trying to access the application code/modules is required to have the 'execute' permission on the module. Make sure all the xquery documents have 'read' and 'execute' permissions for the user trying to access them. This can be verified by executing the following query against your 'modules' database:

                 xdmp:document-get-permissions("/your-xqy-module")

This returns a list of permission on the document - with the capability that each role has, in the below format:

              <sec:permission xmlns:sec="http://marklogic.com/xdmp/security">
              <sec:capability>execute</sec:capability>
              <sec:role-id>4680733917602888045</sec:role-id>
              </sec:permission>
              <sec:permission xmlns:sec="http://marklogic.com/xdmp/security">
              <sec:capability>read</sec:capability>
              <sec:role-id>4680733917602888045</sec:role-id>
              </sec:permission>

You can then map the role-ids to their role names as below: (this should be done against the Security database)

              import module namespace sec="http://marklogic.com/xdmp/security" at "/MarkLogic/security.xqy";
              sec:get-role-names((4680733917602888045))

If you see that the module does not have execute permission for the user, the required permissions can be added as below: (http://docs.marklogic.com/xdmp:document-add-permissions)

             xdmp:document-add-permissions("/document/uri.xqy",

              (xdmp:permission("role-name","read"),
             xdmp:permission("role-name", "execute")))

Introduction: getting more information about the bugs fixed between releases

As a general recommendation, we encourage customers to keep the server up-to-date with patch releases at any case.

If you would like a list of some of the published bugs that were addressed between two releases of the server (for example: 5.0-3 and 5.0-4.1), you can perform the following steps:

- Log into the support portal at http://help.marklogic.com
- Click on the "Fixed bugs" icon to take you to the bugtrack list
- Select 5.0-3 in the From: dropdown box
- Select 5.0-4.1 in the To: dropdown box
- Click 'Show' to generate an HTML table or View PDF to export the results in a PDF document

Step one: login

Provide your credentials and use the form on the left-hand side to log in to access the support portal

Step two: select the "Fixed bugs" link from the icons on the page

Step three: select the release 'range' from the two dropdown lists on the Fixed Bugs page

Use the Show button to update the page or download the list in PDF format as required

Introduction

In Amazon Web Services, AMIs have unique ids based on their region. There will be many cases when you want to use multiple regions (for example: maintenance of two clusters in separate geographical regions). Below is an example of how to find the list of current AMIs.

Log in to Amazon Web Services

Find your MarkLogic instance on Amazon AWS Marketplace

For example: https://aws.amazon.com/marketplace/pp/B00U36DS6Y

Click continue

View the table

Choose the version of MarkLogic Server that you're planning to use from the version dropdown.

You will see a table containing a list of all current regions and the corresponding AMI ID for our instances for each available region.

Further reading

• http://developer.marklogic.com/products/aws
• http://docs.marklogic.com/guide/ec2/overview
• http://docs.marklogic.com/guide/ec2/GettingStarted
• http://docs.marklogic.com/guide/ec2/CloudFormation
• http://docs.marklogic.com/guide/ec2/managing
• https://d0.awsstatic.com/whitepapers/marklogic-on-aws.pdf

Summary

MarkLogic Server has several different features that can help manage data across multiple database instances. Those features differ from each other in several important ways - this article will focus on high-level distinctions and will provide pointers to other materials to help you decide which of these features could work best for your particular use case.

 Details

Backup/Restore - database backup and restore operations in MarkLogic Server provide consistent database-level views of your data. Propagating data from one instance to another via backup/restore involves a MarkLogic administrator using a completed backup from the source instance as the restore archive on the destination instance. You can read more about Backup/Restore here: http://docs.marklogic.com/guide/admin/backup_restore.

Flexible Replication - can be used to maintains copies of data on multiple MarkLogic Servers. Unlike backup/restore (which relies on taking a consistent, database level view of the data at a particular timestamp), Flexible Replication creates a copy of a document in another database and keeps that copy in sync (possibly with some time-lag/latency) with the original in the course of normal operations. You can read more about Flexible Replication here: http://docs.marklogic.com/guide/flexrep/rep_intro. Do note that:

• Flexible Replication is asynchronous. Asynchronous Replication refers to a configuration in which the Master does not wait for confirmation that the update has been received by the Replica before sending further updates.
• Flexible Replication does not use the same transaction boundaries on the replica as on the master. For example, 10 documents might be inserted in a single transaction on a Flexible Replication master. Those 10 documents will eventually be inserted on a Flexible Replication replica, but there is no guarantee that the replica instance will also use a single transaction to do so.

Database Replication - is used maintains copies of data on multiple MarkLogic Servers. Database Replication creates a copy of a document in another database and keeps that copy in sync (possibly with some time-lag/latency) with the original in the course of normal operations. You can read more about Database Replication here: http://docs.marklogic.com/guide/database-replication/dbrep_intro. Note that:

a. Database Replication is, like Flexible Replication, asynchronous.

b. In contrast to Fleixble Replication, Database Replication operates by copying journal frames from the Master database and replays the transactions described by those journal frames on the foreign Replica database.

XA Transactions - MarkLogic Server can participate in distributed transactions by acting as a Resource Manager in an XA/JTA transaction. If there are multiple MarkLogic Server instances participating as XA resources in a given XA transaction, then it's possible to use that XA transaction as a synchronized means of replicating data across those multiple MarkLogic instances. You can read more about XA Transactions in MarkLogic Server here: http://docs.marklogic.com/guide/xcc/concepts#id_57048.

Introduction

Upgrading individual MarkLogic instances and clusters is generally very easy to do and in most cases requires very little downtime. In most cases, shutting down the MarkLogic instance on each host in turn, uninstalling the current release, installing the updated release and restarting each MarkLogic instance should be all you need to be concerned about...

However, unanticipated problems do sometimes come to light and the purpose of this Knowledgebase article is to offer some practical advice as to the steps you can take to ensure the process goes as easily as possible - this is particularly important if you're planning an upgrade between major releases of the product.

Prerequisites

While the steps outlined under the process heading below offer practical advice as to what to do to ensure your data is safeguarded (by recommending that backups are taken prior to upgrading), another very useful step would be to ensure you have your current configuration files backed up.

Each host in a MarkLogic cluster is configured using parameters which are stored in XML Documents that are available on each host. These are usually relatively small files and will zip up to a manageable size.

If you cd to your "Data" directory (on Linux this is /var/opt/MarkLogic; on Windows this is C:\Program Files\MarkLogic\Data and on OS X this is /Users/{username}/Library/Application Support/MarkLogic), you should see several xml files (assignments, clusters, databases, groups, hosts, server).

Whenever MarkLogic updates any of these files, it creates a backup using the same naming convention used for older ErrorLog files (_1, _2 etc). We recommend backing up all configuration files before following the steps under the next heading.

Process

1) Take a backup for each database in your cluster

2) Turn reindexing off for each database in your cluster

3) Starting with the node hosting your Security and Schemas forests, uninstall the current maintenance release MarkLogic version on your cluster, then install the latest maintenance release in that feature release (for example, if you're currently running version 8.0-5, you'll want to update to the latest available MarkLogic 8 maintenance release - at the time of this writing, it is 8.0-8.1).

4) Start up the host in your cluster hosting your Security and Schemas forests, then the remaining hosts in the cluster.

5) Access the Admin UI on the node hosting your Security and Schemas forests and accept the license agreement, either for just that host (Accept button) or for all of the hosts in the cluster (Accept for Cluster button). If you choose the Accept for Cluster button, a summary screen appears showing all of the hosts in the cluster. Click the Accept for Cluster button to confirm acceptance (all of the hosts must be started in order to accept for the cluster). If you accepted the license just for the one host in the previous step, you must go to all of the Admin Interface for all of the other hosts and accept the license for each host before each host can operate.

6) If you're upgrading across feature releases, you may now repeat steps #3-5 until you reach the desired feature and maintenance release on your cluster (for example, if trying to upgrade to the major feautre release MarkLogic 9, after installing 8.0-latest, you'll repeat steps 3-5 for version 9.0-latest).

7) After you've finished upgrading across all the relevant feature releases, re-enable reindexing for each database in your cluster.

For more details, please go through Section 6.4: “Upgrading a Cluster to a New Maintenance Release of MarkLogic Server” of “Scalability, Availability, and Failover” guide available here: http://developer.marklogic.com/pubs/.

If you've got database replication in place across both a master and replica cluster, then be aware that:

1) You do not need to break replication between the clusters

2) You should plan to upgrade both the master cluster and replica cluster. If you upgrade just the master, connectivity between the two clusters will stop due to different XDQP versions. 

3) If the Security database isn't replicated, then there shouldn't be anything special you need to do other than upgrade the two clusters.

4) If the security database is replicated, do the following:

• Upgrade the Replica cluster and run the upgrade scripts. This will update the Replica's Security database to indicate that it is current. It will also do any necessary configuration upgrades.
• Upgrade the Master cluster and run the upgrade scripts. This will update the Master's Security database to indicate that it is current. It will also do any necessary configuration upgrades.

For more here Updating Clusters Configured with Database Replication

Back-out Plan

MarkLogic does not support restoring a backup made on a newer version of MarkLogic Server onto an older version of MarkLogic Server. Your Back-out plan will need to take this into consideration.

See the section below for recommendations on how this should be handled.

Further reading

Backing out of your upgrade: steps to ensure you can downgrade in an emergency

Product release notes

The "Upgrade Support" section of the release notes.

All known incompatibilities between releases

The "Upgrading from previous releases" section of the documentation

MarkLogic Support Fixed Bug List

Introduction

spell:suggest() and spell:suggest-detailed aren't simply looking for character differences between the provided strings and the strings in your dictionaries - they're also factoring in differences in the resulting phonetics represented by these strings.

Detail

There is an undocumented option that can be passed along to increase the phonetic-distance threshold (which is 1, by default). For example, consider the following:

Note that the option "distance-threshold" corresponds to "distance" in the result, and "phonetic-distance" corresponds to "key-distance."

Also note that increasing the phonetic-distance may cause spell:suggest() and spell:suggest-detailed() to use significantly more CPU. Metaphones are short keys, so a larger distance may match a very large fraction of the dictionary, which would then mean each of those matches would need to be checked in the distance algorithms.

Background

A database consists of one or more forests. A forest is a collection of documents (mostly XML trees, thus the name), implemented as a physical directory on disk. Each forest holds a set of documents and all their indexes. 

When a new document is loaded into MarkLogic Server, the server puts this document in an in-memory stand and writes the action to an on-disk journal to maintain transactional integrity in case of system failure. After enough documents are loaded, the in-memory stand will fill up and be flushed to disk, written out as an on-disk stand. As more document are loaded, they go into a new in-memory stand. At some point this in-memory stand fills up as well, and the in-memory stand gets written as yet another new on-disk stand.

To read a single term list, MarkLogic must read the term list data from each individual stand and unify the results. To keep the number of stands to a manageable level where that unification isn't a performance concern, MarkLogic runs merges in the background. A merge takes some of the stands on disk and creates a new singular stand out of them, coalescing and optimizing the indexes and data, as well as removing any previously deleted fragments
Each forest has its own in-memory stand and set of on-disk stands. Loading and indexing content is a largely parallelizable activity so splitting the loading effort across forests and potentially across machines in a cluster can help scale the ingestion work.

Deletions and Multi-Version Concurrency Control (MVCC)

What happens if you delete or change a document? If you delete a document, MarkLogic marks the document as deleted but does not immediately remove it from disk. The deleted document will be removed from query results based on its deletion markings, and the next merge of the stand holding the document will bypass the deleted document when writing the new stand. MarkLogic treats any changed document like a new document, and treats the old version like a deleted document.

This approach is known in database circles as which stands for Multi-Version Concurrency Control (or MVCC).
In an MVCC system changes are tracked with a timestamp number which increments for each transaction as the database changes. Each fragment gets its own creation-time (the timestamp at which it was created) and deletion-time (the timestamp at which it was marked as deleted, starting at infinity for fragments not yet deleted).

For a request that doesn't modify data the system gets a performance boost by skipping the need for any URI locking. The query is viewed as running at a certain timestamp, and throughout its life it sees a consistent view of the database at that timestamp, even as other (update) requests continue forward and change the data.

Updates and Deadlocks

An update request, because it isn't read-only, has to use read/write locks to maintain system integrity while making changes. Read-locks block for write-locks; write-locks block for both read and write-locks. An update has to obtain a read-lock before reading a document and a write-lock before changing (adding, deleting, modifying) a document. Lock acquisition is ordered, first-come first-served, and locks are released automatically at the end of a request.

In any lock-based system you have to worry about deadlocks, where two or more updates are stalled waiting on locks held by the other. In MarkLogic deadlocks are automatically detected with a background thread. When the deadlock happens on the same host in a cluster, the update farthest along (with the most locks) wins and the other update gets restarted. When it happens on different hosts, because lock count information isn't in the wire protocol, both updates start over. MarkLogic differentiates queries from updates using static analysis. Before running a request, it looks at the code to determine if it includes any calls to update functions. If so, it's an update. If not, it's a query. Even if at execution time the update doesn't actually invoke the updating function, it still runs as an update.

For the most part it's not under the control of the user. The one exception is there's an xdmp:lock-for-update($uri) call that requests a write-lock on a document URI, without actually having to issue a write and in fact without the URI even having to exist.

When a request potentially touches millions of documents (such as sorting a large data set to find the most recent items), a query request that runs lock-free will outperform an update request that needs to acquire read-locks and writelocks. In some cases you can speed up the query work by isolating the update work to its own transactional context. This technique only works if the update doesn't have a dependency on the outer query, but that turns out to be a common case. For example, let's say you want to execute a content search and record the user's search string to the database for tracking purposes. The database update doesn't need to be in the same transactional context as the search itself, and would slow things down if it were. In this case it's better to run the search in one context (read-only and lock-free) and the update in a different context. See the xdmp:eval() and xdmp:invoke() functions for documentation on how to invoke a request from within another request and manage the transactional contexts between the two.

Document Lifecycle

Let's track the lifecycle of a document from first load to deletion until the eventual removal from disk. A document load request acquires a write-lock for the target URI as part of the xdmp:document-load() function call. If any other request is already doing a write to the same URI, our load will block for it, and vice versa. At some point, when the full update request completes successfully (without any errors that would implicitly cause a rollback), the actual insertion work begins, processing the queue of update work orders. MarkLogic starts by parsing and indexing the document contents, converting the document from XML to a compressed binary fragment representation. The fragment gets added to the in-memory stand. At this point the fragment is considered a nascent fragment, a term you'll see sometimes on the administration console status pages. Being nascent means it exists in a stand but hasn't been fully committed. (On a technical level, nascent fragments have creation and deletion timestamps both set to infinity, so they can be managed by the system while not appearing in queries prematurely.) If you're doing a large transactional insert you'll accumulate a lot of nascent fragments while the documents are being processed. They stay nascent until they've been committed. Once the fragment is placed into the in-memory stand, the request is ready to commit. It obtains the next timestamp value, journals its intent to commit the transaction, and then makes the fragment available by setting the creation timestamp for the new fragment to the transaction's timestamp. At this point it's a durable transaction, replayable in event of server failure, and it's available to any new queries that run at this timestamp or later, as well as any updates from this point forward (even those in progress). As the request terminates, the write-lock gets released.

Our document lives for a time in the in-memory stand, fully queryable and durable, until at some point the in-memory stand fills up and gets written to disk. Our document is now in an on-disk stand. Sometime later, based on merge algorithms, the on-disk stand will get merged with some other on-disk stands to produce a new on-disk stand. The fragment will be carried over, its tree data and indexes incorporated into the larger stand. This might happen several times.

At some point a new request makes a change to the document, such as with an xdmp:node-replace() call. The request making the change first obtains a read-lock on the URI when it first accesses the document, then promotes the read-lock to a write-lock when executing the xdmp:node-replace() call. If another write-lock were already present on the URI from another executing update, the read-lock would have blocked until the other write-lock released. If another read-lock were already present, the lock promotion to a write-lock would have blocked. Assuming the update request finishes successfully, the work runs similar to before: parsing and indexing the document, writing it to the in-memory stand as a nascent fragment, acquiring a timestamp, journaling the work, and setting the creation timestamp to make the fragment live. Because it's an update, it has to mark the old fragment as deleted also, and does that by setting the deletion timestamp of the original fragment to the transaction timestamp. This combination effectively replaces the old fragment with the new. When the request concludes, it releases its locks. Our document is now deleted, replaced by the new version.

The old fragment still exists on disk, of course. In fact, any query that was already in progress before the update incremented the timestamp, or any query doing time travel with an old timestamp, can still see it. Eventually the on-disk stand holding the fragment will be merged again, at which point the old fragment will be completely removed from the system. It won't be written into the new on-disk stand. That is, unless the administration "merge timestamp" was set to allow deep time travel. In that case it will live on, sticking around in case any new queries want to time travel to see old fragments.

Summary

The following article explains the way in-memory caches are used by MarkLogic Server and how can they be utilized to improve query execution.

Detail

MarkLogic Server provides several caches that are used to improve the performance during query execution. When a query executes for the first time, the Server will populate these caches to store termlist and data fragments in memory.

MarkLogic Server keeps a lot of its configuration information in databases, and has a lot of caches to make it run faster, but those caches get populated the first time things are accessed. The server also uses book-keeping terms in the indexes to keep track of whether all documents have been indexed with the current settings. MarkLogic caches this information, but has to query the indexes on the first request to warm the cache.

The in-memory cache in MarkLogic Server holds data that was recently added to the system and is still in an in-memory stand; that is, it holds data that has not yet been written to disk.

For updates, if there is no in-memory stand on a forest when a new document is inserted, the server will create it. This stand is big enough for thousands of documents, but the cost of creating it will be seen in the time taken for the first document added to it.

How will the in-memory cache help improve query execution

When a query is executed, the in-memory data structures like range indexes and lexicons get pinned into RAM the first time they are used.  The easiest way to speed things up is to "warm the caches” by running a small sample program that exercises the type-ahead prior to starting production. You can also keep the server warm by doing a non-time-critical stub update at time intervals (every 30 sec to 1 minute). If the server is idle, then it will serve to keep caches and in-memory stand warm. If the server is really busy then it would only take a small amount of extra work. Once this is done, the functionality will be fast for all users in all future sessions.

Introduction

Be aware that MarkLogic does not recommend having more than one forest for the Security database.

The Security database is typically fairly small and there is really no reason to have more than one forest for the Security database. Having more than one Security forest causes additional complexity during failover events, server upgrades, and restarts. Given how critical the Security database is for the cluster, it is less damaging for it to have a single forest and failover when it goes down, rather than go into a difficult to resolve state with multiple forests.

In terms of forest failover, one or at most two local disk failover forests may and should be configured for high availablity. In terms of database replication, replica forests in replica clusters can be configured.

If you have more than one Security forest(s):

We have seen incidents where customers attached more than one Security forest either intentionally or inadvertently (scripting bug or user error) and run into issues while detaching them.

Most of the issues are rebalancer related. Since the rebalancer is 'on' for all databases by default, when a new forest is attached, the database would automatically redistribute the content across all attached forests. This is expected behavior and how the rebalancer normally works. However, problems arise when forests are detached without first retiring them. This is generally true for any database, but it is particularly problematic when dealing with the Security database.

When a Security database forest is detached without first retiring it, a portion of the Security documents are effectively removed from the database - which causes users to be locked out of the cluster (in this case, please contact MarkLogic Support to help with the repair).

Best Practice

• We do not recommend configuring more than one forest for system databases like Security
• If you must have multiple forests in your Security database:
  • it is very important to retire forests from the Security database, before detaching them
  • The rebalancer should turned 'off' for the Security database.

Further reading

Administering Security in MarkLogic

Database Rebalancing in MarkLogic

Restoring Security Database

Security Database restore leading to lingering Certificate Template id in Config files

Introduction

This Knowledgebase article is a general guideline for backups using the journal archiving feature for both free space requirements and expected file sizes written to the archive journaling repository when archive journaling is enabled and active.

The MarkLogic environment used here was an out-of-the box version 9.x with one change of adding a new directory specific to storing the archive journal backup files.

It is assumed that the reader of this article already has a basic understanding of the role of Journal Archiving in the Backup and Restore feature of MarkLogic Server. See references below for further details(below).

How much free space is needed for the Archive Journal files in a backup?

MarkLogic Server uses the forest size of the active forest to confirm whether the journal archive repository has enough free space to accommodate that forest, but if additional forests already exist on the same volume, then there may be an issue in the Server's "free-space" calculation as the other forests are never used in the algorithm that calculates the free space available for the backup and/or archive journal repositories. Only one forest is used in the free-space calculation.

In other words, if multiple forests exist on the same volume, there may not be enough free space available on that specific volume due to the additional forests; especially during a high rate of ingestion. If that is the case, then it is advised to provide enough free space on that volume to accommodate the sizes of all the forests. Required Free Space(approximately) = (Number of Forests) x (Size of largest Forest).

What can we expect to see in the journal archiving repository in terms of files sizes for specific ingestion types and sizes? That brings us to the other side.

How is the Journal Archive repository filling up?

1 MByte of raw XML data loaded into the server (as either a new document ingestion or a document update) will result in approximately 5 to 6 MBytes of data being written to the corresponding Journal Archive files.  Additionally, adding Range Indexes will contribute to a relatively small increase in consumed space.

Ingesting/updating RDF data results in slightly less data being written to the journal archive files.

In conclusion, for both new document ingestion and document updates, the typical expansion ratio of Journal Archive size to Input file size is between 5 an 6 but can be higher than that depending on the document structure and any added range indexes.

References:

• Backup with Journal Archiving
• Understanding the role of Journals in relation to Backup and Restore (Journal Archiving)

Introduction

Content processing applications often require multi-step processing. Each step in the process performs a particular task or set of tasks. The Content Processing Framework in MarkLogic Server supports these types of multi-step conversion processes. Sometimes during document delete operation, it is possible that the CPF action might fail with 'XDMP-CONFLICTINGUPDATES' error, which can be seen in document-properties file like:

Sample message:

<error:format-string>XDMP-CONFLICTINGUPDATES: xdmp:document-set-property("FILE-NAME", <cpf:state xmlns:cpf="http://marklogic.com/cpf">http://marklogic.com/states/deleted</cpf:state>) -- Conflicting updates xdmp:document-set-property("FILE-NAME", /cpf:state) and xdmp:document-delete("FILE-NAME")</error:format-string>

This error message indicates that an update statement (for e.g. xdmp:document-set-property) is trying to update a document that is conflicting with other update occurring (e.g. xdmp:document-delete) in the same transaction.

Detail

Actions that want to delete the target URI need special handling because MarkLogic CPF also wants to keep track of progress in the properties, and just having document-delete [ xdmp:document-delete($cpf:document-uri) ]can't do that.

Following are ways to achieve the expected behavior and get past the XDMP-CONFLICTINGUPDATES error:

1) Performing a "soft delete" on the document and then let CPF take care of deleting the document. This can be done by setting the document status to "deleted" via cpf:document-set-processing-status API function. Setting the document's processing status to "deleted" will tell CPF to clean up the document and not update properties at the same time.

cpf:document-set-processing-status( $uri-to-delete, "deleted" )

Additional details can be found at: http://docs.marklogic.com/cpf:document-set-processing-status


2) If you want to keep a record of the URI that is being deleted, you can delete its root node instead of the document. The CPF state will be able be recorded in document-properties, even if the document is gone.

xdmp:node-delete(doc($uri-to-delete))

Details at: http://docs.marklogic.com/xdmp:node-delete

Introduction

Sometimes, when a host is removed from a cluster in an improper manner -- e.g., by some means other than the Admin UI or Admin API, a remote host can still try to communicate with its old cluster, but the cluster will recognize it as a "foreign IP" and will log a message like the one below:

2014-12-16 00:00:20.228 Warning: XDQPServerConnection::init(10.0.80.7:7999-10.0.80.39:44247): SVC-SOCRECV: Socket receive error: wait 10.0.80.7:7999-10.0.80.39:44247: Timeout

Explanation: 

XDQP is the internal protocol that MarkLogic uses for internal communications amongst the hosts in a cluster and it uses port 7999 by default. In this message, the local host 10.0.80.7 is receiveng socket connections from foreign host 10.0.80.39.

Debugging Procedure, Step 1

To find out if this message indicates a socket connection from an IP address that is not part of the cluster, the first place is to look is in the hosts.xml files. If the IP address in not found in the hosts.xml, then it is a foreign IP. In that case, the following are the steps will help to identify the the processes that are listening on port 7999.

Debugging Procedure, Step 2

To find out who is listening on XDQP ports, try running the following command in a shell window on each host:

      $ sudo netstat -tulpn | grep 7999

You should only see MarkLogic as a listner:

     tcp 0 0 0.0.0.0:7999 0.0.0.0:* LISTEN 1605/MarkLogic

If you see any other process listening on 7999, yopu have found your culprit. Shot down those processes and the messages will go away.

Debugging Procedure, Step 3

If the issue persists, run tcpdump to trace packets to/from "foreign" hosts using the following command:

     tcpdump -n host {unrecognized IP}

Shutdown MarkLogic on those hosts. Also, shutdown any other applications that are using port 7999.

Debugging Procedure, Step 4

If the cluster are hosts on AWS, you may also want to check on your Elastic Load Balancer ports. This may be tricky, because instances will change IP addresses if they are rebooted, so  work with AWS Support to help you find the AMI or load balancer instance that is pinging your cluster.

In the case that the "foreign host" is an elastic load balancer, be sure to remove port 7999 from its rotation/scheduler. In addition, you should set the load balancer to use port 7997 for the heartbeat functionality.

Introduction

Sometimes, when a cluster is under heavy load, your cluster may show a lot of XDQP-TIMEOUT messages in the error log. Often, a subset of hosts in the cluster may become so busy that the forests they host get unmounted and remounted repeatedly. Depending on your database and group settings, the act of remounting a forest may be very time-consuming, due to the fact that that all hosts in the cluster are being forced to do extra work of index detection.

Forest Remounts

Every time a forest remounts, the error log will show a lot messages like these:

2012-08-27 06:50:33.146 Debug: Detecting indexes for database my-schemas
2012-08-27 06:50:33.146 Debug: Detecting indexes for database Triggers
2012-08-27 06:50:35.370 Debug: Detected indexes for database Last-Login: sln
2012-08-27 06:50:35.370 Debug: Detected indexes for database Triggers: sln
2012-08-27 06:50:35.370 Debug: Detected indexes for database Schemas: sln
2012-08-27 06:50:35.370 Debug: Detected indexes for database Modules: sln
2012-08-27 06:50:35.373 Debug: Detected indexes for database Security: sln
2012-08-27 06:50:35.485 Debug: Detected indexes for database my-modules: sln
2012-08-27 06:50:35.773 Debug: Detected indexes for database App-Services: sln
2012-08-27 06:50:35.773 Debug: Detected indexes for database Fab: sln
2012-08-27 06:50:35.805 Debug: Detected indexes for database Documents: ss, fp

... and so on ...

This can go on for several minutes and will cost you more down time than necessary, since you already know the indexes for each database.

Improving the situation

Here are some suggestions for improving this situation:

1. Browse to Admin UI -> Databases -> my-database-name
2. Set ‘index detection’ to ‘none’
3. Set ‘expunge locks’ to ‘none’

Repeat steps 1-4 for all active databases.

Now tweak the group settings to make the cluster less sensitive to an occasional busy host:

1. Browse to Admin UI -> Groups -> E-Nodes
2. Set ‘xdqp timeout’ to 30
3. Set ‘host timeout’ to 90
4. Click OK to make this change effective.

The database-level changes tell the server to speed up cluster startup time when a server node is perceived to be offline. The group changes will cause the hosts on that group to be a little more forgiving before declaring a host to be offline, thus preventing forest unmounting when it's not really needed.

If after performing these changes, you find that you are still experiencing XDQP-TIMEOUT's, the next step is to contact MarkLogic Support for assistance. You should also alert your Development team, in case there is a stray query that is causing the data nodes to gather too many results.

Related Reading

XML Data Query Protocol (XDQP)

Introduction

Under normal operations, only a single user object is created for a user-name. However, when users are migrated from another security database and if the recommend checking is not performed, duplicate user-names might be created.

Resolution

When there are duplicate user-names in the database, you may see the following message on the Admin UI or in the error logs:

500: Internal Server Error
XDMP-AS: (err:XPTY0004) get-element($col, "sec:user", "sec:user-name", $user-name, "SEC-USERDNE") -- Invalid coercion: (fn:doc("http://marklogic.com/xdmp/users/*******")/sec:user, fn:doc("http://marklogic.com/xdmp/users/*******")/sec:user) as element()?

To fix duplicate user-names, the extra security object that is created needs to be removed. You can delete one of the extra security objects, which should have a URI similar to:

http://marklogic.com/xdmp/users/******* where "*******" represents the user-id's.

To resolve the issue, follow the below steps:

1. Perform a backup of your Security database in case manual recovery is required.

2. Login to the QConsole with admin credentials.

3. Select "Security" database as the content-source

4. Delete the security object by executing xdmp:document-delete($uri) with $uri set to the Uri of the duplicate user.

Introduction

For hosts that don't use a standard US locale (en_US) there are instances where some lower level calls will return data that cannot be parsed by MarkLogic Server. An example of this is shown with a host configured with a different locale when making a call to the Cluster Status page (cluster-status.xqy):

The problem

The problem you have encountered is a known issue: MarkLogic Server uses a call to strtof() to parse the values as floats:

http://linux.die.net/man/3/strtof

Unfortunately, this uses a locale-specific decimal point. The issue in this environment is likely due to the Operating System using a numeric locale where the decimal point is a comma, rather then a period.

Resolving the issue

The workaround for this is as follows:

1. Create a file called /etc/marklogic.conf (unless one already exists)

2. Add the following line to /etc/marklogic.conf:

export LC_NUMERIC=en_US.UTF-8

After this is done, you can restart the MarkLogic process so the change is detected and try to access the cluster status again.

Introduction

This article discusses the effects of the incremental backup implementation on Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO).

Details

With MarkLogic 8 you can have multiple daily incremental backups with minimal impact on database performance.

Incrementals complete more quickly than full backups reducing the backup window. A smaller backup window enables more frequent backups, reducing the RPO of the database in case of disaster.

However, RTO can be longer when using incremental backups compared to just full backups, because multiple backups must be restored to recover.

There are two modes of operation when using incremental backups:

Incremental since last full. Here, each incremental has to store all the data that has changed since the last full backup. Since a restore only has to go through a single incremental data set, the server is able to perform a faster restore.  However, each incremental data set is bigger and takes longer to complete than the previous data set because it stores all changes that were included in the previous incremental.

Please note when doing “Incremental since last full”:-

- Create a new incremental backup directory for each incremental backup

- Call database-incremental-backup with incremental-dir set to the new incremental backup directory

Incremental since last incremental.  In this case, a new incremental stores only changes since the last incremental, also known as delta backups. By storing only the changes since the last incremental, the incremental backup sets are smaller in size and are faster to complete.  However, a restore operation would have to go through multiple data sets.

Please note when doing “Incremental since last incremental”:-

- Create an incremental backup directory ONCE

- Call database-incremental-backup with the same incremental backup directory.

See also the documentation on Incremental Backup.

Indexing Best Practices

MarkLogic Server indexes records (or documents/fragments) on ingest. When a database's index configuration is changed, the server will consequently reindex all matching records.

Indexing and reindexing can be a CPU and I/O intensive operation. Reindexing creates a lot of new fragments, with the original fragments being marked for deletion. These deleted fragments will then need to be merged out. All of this activity can potentially affect query performance, especially in systems with under-provisioned hardware.

Reindexing in Production

If you need to add or modify an index on a production cluster, consider scheduling the reindex during a time when your cluster is less busy. If your database is too large to completely reindex during a single period of low usage, consider running the reindex over several periods of time. For example, if your low usage period is during a weekend, the process may look like:

• Change your index configuration on a Friday night
• Let the reindex run for most of the weekend
• To pause the reindex, set the reindexer-enable field to 'false' for the database being reindexed. Be sure to allow sufficient time for the associated merging to complete before system load comes back.
• If needed, reindexing can continue over the next weekend - the reindexer process will pick up where it left off before it was disabled.

You can refer to https://help.marklogic.com/Knowledgebase/Article/View/18/15/how-reindexing-works-and-its-impact-on-performance for more details on invoking reindexing on production.

      When you have Database Replication Configured

If you have to add or modify indexes on a database which has database replication configured, make sure the same changes are made on the Replica cluster as  well. Starting with ML server version 9.0-7, index data is also replicated from the Master to the Replica, but it does not automatically check if both sides have the same index settings. Reindexing is disabled by default on a replica cluster. However, when database replication configuration is removed (such as after a disaster),  the replica database will reindex as necessary. So it is important that the Replica database index configuration matches the Master’s to avoid unnecessary reindexing.

Further reading -

Master and Replica Database Index Settings

Database Replication - Indexing on Replica Explained

Avoid Unused Range Indexes, Fields, and Path Indexes

In addition to taking up extra disk space, Range, Field, and Path Indexes require extra work when it's time to reindex. Field and Path indexes may also require extra indexing passes.

Avoid Using Namespaces to Implement Multi-Tenancy

It's a common use case to want to create some kind of partition (or multiple partitions) between documents in a particular database. In such a scenario it's far better to 1) constrain the partitioning information to a particular element in a document (then include a clause over that element in your searches), than it is to 2) attempt to manage partitions via unique element namespaces corresponding to each partition. For example, given two documents in two different partitions, you'll want them to look like this:

1a. <doc><partition>partition1</partition><name>Joe Smith</name></doc>

1b. <doc><partition>partition2</partition><name>John Smith</name></doc>

...vs. something like this:

2a. <doc xmlns:p="http://partition1"><p:name>Joe Smith</p:name></doc>

2b. <doc xmlns:p="http://partition2"><p:name>John Smith</p:name></doc>

Why is #1 better? In terms of searching the data once it's indexed, there's actually not much of a difference - one could easily create searches to accommodate both approaches. The issue is how the indexing works in practice. MarkLogic Server indexes all content on ingest. In scenario #2, every time a new partition is created, a new range element index needs to defined in the Admin UI, which means your index settings have changed, which means the server now needs to reindex all of your content - not just the documents corresponding to the newly introduced partition. In contrast, for scenario #1, all that would need to be done is to ingest the documents corresponding to the new partition, which would then be indexed just like all the other existing content. There would be a need, however, to change the searches in scenario #1, as they would not yet include a clause to accommodate the new partition (for example: cts:element-value-query(xs:QName("partition"), "partition2")) - but the overall impact of adding a partition is changing the searches in scenario #1, which is ultimately far, far less intrusive a change than reindexing your entire database as would be required in scenario #2. Note that in addition to a database-wide reindex, searches would also need to change in scenario #2, as well.

Keep an Eye on I/O Throughput

Reindexing can lead to heavy merge activity and may lead to disk I/O bottlenecks if not managed carefully. If you have a system that is available 24-7 with no downtime window, then you may need to throttle the reindexer in order to keep the disk I/O to a minimum. We suggest the following database settings for reindexing a system that must always remain in use:

• reindexer-throttle = 3
• large-size-threshold = 1048576

You can also adjust the following group settings to help limit background I/O:

• background-io-limit = 100

This will limit the background I/O for that group to 100 MB/sec per host across all hosts in that group. This should only be configured if merges are causing problems—it is a way of throttling back the I/O used by the merging process.This is good starting point, and may be increased in increments of 50 if you find that your merges are progressing too slowly.  Proceed with caution as too low of a background IO limit can have negative performance or even catastrophic consequences. 

General Recommendations

In general, your indexing/reindexing and subsequent search experience will be better if you

• Think of documents as rows in a relational system, not like tables (Refer to https://developer.marklogic.com/learn/data-model/ for more details)
• Insure that a document-unique property that requires a range index should have a unique QName. Be aware of cases where a range index can match more than one thing per document (i.e. ingest date and edit date could both be represented by xs:date). This avoids a required index having many more entries, and using up RAM unnecessarily for entries that are never sorted or accessed from a lexicon. (Refer to https://docs.marklogic.com/guide/admin/range_index#id_93351 for more details on range index)
• Store dates and times as Schema datatypes
• Avoid huge fragment sizes (Refer to https://docs.marklogic.com/guide/admin/fragments for further details)
• Avoid fragmentation via fragment roots, unless absolutely necessary. Check this KB, https://help.marklogic.com/Knowledgebase/Article/View/529/0/search-and-fragmentation and https://www.marklogic.com/blog/fragment-ed-thoughts/ for more details on fragmentation.
• Avoid indiscriminately using properties fragments (which doubles the fragment count). Use if you need CPF, for example.
• Avoid leaving directory creation on if not needed (needed for webdav, but not for directory-query)
• Avoid leaving last-modified on if not needed
• Avoid using a schema that defines elements in no-namespace (which will leak to all other non-namespaced elements)

Summary

MarkLogic Admin GUI is convenient place to deploy the Normal Certificate infrastructure or use the Temporary Certificate generated by MarkLogic. However for certain advance solutions/deployment we need XQuery based admin operations to configure MarkLogic.

This knowledgebase discusses the solution to deploy SAN or Wildcard Certificate in 3 node (or more) cluster.

Certificate Types and MarkLogic Default Config

Certificate Types

In general, When browsers connect to a Server using HTTPS, they check to make sure your SSL Certificate matches the host name in the address bar. There are three ways for browsers to find a match:

a).The host name (in the address bar) exactly matches the Common Name in the certificate's Subject.

b).The host name matches a Wildcard Common Name. Please find example at end of article. 

c).The host name is listed in the Subject Alternative Name (SAN) field as part of X509v3 extensions. Please find example at end of article.

The most common form of SSL name matching is for the SSL client to compare the server name it connected to with the Common Name (CN field) in the server's Certificate. It's a safe bet that all SSL clients will support exact common name matching.

MarkLogic allows this common scenario (a) to be configured from Admin GUI, and we will discuss the Certificate featuring (b) and (c) deployment further.

Default Admin GUI based Configuration 

By default, MarkLogic generates Temporary Certificate for all the nodes in the group for current cluster when Template is assigned to MarkLogic Server ( Exception is when Template assignment is done through XQuery ).

The Temporary Certificate generated for each node do have hostname as CN field for their respective Temporary Certificate - designed for common Secnario (a).

We have two path to install CA signed Certificate in MarkLogic

1) Generate Certificate request, get it signed by CA, import through Admin GUI

or 2) Generate Certificate request + Private Key outside of MarkLogic, get Certificate request signed by CA, import Signed Cert + Private Key using Admin script

Problem Scenario

In both of the above cases, while Installing/importing Signed Certificate, MarkLogic will look to replace Temporary Certificate by comparing CN field of Installed Certificate with Temporary Certificaet CN field.

Now, if we have WildCard Certificate (b) or SAN Certificate (c), our Signed Certificate's CN field will never match Temporary Certificate CN field, hence MarkLogic will Not remove Temporary Certificates - MarkLogic will continue using Temporary Certificate.

Solution

After installing SAN or wildcard Certificate, we may run into AppServer which still uses Temporary installed Certificate ( which was not replaced while installing SAN/wild-card Certificate).

Use below XQuery against Security DB to remove all Temporary Certificates. XQuery needs uri lexicon to be enabled (default enabled). [Please change the Certificate Template-Name in below XQuery to reflect values from your environment.] 

xquery version "1.0-ml";

import module namespace pki = "http://marklogic.com/xdmp/pki"  at "/MarkLogic/pki.xqy";
import module namespace admin = "http://marklogic.com/xdmp/admin"  at "/MarkLogic/admin.xqy";
      

let $hostIdList := let $config := admin:get-configuration()
                   return admin:get-host-ids($config)
                     
for $hostid in $hostIdList
return
  (: FDQN name matching Certificate CN field value :)
  let $fdqn := "TestDomain.com"

  (: Change to your Template Name string :)
  let $templateid := pki:template-get-id(pki:get-template-by-name("YourTemplateName"))

  for $i in cts:uris()
  where 
  (   (: locate Cert file with Public Key :)
      fn:doc($i)//pki:template-id=$templateid 
      and fn:doc($i)//pki:authority=fn:false()
      and fn:doc($i)//pki:host-name=$fdqn
  )
  return <h1> Cert File - {$i} .. inserting host-id {$hostid}
  {xdmp:node-insert-child(doc($i)/pki:certificate, <pki:host-id>{$hostid}</pki:host-id>)}
  {
      (: extract cert-id :)
      let $certid := fn:doc($i)//pki:certificate/pki:certificate-id
      for $j in cts:uris()
      where 
      (
          (: locate Cert file with Private key :)
          fn:doc($j)//pki:certificate-private-key/pki:template-id=$templateid 
          and fn:doc($j)//pki:certificate-private-key/pki:certificate-id=$certid
      )
      return <h2> Cert Key File - {$j}
      {xdmp:node-insert-child(doc($j)/pki:certificate-private-key,
        <pki:host-id>{$hostid}</pki:host-id>)}
      </h2>
  } </h1>

Above will remove all Temporary Certificates (including Template CA) and their private-key, leaving only Installed Certificate associated with Template, forcing all nodes to use Installed Certificate. 

Example: SAN (Subject Alternative Name) Certificate

For 3 node cluster (engrlab-128-101.engrlab.marklogic.com, engrlab-128-164.engrlab.marklogic.com, engrlab-128-130.engrlab.marklogic.com)

Example: Wild-Card Certificate

For 3 node cluster (engrlab-128-101.engrlab.marklogic.com, engrlab-128-164.engrlab.marklogic.com, engrlab-128-130.engrlab.marklogic.com). 

Introduction

Okta provides secure identity management and single sign-on to any application, whether in the cloud, on-premises or on a mobile device.

The following procedure describes the procedure required to integrate MarkLogic with Okta identity management and Microsoft Windows Active Directory using the Okta AD Agent.

This document assumes that the users accessing MarkLogic are defined in the Windows Active Directory only and do not currently have Okta User Profiles defined.

Authentication Flow

 The authentication flow in this scenario will be as follows:

1. The user opens a Browser connection to the Site Single Sign-On Portal page.
2. The user enters their Active Directory credentials
3. Okta verifies the user credentials using the Okta LDAP Agent
4. If successful, the user is presented with a selection of applications they can sign-on to.
5. The user selects the required application and Okta completes the sign-on using the stored user credentials.

Requirements

• MarkLogic Server version 8 or 9
• Okta Admin account access
• Okta AD Agent
• Active Directory Server

For the purpose of this document the following Active Directory user entry will be used as an example:

# LDAPv3
# base <dc=MarkLogic,dc=Local> with scope subtree
# filter: (sAMAccountName=martin.warnes)
# requesting: *
#

# Martin Warnes, Users, marklogic.local
dn: CN=Martin Warnes,CN=Users,DC=marklogic,DC=local
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: Martin Warnes
sn: Warnes
givenName: Martin
distinguishedName: CN=Martin Warnes,CN=Users,DC=marklogic,DC=local
sAMAccountName: martin.warnes
memberOf: CN=mladmins,CN=Users,DC=marklogic,DC=local
sAMAccountType: 805306368
userPrincipalName: martin.warnes@marklogic.local

Notes

1. By default, Okta uses the email address as the username, however, MarkLogic usernames cannot contain certain special characters such as the @ symbol so the sAMAccountName will be used to sign-on on to MarkLogic. This will be configured later during the Okta Application definition.
2. One or more memberOf attributes should be assigned to the Active Directory user entry and these will be used to assign MarkLogic Roles without requiring the need to configure duplicate user entries in the MarkLogic security database.

Step 1. Create a MarkLogic External Security definition

 An External Security definition is required to authenticate and authorize Okta users against a Microsoft Windows Active Directory server.

 Full details on configuring an external security definition can be found at:

 https://docs.marklogic.com/8.0/guide/security/external-auth

 You should ensure that both “authentication” and “authorization” are set to “ldap”, for details on the remaining settings you should consult your Active Directory administrator.

Step 2. Assign Active Directory group membership to MarkLogic Roles

In order to assign the correct Roles and Permission to Okta users, you will need to map Active Directory memberOf attributes to MarkLogic rolls.

In my example Active Directory user entry martin.warnes belongs to the following Group:

 memberOf: CN=mladmins,CN=Users,DC=marklogic,DC=local

To ensure that all members of this Group are assigned MarkLogic Admin roles you simply need to add the memberOf attribute value as an external name in the admin role as below:

Step 3. Configure the MarkLogic AppServer

For each App Server that you wish to integrate with Okta, you will need to set the “authentication” to “basic” and select the “external security” definition.

As HTTP Basic Authentication is considered insecure it is highly recommended that you secure the AppServer connection using HTTPS by configuring and selecting a “SSL certificate template”.

 Further details on configuring SSL for AppServers can be found at:

 https://docs.marklogic.com/8.0/guide/admin/SSL

Step 4. Install and Configure Okta AD Integration

In order for Okta to authenticate your Active Directory users, you will first need to download and install the Okta AD Agent using the following instructions supplied by Okta

https://support.okta.com/help/Documentation/Knowledge_Article/Install-and-Configure-the-Okta-Active-Directory-Agent-1689483166

 Once installed your Okta Administrator will be able to complete the AD Agent configuration to select which AD users to import into Okta.

Step 5. Create Okta MarkLogic application

From the Okta Administrator select “Add Application”, search for the Basic Authentication template and click “Add”

On the “General Settings” tab, enter the MarkLogic AppServer URL, ensure to use HTTP or HTTPS depending on whether you have chosen to secure the listening port using TLS.

 Check the “Browser plugin auto-submit” option.

On the Sign-On options panel select “Administrator sets username, password is the same as user’s Okta password”

 For “Application username format” select “AD SAM Account name” from the drop-down selection.

Once the Okta application is created you should assign the users permitted to access the application

When assigning a user, you will be prompted to check the AD Credentials, at this point you should just check that Okta has selected the correct "sAMAccountName" value, the password will not be modifiable.

Repeat Step 5. for each AppServer you wish to access via the Okta SSO portal.

Step 6. Sign-on to Okta SSO Portal

All assigned MarkLogic applications should be shown:

Selecting one of the MarkLogic applications should automatically log you in using your AD Credentials stored within Okta.

Additional Reading

• MarkLogic Documentation: Assigning external LDAP group membership to a Role

Introduction

MarkLogic server provides pre-commit or post-commit triggers and these triggers listens for certain events to occur and then invokes a configured XQuery module to run after event occurs. It is a common use case to create a common function in a library module which is shared among different trigger modules called by various triggers. This article shows an example to create and use such a shared library module in a post-commit trigger.

Example

This example shows a simple post commit trigger that fires when a new document is created.

1. For this example create a database 'minidb' and after that set its triggers database as self (minidb). Also, create another database 'minimodule' to store all modules.

2. Using Query Console, create a trigger using trigger definition by evaluating below XQuery against triggers database (minidb)

3. Create a module by running below XQuery against modules database:

4. Insert a library module into the modules database (minimodules):

5. Now insert the sample document into the content database (minidb):

6. Check output in logs:

After a new document having its URI prefixed with "/mini" is inserted into the content database, TaskServer Logs file logs the below message:

2018-04-25 11:40:50.224 Info: *****Document with /mini root /mini/test-25-1-1.xml was created.*****2018-04-25T11:40:50+05:30

NOTE: Module imports are relative to root.

References:

1. Creating and Managing Triggers With triggers.xqy - https://docs.marklogic.com/guide/app-dev/triggers

Introduction

We are always looking for ways to understand and address performance issues within the product and we are addressing this by adding the following new diagnostic features to the product.

New Trace Events in MarkLogic Server

Some new diagnostic trace events have been added to MarkLogic Server:

• Background Time Statistics - Background thread period and further processing timings are added to xdmp:host-status() output if this trace event is set.
• Journal Lag 30 - A forest will now log a warning message if a frame takes more than 30 seconds to journal.
  • Please note that this limit can be adjusted down by setting the Journal Lag # trace event (where # is {1, 2, 5 or 10} seconds).
• Canary Thread 10 - A new "canary thread" that does nothing but sleep for a second and check how long is was since it went to sleep.
  • It will log messages if the interval between sleeping has exceeded 10 seconds.
  • This can be adjusted down by setting the Canary Thread # trace event (where # is {1, 2, 5 or 10} seconds).
• Canary Thread Histogram - Adding this trace event will cause MarkLogic to write to the ErrorLog a histogram of timings once every 10 minutes.
• Forest Fast Query Lag 10 - By default, a forest will now warn if the fast query timestamp is lagging by more than 30 seconds.
  • This can be adjusted down by setting the Forest Fast Query Lag # (where # is {1, 2, 5, or 10} seconds).
  • Note that Warning level messages will be repeatedly logged at intervals while the lag limit is exceeded, with the time between logged messages doubling until it reaches 60 seconds.
  • There will be a final warning when the lag drops below the limit again as a way to bracket the period of lag.

Examples of some of new statistics can be viewed in the Admin UI by going to the following URL in a browser (replacing hostname with the name of a node in your cluster and replacing TheDatabase with the name of the database that you would like to monitor):

• http://hostname:8001/forest-insert-statistics.xqy?database=TheDatabase
• http://hostname:8001/host-background-statistics.xqy
• http://hostname:8001/forest-journal-statistics.xqy?database=TheDatabase

You can clear the forest insert and journal statistics by adding clear=true to your request; executing the following in a browser:

• http://hostname:8001/forest-insert-statistics.xqy?database=TheDatabase&clear=true
• http://hostname:8001/forest-journal-statistics.xqy?database=TheDatabase&clear=true

These changes now feature in the current releases of both MarkLogic 7 and MarkLogic 8 and are available for download from our developer website:

• http://developer.marklogic.com/products
• http://developer.marklogic.com/products/marklogic-server/7.0

Hints for interpreting new diagnostic pages

Here's some further detail on what the numbers mean.

First, a note about how bucketing is performed on these diagnostic pages:

For each operation category (e.g. Timestamp Wait, Semaphore, Disk), the wait time will fall into a range of values, which need to be bucketed.

The bucketing algorithm starts with 1000 buckets to cover the whole range, but then collapses them into a small set of buckets that cover the whole span of values. The algorithm aims to

1. End up with a small number of buckets

2. Include extreme (out-liers) values

3. Spread out multiple values so that they are not too "bunched-up" and are therefore easier to interpret.

Forest Journal Statistics (http://hostname:8001/forest-journal-statistics.xqy?database=TheDatabase)

When we journal a frame, there are a sequence of operations.

1. Wait on a semaphore to get access to the journal.
2. Write to the journal buffer (possibly waiting for I/O if exceeding the 512k buffer)
3. Send the frame to replica forests
4. Send the frame to journal archive/database replica forests
5. Release the semaphore so other threads can access the journal
6. Wait for everything above to complete, if needed.
   1. If it's a synchronous op (e.g. prepare, commit, fast query timestamp), we wait for disk I/O
   2. If there are replica forests, we wait for them to acknowledge that they have journaled and replayed.
   3. If the journal archive or database replica is lagged, wait for it to no longer be lagged.

We note the wall clock time before/after these various options, so we can track how long they're taking.

On the replica side, we also measure the "Journal Replay" time which would be inserting into the in-memory stand, committing, etc.

Here's an example for a master and its replica.

Forest F-1-1

Timestamp Wait

Semaphore

Disk

Local-Disk Replication

Journal Archive

No Information

Database Replication

No Information

Journal Total

Journal Replay

No Information

Forest F-1-1-R

Timestamp Wait

No Information

Semaphore

Disk

Local-Disk Replication

No Information

Journal Archive

No Information

Database Replication

No Information

Journal Total

Journal Replay

Forest Insert Statistics (http://hostname:8001/forest-insert-statistics.xqy?database=TheDatabase)

When we're inserting a fragment into an in-memory stand, we also have a sequence of operations.

1. Wait on a semaphore to get access to the in-memory stand.
2. Wait on the insert throttle (e.g. if there are too may stands)
3. Wait for the stand's journal semaphore, to serialize with the previous insert if needed.
4. Release the stand insert semaphore.
5. Journal the insert.
6. Release the stand journal semaphore.
7. Start the checkpoint task if the stand is full.

As with the journal statistics, we note the wall clock time between these operations so we can track how long they're taking.

On the replica side, the behavior is similar, although the journal and insert are in reverse order (we journal before inserting into the in-memory stand). If it's a database replica forest, we also have to regenerate the index information (Filled IPD).

Here is a example for a master and its replica.

Forest F-1-1

Journal Throttle

Insert Sem

Filled IPD

No Information

Stand Throttle

Stand Insert

Journal Sem

Journal

Total

Forest F-1-1-R

Journal Throttle

No Information

Insert Sem

Filled IPD

No Information

Stand Throttle

Stand Insert

Journal Sem

Journal

No Information

Total

Further reading

To learn more about diagnostic trace events, please refer to our documentation and Knowledgebase articles and note that some trace events may only log information if logging is set to debug:

• https://docs.marklogic.com/guide/cpf/error#id_38147
• https://help.marklogic.com/knowledgebase/article/View/46/19/how-to-use-diagnostic-trace-events

Summary

The jemalloc library is included with the MarkLogic install and is recommended to use as it has shown a performance boost over the default Linux malloc library.  It is included with the MarkLogic server install and is configured to be used by default. 

There have been cases where even if configured, the library is not used.  This article will give possible solutions to debug that.

Diagnostics

ErrorLog message on startup if jemalloc is not allocated:

Warning: Memory allocator is not jemalloc; check /etc/sysconfig/MarkLogic

Solutions

1) Make sure to use superuser shell or sudo and run the 'service MarkLogic restart'

2) Verify that the jemalloc library is present in the install directory (ie /opt/MarkLogic/lib/libjemalloc.so.1).

3) Has the /etc/sysconfig/MarkLogic configuration file been modified from the default?  Try setting the configuration file back to the default and restarting the server.

Details

For more information on the jemalloc library, please review the article provided by Facebook Engineering

https://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919/

Introduction

How is native JSON different than the previous JSON support?

Introduction

MarkLogic Server provides a couple of useful techniques for keeping values in memory or resolving values without having to scan for documents on-disk.

Options

There are a few options available:

1. cts:element-values performs a lexicon lookup so it's directly getting those values from the range indexes; you can add an options node and use the "map" parameter to get the call to return a map directly as per the documentation, which may give you what you need without having to do any further work.

See: http://docs.marklogic.com/cts:element-values

2. Storing a map as a server field is a popular approach and is widely used for storing data that needs to be accessed routinely by queries.

Bear in mind that there is a catch to this approach as the map is not available to all nodes in a cluster - it is only available to the node responsible for evaluating the original request, so if you're using this technique in a clustered environment, the results may not be what is expected.

Also note that if you're planning on storing a large number of maps in server fields on nodes on the cluster, it's important to make sure the hosts are provisioned with enough memory to accommodate these maps on top of group level caches and memory for query allocation, stands, range indexes document retrieval and the like.

See: http://docs.marklogic.com/map:map

And: http://docs.marklogic.com/xdmp:set-server-field

3. xdmp:set only allows you to set a value for the life of a single query but this technique can be useful in some circumstances - especially in situations where you're interested in keeping track of certain values throughout the processing of a module or a function within a module.

See: http://docs.marklogic.com/xdmp:set

4. If you have a situation where you have a large number of complex queries - particularly ones where lexicon lookups or calls to range indexes won't resolve the data you need and where lots of documents will need to be retrieved from disk, you should consider using registered queries.

See: http://docs.marklogic.com/cts:registered-query

Note that registered queries utilise the List Cache so, if you plan to adopt this method, we recommend careful testing to ensure your caches are sized sufficiently to suit the needs of your application.

Introduction

MarkLogic Server allows you to configure MarkLogic Server so that users are authenticated using an external authentication protocol, such as Lightweight Directory Access Protocol (LDAP) or Kerberos. These external agents serve as centralized points of authentication or repositories for user information from which authorization decisions can be made. If, after following the configuration instructions in our documentation, the authentication does not work as expected, this article gives some additional debugging ideas.

Details

The following are areas should be checked when your LDAP Authentication is not working as expected:

1. Verify that cyrus-sasl-md5 library is installed on MarkLogic Server node.

2. Run the following LDAP search command to check if LDAP server is properly setup.

ldapsearch -H ldap://{Your LDAP Serevr URI}:389 -x -s base

a. Once you run the ldap search command, make sure digest-md5 is supported. 

supportedSASLMechanisms: DIGEST-MD5

b. Identify the correct LDAP Service name:

e.g ldapServiceName: MLTEST1.LOCAL:dc1$@MLTEST1.LOCAL

3. On Windows platforms, the services.keytab file is created using Active Directory Domain Services (AD DS) on a Windows server. If you are using Active Directory Domain Services (AD DS) on a computer that is running Windows Server 2008 or Windows Server 2008 R2, be sure that you have installed the hot fix described in http://support.microsoft.com/kb/975697.

Introduction: the issue

MarkLogic performs Nested lookups on the LDAP Groups assigned to a user to determine which roles the user will be assigned. If the groups belong to multiple Active Directory Domains within a federated Active Directory Forest then MarkLogic user authorization could fail with a subordinate Referral error, as seen below:

2019-07-30 13:27:23.002 Notice: XDMP-LDAP: ldap_search_s failed on ldap server ldap://ad1.myhost.com:389: Referral (10)

Cause

MarkLogic has been configured to connect to the Local Domain Controller LDAP ports 389 (LDAP) or 636 (LDAPs), however, a Local Domain Controller can only search domains to which it has access.

Example

A user is a member of the following groups which belong to two separate Active Directory domains, subA, and subC.

Using a Local Domain Controller for subA for external authorization would result in a login failure when attempting to perform the nested group lookup for the domain subC

member=CN=Group Onw,OU=OrgUnitAGroups,OU=OrgUnitA,DC=subA,DC=domain
member=CN=Group Two,OU=OrgUnitAGroups,OU=OrgUnitA,DC=subA,DC=domain
member=CN=Group Three,OU=OrgUnitCGroups,OU=OrgUnitC,DC=subC,DC=domain

Solution

If you have multiple Active Directory Domains federated into an Active Directory forest you should use the Global Catalog port 3278 (LDAP) or 3279 (LDAPS) to prevent failures when searching for group memberships that are defined in other domains.

Optional workaround

A large number of nested groups can potentially lead to a decrease in login time performance, if you do not need to really on nested lookups to determine group membership for MarkLogic roles, i.e. all groups required are returned from the initial user search request then you should consider disabling setting the "ldap nested lookup" parameter to false in the External Security configuration.

Doing this would also prevent subordinate domain searches and allow you to continue to use an Active Directory Domain Controller instead of switching to the Global Catalog.

Further reading

• Microsoft Technet article on Global Catalog and LDAP searches

Summary

A leap second, as defined by wikipedia is "a one-second adjustment that is occasionally applied to Coordinated Universal Time (UTC) in order to keep its time of day close to the mean solar time. Without such a correction, time reckoned by Earth's rotation drifts away from atomic time because of irregularities in the Earth's rate of rotation."  At the time of this writing, the next leap second to be inserted is on June 30, 2015 at 23:59:60 UTC.

For systems that use the Network Time Protocol (NTP) to synchronize the network time across all the host in their MarkLogic Cluster, the Marklogic Server Software is not impacted by the leap second (i.e. we expect everything to work fine at the MarkLogic layer)

For systems where the synchronization of their system clocks require UTC time to be set backwards, then anywhere time dependent data is stored, it must be accounted for. In this case, we recommend that our customers implement NTP in their environment.  Otherwise, the application layer will need to handle discontinuous time. 

Transactional Consistency

The algorithm that MarkLogic Server uses to maintain transactional consistency of data is not wall clock dependent and, as such, is not affected by the leap second.

Network Time Protocol (NTP)

NTP generally works really really hard not to make time go backwards as clock readings are constrained to always increase - every reading increases the NTP clock. NTP adjusts things gradually by slowing down or speeding up the clock and not by making discrete changes unless time is off by a lot. A second is not a lot.  An hour is a lot. Regardless of the leap second, adjustments for computer clock drift can easily be more than a second and happen frequently. 

When Time Goes Backwards

Without NTP and left on their own, computer clocks are really not that accurate. If synchronization of the system clocks on the hosts of a MarkLogic cluster require the clocks to be set backwards, then the application layer will need to account for and handle discontinuous date-time in their data. 

Beginning with MarkLogic Server version 8,  the temporal feature was introduced.  If the system clock is adjusted backwards, there are conditions where temporal document inserts and updates will fail with an appropriate error code.  This is by design and expected.

Our recommendation is to implement NTP on all hosts of a MarkLogic cluster to eliminate the need to handle discontinuous time at the application layer. 

Further Reading

Redhat article on the Leap Second - https://access.redhat.com/articles/15145 ;

Microsoft Support article on the Leap Second - http://support.microsoft.com/kb/909614 ;

Summary

The internal mechanisms MarkLogic Server uses to implement security are query constraints. Lexicon search performance may be impacted by security query contraints.  If performed with admin credentials, Lexicon searches will not be impacted by the security query constraints.  

Detail

Query time grows proportionately with the number of matches from a given search across a set of documents (not the actual number of documents in your database). The presence of security constraints will contribute a significantly larger number of matches than if the same lexicon search was performed with admin credentials.  In order to minimize the number of matches (and therefore query time) for a given lexicon search, you'll want to amp your lexicon searches to an admin user.

Summary

MarkLogic recommends the default "ordered" option for Linux ext3 and ext4 file-systems.

File System administrators in Linux are tempted to use the data=writeback option to achieve higher throughput from their file-system, but this comes with the side-effects of potential data corruption and data-secuity breach. This article explains both file system options with respect to MarkLogic Server. 

"data=ordered"

Linux ext3 and ext4 file system has default data option of "ordered", which writes to the main file system before committing to the journal.

https://www.kernel.org/doc/Documentation/filesystems/ext4.txt

https://www.kernel.org/doc/Documentation/filesystems/ext3.txt

Both of these file-system goes the extra mile to protect your files and writes data associated with that meta data by default with data=ordered, thus assuring file-system integrity to application layer - essential for MarkLogic Server data integrity. 

"data=writeback"

Other journaled file systems like XFS and JFS write meta data to the disk;  to make ext3 and ext4 behave like XFS and other journal file system, an administrator could set 'data=writeback' in their mount options.

The 'data=writeback' mode does not preserve data ordering when writing to the disk, so commits to the journal may happen before the data is written to the file system. This method is faster because only the meta data is journaled, but is not good at protecting data integrity in the face of a system failure.

If there is a crash between the time when metadata is commited to the journal and when data is written to disk, the post-recovery metadata can point to incomplete, partially written or incorrect data on disk; which can lead to corrupt data files. Additionally, data which was supposed to be overwritten in the filesystem could be exposed to users - resulting in a security risk.

Linus Torvalds comments on 'data=writeback'

"it makes things much smoother, since now the actual data is no longer in the critical path for any journal writes, but anybody who thinks that's a solution is just incompetent.  We might as well go back to ext2 then. If your data gets written out long after the metadata hit the disk, you are going to hit all kinds of bad issues if the machine ever goes down."   - http://thread.gmane.org/gmane.linux.kernel/811167/focus=811654

Introduction

Here we discuss management of temporal documents.

Details

In MarkLogic, a temporal document is managed as a series of versioned documents in a protected collection. The ‘original’ document inserted into the database is kept and never changes. Updates to the document are inserted as new documents with different valid and system times. A delete of the document is also inserted as a new document.

In this way, a temporal document always retains knowledge of when the information was known in the real world and when it was recorded in the database.

API's

By default the normal xdmp:* document functions (e.g., xdmp:document-insert) are not permitted on temporal documents.

The temporal module (temporal:* functions; see Temporal API) contains the functions used to insert, delete, and manage temporal documents.

All temporal updates and deletes create new documents and in normal operations this is exactly what will be desired.

See also the documentation: Managing Temporal Documents.

Updates and deletes outside the temporal functions

Note: normal use of the temporal feature will not require this sort of operation.

The function temporal:collection-set-options can be used with the updates-admin-override option to specify that users with the admin role can change or delete temporal documents using non-temporal functions, such as xdmp:document-insert and xdmp:document-delete.

For example, if you need to do a corb or other administrative transform, but do not want to update the system dates on the documents; say, you want to change the values M/F to Male/Female.

Introduction

When CPF is installed, a number of new documents are created for the nominated Triggers database associated with that database.

This Knowledgebase article is designed to show you what CPF creates on install, in the event that you want to safely disable and remove it from your system.

Getting started

Below is a layout of all databases and their associated document counts with a clean install of MarkLogic 9.0-2: