Knowledgebase: Errors
Understanding XDMP-DEADLOCK
04 February 2020 02:40 PM

Summary

Deadlocks occur when two transactions are each waiting to acquire a lock and neither can continue until the other releases a lock. The XDMP-DEADLOCK error log message indicates that MarkLogic Server detected a deadlock. When the XDMP-DEADLOCK error log message occurs at the ‘Debug’ log level, the deadlock was successfully resolved by MarkLogic Server. When the XDMP-DEADLOCK error log message occurs at the ‘Notice’ log level, the deadlock was not resolvable by MarkLogic.

Deadlock messages across MarkLogic versions:

MarkLogic 8.0-8.1
2019-10-15 11:10:30.161 Info: XDMP-DEADLOCK: Deadlock detected locking Documents /test.xml
2019-10-15 11:10:27.994 Info: XDMP-DEADLOCK: Deadlock detected locking Documents #10466205726835857951
2019-10-15 11:10:27.992 Info: XDMP-DEADLOCK: Deadlock detected locking Documents #10466205726835857951

MarkLogic 9.0-9.4
2019-10-15 11:14:36.544 Info: Deadlock detected locking Documents /test.xml
2019-10-15 11:14:34.414 Info: Deadlock detected locking Documents #10466205726835857951
2019-10-15 11:14:34.413 Info: Deadlock detected locking Documents #10466205726835857951

Details

MarkLogic Server is designed to automatically detect and resolve deadlocks. When a deadlock is detected, one of the deadlocked transactions is retried, allowing the other to acquire the lock and continue. When this expected behavior occurs, an XDMP-DEADLOCK is written to the e-node error log as a ‘Debug’ message to indicate that a deadlock occurred and was resolved.

If the deadlock cannot be resolved by repeated retries, an XDMP-DEADLOCK message is written to the e-node error log as a ‘Notice’ message.

Deadlocks are also reported at ‘Info’ level on the d-node on which they occur.

It is common for deadlocks to be the cause of a poor performing application.  The file log level needs to be set to the ‘debug’ level or higher in order to detect that the retryable deadlock conditions are occurring. You can set the file log level in the Admin UI by navigating to -> Configure -> Groups -> {group-name} -> Configure tab -> file log level = debug; press the “ok” button. 

Response

If XDMP-DEADLOCK appears as an infrequent ‘Debug’ message, no action is required. Deadlocks are a normal part of database operations, and the system successfully resolved the deadlock.

If XDMP-DEADLOCK appears frequently as a Debug message, you may have a performance issue. Revise your query or content structure to reduce the frequency of the deadlock.

If XDMP-DEADLOCK appears as a ‘Notice’ message, MarkLogic Server was unable to automatically resolve the deadlock. Examine the error message for details about the contentious resource. Revise your query or content structure to avoid the deadlock.

Diagnostics

You can obtain additional information in the error logs for debugging deadlocks

  1. Setting the "file log level" to finer will yield log messages showing what updates are doing. You can set the file log level in the Admin UI by navigating to -> Configure -> Groups -> {group-name} -> Configure tab -> file log level = finer; press the “ok” button. 
  2. The trace event "Locking" will give additional locking information in the error logs. You can add the "Locking" trace event in the Admin UI by navigating to -> Configure -> Groups -> {group-name} -> Diagnostics -> trace events activated = true; Add "Locking"; press the “ok” button. 
  3. You may also be able to identify the transaction request by correlating timestamps in the error logs with those in the access logs. 

Application Level Remediation

Once you have identified a transaction that references a common document, there are a couple of things that you can do to remediate XDMP-DEADLOCK issues.

  1. Re-architect your application so that it does not reference the common document.
  2. Isolate all updates to shared documents inside their own transactions. This can be done using xdmp:eval with isolation level of ‘different transaction’.
  3. Call the xdmp:lock-for-update() function at the beginning of the update transaction for the referenced documents. When called at the beginning of a transaction, xdmp:lock-for-update() locks the URI immediately, as opposed to later in the transaction as MarkLogic Server is a lazy evaluator. If there are multiple transactions doing a bunch of work and then attempting to lock the same URI, some of the transaction could be forced to redo work (as is often the case when you see the Debug level XDMP-DEADLOCK log message). By using xdmp:lock-for-update(), the lock is acquired by the first transaction immediately, and the other transactions wait to acquire the lock, resulting in no extra work being performed and some deadlock conditions are avoided. xdmp:lock-for-update() simply forces the update lock upfront and the lock will be released at the end of the transaction. 

Other Problematic Application Patterns

There are some application patterns that might lead to deadlocks in a system with concurrent queries:

  • Update transactions where the result of a search returns a large number of documents:  In this case, the update transaction will read-lock every matching document.  Some alternatives:
    • rewrite searches to return a small number of documents;
    • isolate the transaction context of updates from the search queries that return large result sets;
    • If you just want to get a list of URIs without locking you can use cts:uris instead of cts:search, as cts:uris does not lock.
  • Undetected deadlock
    • The most likely reason for an undetected deadlock is a request deadlocking against itself. This can happen when an update holding a lock then calls an xdmp:eval() or xdmp:invoke(), to invoke another nested update that tries to acquire the lock.
    • One way this shows up in the error logs is as a timeout on xdmp:lock-for-update()

                    Notice:  SVC-EXTIME: xdmp:lock-for-update(“/test-uri.xml") -- Time limit exceeded

Directory Creation

The “directory creation” database configuration set to ‘automatic’ is a common cause of deadlocks. For document inserts, the server has to lock all the directory (properties) fragments that match the document URI "path", which could cause deadlocks if other insert threads are trying to lock the same set of documents.

For example: If you have insert the document ‘/a/path/to/content.xml’, the server would create the following directories:

 /

/a/

/a/path/

/a/path/to/

These directories would have to be locked every time you want to put content into the "/a/path/to/" URI path. If you are experiencing deadlocks on directories, then you'll want to set directory creation to "manual". Note that "automatic" directory creation was the default setting in MarkLogic Server 5.0 and prior, and "automatic" is still required if the database will be attached to a WebDav Application Server. However, beginning in MarkLogic Server 6.0, directory creation was changed to "manual" by default.

Update Transactions, Locks and Lifetime

Update transactions do not run at a set timestamp (as Query transactions do).  Update transactions see the latest view of any given document at the time it is first accessed by any statement in the transaction. Because an update transaction must successfully obtain locks on all documents it reads or writes in order to complete evaluation, there is no chance that a given update transaction will see 'half' or 'some' of the updates made by other transactions (i.e. transactional integrity is enforced).

Within an update transaction, query operations require read locks and update operations require reader/writer locks. A reader/writer lock is an exclusive lock and cannot be acquired until any locks held by other transactions are released. A read lock is not exclusive. Once a lock is acquired, it is held until the transaction ends. This prevents other transactions from updating a read locked document and ensures a read-consistent view of the document. 

For a more thorough examination of MarkLogic transactions, refer to the "Understanding Transactions in MarkLogic Server" section of the MarkLogic Server Application Developer's Guide.

Further Reading


MarkLogic Knowlegebase: Understanding the Lock Trace diagnostic event

MarkLogic Documentation: Understanding Transactions in MarkLogic Server

MarkLogic Knowedgebase: How do updates work in MarkLogic Server?

MarkLogic Knowledgebase: Read only queries run at a timestamp and Update transactions use locks

 

(18 vote(s))
Helpful
Not helpful

Comments (0)