Knowledgebase:
Understanding the Logs During Rebalancer and Reindexer Activity
15 May 2023 06:01 PM

Introduction

MarkLogic Sever sends log messages to both the operating system log and the MarkLogic Server file log. This server log file is maintained as a simple text file. You may view/access the log files from the 'Log' tab on the main page of the Admin UI.

Each file is stored in the MarkLogic Server data directory for your platform. The default location for MarkLogic ErrorLog files for a Linux environment is /var/opt/MarkLogic/Logs and Windows is  C:\Program Files\MarkLogic\Data\Logs. The server gives an option of specifying the minimum log level message sent to the log file. Based on the application requirement, the default level can be changed by navigating to Admin UI -> Configure -> Groups -> {Select Group} -> 'file log level' to log additional details in the ErrorLog.txt.

This article gives information about how to interpret the logs with any index setting changes made on a database or when rebalancer activity is triggered.

 

Details

1) Database Merging activity

Database merges are a way of self-tuning the performance of the system, and MarkLogic Server continuously assesses the state of each database to see if it would benefit from self-tuning through a merge. As part of merging, multiple stands are combined for performance reasons, disk space is reclaimed and Indexes and lexicons are combined and re-optimized based on their new size. Since merges can be resource intensive (both disk I/O and CPU)   we recommend controlling when merges occur and/or when they do not occur as well watching for the unusual merge activity.

The Server provides multiple options to review the merging activity –

a. Monitoring History Dashboard

MarkLogic Server’s Monitoring History feature gives information about the merging activity over a period of time along with Merge read/write rates indicating the average of reading or writing merge data from disk.

b. The Database Status page

The Database Status page lists the merge state, which indicates if a merge is going on, shows the size of the merge, and estimates how long it will take the merge to complete.

 

c. MarkLogic Server ErrorLog.txt

MarkLogic Server logs INFO level messages to the ErrorLog.txt file whenever a merge begins, completes, or is canceled. Additionally, there are other log messages that are logged at more detail logging levels during a merge.  When there are any database index setting changes, the reindexer activity will start shortly and you will be able to see "Config" and "Debug" level log messages as shown below:

2015-06-22 10:34:50.503 Config: Wrote /var/opt/MarkLogic/databases.xml
2015-06-22 10:34:50.503 Config: Loading /var/opt/MarkLogic/databases.xml

2015-06-22 10:34:55.550 Debug: Detecting indexes for database Modules
2015-06-22 10:34:55.550 Debug: Detecting indexes for database Documents

During a merge, the merge rates are reported. The rate reported in the Merging status is the merge rate of all merges on the forest, averaged over the last few seconds. Low merge rates are usually an indication of not getting enough IO throughput to handle the load.

2015-03-12 12:56:06.633 Info: Merged 15 MB in 65 sec at 0 MB/sec to /opt/marklogic-forests/Forests/...
2015-03-12 12:56:18.083 Info: Merged 42 MB in 77 sec at 1 MB/sec to /opt/marklogic-forests/Forests/...
2015-03-12 12:56:27.754 Info: Merged 49 MB in 87 sec at 1 MB/sec to /opt/marklogic-forests/Forests/...

This Knowledge Base article on Server IO requirements will be helpful on how to provision the I/O capacity of their MarkLogic installation.

Also, if there’s continuous ingestion, the database will merge continuously and merging never ends. As a result, the logs won’t be able to give the total reindexing time. However, the Admin UI will give you an estimate based on a search estimate because it is designed to be a fast report.

 

2)Database rebalance activity

A database rebalancer consists of two parts: an assignment policy for data insert and rebalancing and a rebalancer for data movement. The rebalancer runs on each forest and consults the database's assignment policy to determine which documents do not 'belong to' this forest and then pushes them to the correct forests. In addition to the rebalancer periodically rebalancing the database, the following events trigger the rebalancer process:

  • Any configuration changes to the database, such as adding a new forest or retiring an existing forest.
  • Upon completion of a restore operation on the database.
  • Upon completion of a backup operation on the database.

The Server provides multiple options to review the rebalancer activity –

a. Database Status page –

When the rebalancer is enabled on the database, you can check the state of the rebalancer, along with an estimated completion time, on the Database Status page.

 

If the rebalancer is disabled, the Show Rebalance button on the Database Status page will give number of fragments that are pending rebalancing.

b. MarkLogic Server Logs -

When a rebalancer activity is started on a database, a "Debug" level alert is logged that looks like shown below:

2015-06-22 11:29:48.390 Debug: Rebalanced 10000 fragments in 7 sec at 1379 fragments/sec on forest Documents with the bucket policy.
2015-06-22 11:29:53.448 Debug: Rebalanced 10000 fragments in 5 sec at 1977 fragments/sec on forest Documents with the bucket policy.

Additionally, you may also use the 'Rebalancer State' trace event that gives status on the rebalancer running for all the forests:

2015-06-22 11:46:10.514 Info: [Event:id=Rebalancer State] The rebalancer on Documents is starting
2015-06-22 11:46:10.516 Info: [Event:id=Rebalancer State] The rebalancer on Documents starts running

3)Database reindex activity

a. Database Status page –

Reindexing work that is incomplete or pending (if the reindexer is disabled) can be checked in the Admin UI, at the database status page, similar to rebalancing.

b. MarkLogic Server Logs -

When reindexing, Debug-level messages will note progress in the reindexing, while a final Info-level message gives a total of the work done:

2023-05-15 17:07:06.057 Debug: Reindexed range-indexes 10000 fragments in 4 sec at 2391 fragments/sec on forest Documents-f1
2023-05-15 17:07:10.113 Debug: Reindexed range-indexes 10184 fragments in 4 sec at 2511 fragments/sec on forest Documents-f1
2023-05-15 17:07:11.014 Info: Reindexed range-indexes 22001 fragments in 9 sec at 2407 fragments/sec on forest Documents-f1

c. Reindexer Preview with xdmp:forest-counts -

You can also check the incomplete/pending work with a call to xdmp:forest-counts like

xdmp:forest-counts(xdmp:database-forests(xdmp:database("Documents")), (),("preview-reindexer"))

(2 vote(s))
Helpful
Not helpful

Comments (0)