With the release of MarkLogic Server versions 8.0-8 and 9.0-4, detailing memory use broken out by major areas is periodically recorded to the error log. These diagnostic messages can be useful for quickly identifying memory resource consumption at a glance and aid in determining where to investigate memory-related issues.
Error Log Message and Description of Details
At one hour intervals, an Info level log message will be written to the server error log in the following format:
Info: Memory 46% phys=255137 size=136452(53%) rss=20426(8%) huge=97490(38%) anon=1284(0%) swap=1(0%) file=37323(14%) forest=49883(19%) cache=81920(32%) registry=1(0%)
The error log entry contains memory-related figures for non-zero statistics: raw figures are in megabytes; percentages are relative to the amount of physical memory reported by the operating system. Except for
phys, all values are for the MarkLogic Server process alone. The figures include
Memory: percentage of physical memory consumed by the MarkLogic Server process
phys: size of physical memory in the machine
size: total process memory for the MarkLogic process; basically huge+anon+swap+file on Linux. This includes memory-mapped files, even if they are not currently in physical memory.
swap: swap consumed by the MarkLogic Server process
rss: Resident Set Size reported by the operating system
anon: anonymous mapped memory used by the MarkLogic Server
file: total amount of RAM for memory-mapped data files used the MarkLogic Server---the MarkLogic Server executable itself, for example, is memory-mapped by the operating system, but is not included in this figure
forest: forest-related memory allocated by the MarkLogic Server process
cache: user-configured cache memory (list cache, expanded tree cache, etc.) consumed by the MarkLogic Server process
registry: memory consumed by registered queries
huge: huge page memory reserved by the operating system
join: memory consumed by joins for active running queries within the MarkLogic Server process
unclosed: unclosed memory, signifying memory consumed by unclosed or obsolete stands still held by the MarkLogic Server process
In addition to reporting once an hour, the Info level error log entry is written whenever the amount of main memory used by MarkLogic Server changes by more than five percent from one check to the next. MarkLogic Server will check the raw metering data obtained from the operating system once per minute. If metering is disabled, the check will not occur and no log entries will be made.
With the release of MarkLogic Server versions 8.0-8 and 9.0-5, this same information will be available in the output from the function
. . .
. . .
Additionally, with the release of MarkLogic Server 8.0-9.3 and 9.0-7, Warning-level log messages will be reported when the host may be low on memory. The messages will indicate the areas involved, for example:
Warning: Memory low: forest+cache=97%phys
Warning: Memory low: huge+anon+swap+file=128%phys
The messages are reported if the total memory used by the mentioned areas is greater than 90% of physical memory (
phys). As best practice for most use cases, the total of the areas should not be more than around 80% of physical memory, and should be even less if you are using the host for query processing.
file include memory-mapped files; for example, range indexes. Since the OS manages the paging in/out of the files, it knows and reports the actual RAM in use; MarkLogic reports the amount of RAM needed if all the mapped files were in memory at once. That's why MarkLogic can even report >100% of RAM in use---if all the memory-mapped files were required at once the machine would be out of memory.
Data Encryption Scenario: An encrypted file cannot be memory-mapped and is instead decrypted and read into anon memory. Since the file that is decrypted in memory is not file-backed it cannot be paged out. Therefore, even though encrypted files do not require more memory than unencrypted files, they become memory-resident and require physical memory to be allocated when they are read.
If the hosts are encountering these warnings, memory use should be monitored closely.
Remedial action to support memory requirements might include:
- Adding more physical memory to each of the hosts;
- Adding additional hosts to the cluster to spread the data across;
- Adding additional forests to any under-utilized hosts.
Other action might include:
- Archiving/dropping any older forest data that is no longer used;
- Reviewing the group level cache settings to ensure they are not set too high, as they make up the
cache part of the total. For reference, default (and recommended) group level cache settings based on common RAM configurations may be found in our Group Level Cache Settings based on RAM Knowledge base article.
This enhancement to MarkLogic Server allows for easy periodic monitoring of memory consumption over time, and records it in a summary fashion in the same place as other data pertaining to the operation of a running node in a cluster. Since all these figures have at their source raw Meters data, more in-depth investigation should start with the Meters history. However, having this information available at a glance can aid in identifying whether memory-related resources need to be explored when investigating performance, scale, or other like issues during testing or operation.
Knowledgebase: RAMblings - Opinions on Scaling Memory in MarkLogic Server