With the release of MarkLogic Server versions 8.0-8 and 9.0-4, detailing memory use broken out by major areas is periodically recorded to the error log. These diagnostic messages can be useful for quickly identifying memory resource consumption at a glance and aid in determining where to investigate memory-related issues.
Error Log Message and Description of Details
At one hour intervals, an Info level log message will be written to the server error log in the following format:
Info: Memory 18% phys=147456 virt=246146(166%) rss=27330(18%) anon=53794(36%) file=250(0%) forest=1021(0%) cache=40960(27%) registry=1(0%)
The error log entry contains memory-related figures for non-zero statistics: Raw figures are in megabytes; Percentages are relative to the amount of physical memory reported by the operating system. The figures include:
Memory: Percentage of physical memory consumed by the MarkLogic Server process;
phys: Size of physical memory in the machine ;
virt: Size of virtual address space reported by the operating system. This figure is often greater than 100%;
swap: The amount of swap consumed by the MarkLogic Server process;
rss: Resident Set Size reported by the operating system;
anon: Anonymous mapped memory used by the MarkLogic Server;
file: Total amount of memory-mapped data files used the MarkLogic Server. (The MarkLogic Server executable itself, for example, is memory-mapped by the operating system, but is not included in this figure.) ;
forest: Forest-related memory allocated by the MarkLogic Server process;
cache: User configured cache memory (list cache, expanded tree cache, etc) consumed by the MarkLogic Server process;
registry: Amount of memory consumed by registered queries;
huge: Huge page memory reserved by the operating system, and percentage comparing this to total physical memory;
join: Memory consumed by joins for active running queries within the MarkLogic Server process, and percentage comparing this to total physical memory;
unclosed: Unclosed memory, signifying memory consumed by unclosed or obsolete stands still held by the MarkLogic Server process, and percentage comparing this figure to total physical memory.
In addition to reporting once an hour, the Info level error log entry is written whenever the amount of main memory used by MarkLogic Server changes by more than five percent from one check to the next. MarkLogic Server will check the raw metering data obtained from the operating system once per minute. If metering is disabled, the check will not occur and no log entries will be made.
With the release of MarkLogic Server versions 8.0-8 and 9.0-5, this same information will be available in the output from the function
. . .
. . .
Additionally, with the release of MarkLogic Server 8.0-9.3 and 9.0-7, Warning-level log messages may be reported when the host is low on memory — the messages will indicate the areas involved, for example:
Warning: Memory low: forest+cache=97%phys
Warning: Memory low: huge+anon+swap+file=128%phys
The messages are reported if the total memory used by the mentioned areas is greater than 90% of physical memory (
phys). As best practice, the total of the areas should never be more than around 80% of physical memory, and should be even less if you are using the host for query processing.
If the hosts are regularly encountering these warnings, remedial action to support the memory requirements might include:
- Adding more physical memory to each of the hosts;
- Adding additional hosts to the cluster to spread the data across;
- Adding additional forests to any under-utilized hosts.
Other action might include:
- Archiving/dropping any older forest data that is no longer used;
- Reviewing the group level cache settings to ensure they are not set too high, as they make up the
cache part of the total. For reference, default (and recommended) group level cache settings based on common RAM configurations may be found in our Group Level Cache Settings based on RAM Knowledgebase article.
This enhancement to MarkLogic Server allows for easy periodic monitoring of memory consumption over time, and records it in a summary fashion in the same place as other data pertaining to the operation of a running node in a cluster. Since all these figures have at their source raw Meters data, more in-depth investigation should start with the Meters history. However, having this information available at a glance can aid in identifying whether memory-related resources need to be explored when investigating performance, scale, or other like issues during testing or operation.