Pitfalls Running MarkLogic Process as non-root user
30 September 2022 02:54 PM
|
|
IntroductionSome customers choose to run MarkLogic without the watchdog process running as root. As this is increasingly becoming a popular topic, there is an additional Knowledgebase article that discusses this in further detail: Knowledgebase: Start and Stop MarkLogic Server as Non-Root User The aim of this Knowledgebase article is to recommend some of the modifications you should consider making to the user that is taking the responsibility of running as the root process would have done. MarkLogic server's root process makes a number of OS-specific settings to allow the product to run optimally. If you choose to make these modifications, this article aims to provide you with enough information to ensure you can match the settings that the server changes. Points to consider
The root MarkLogic process is simply a restarter process, waiting the non-root (daemon) process to exit - and if the daemon process exits abnormally, for any reason, the root process will fork and exec another process under the daemon process. The root process runs no XQuery scripts, opens no sockets, and accesses no database files. We strongly recommend starting MarkLogic as root, and to let it switch to the non-root user on its own.When the server initializes, if it initialises with the default root process, it performs some privileged kernel calls to configure sockets, memory, and threads. For example:
MarkLogic Server will function if it isn't started as root, but it may not perform as well. Problems Seen by Customers running MarkLogic as a non-root user1. If non-root user account isn't able authenticate due to any underlying system issue, MarkLogic can't startup properly. This can result in an endless restart loop of MarkLogic Server. Getting startedYou should check the following settings which are configured by the root process when MarkLogic first starts. 1. maxproc soft limitThe maxproc soft limit is set to 1024 by default. In /etc/init.d/MarkLogic the following line raises the soft limit to match the hard limit for the current process heirarchy:
2. Ensure Huge Pages are assigned correctlyIf you see something like this in /var/log/messages
If you look in /etc/sysctl.conf, you should see (or add) a line:
Here the {gid} is the group id of the user that runs MarkLogic. Again, it would make sense to ensure that both users (whatever you're using in place of root and daemon) are able to do this. 3. Server HugePages calculations
4. Additional kernel parameters to be defined in /etc/sysctl.conf
The above values influence shared memory handling and these values are set automatically if MarkLogic runs with the default root/daemon settings. On Redhat (RHEL) these values are pre-defined but not on SuSE. We recommend these values should be updated in sysctl.conf anyway. First step: get the current PAGE_SIZE by the following cmd call:
With the PAGE_SIZE you can calculate kernel.shmall as per the instructions below:
And you can set kernel.shmmax and kernel.shmmni accordingly:
5. Configure vm.hugetlb_shm_groupIn case MarkLogic runs under a different user ID some more parameters needs to be added to /etc/sysctl.conf:
6. Configuring limitsYou can also set memory limits in /etc/security/limits.conf
7. Configure / increase the vm.max_map_countThe vm.max_map_count allows for the restriction of the number of individual VMAs (Virtual Memory Areas) that a particular process can use. A Virtual Memory Area is a contiguous area of virtual address space. The amount of VMAs a process is allowed to create as specified by the OS. By default, there are usually around 65530 memory map entries allowed per process. From the kernel documentation for max_map_count: This file contains the maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries. While most applications need less than a thousand maps, certain programs, particularly malloc debuggers, may consume lots of them, e.g., up to one or two maps per allocation. The default value is 65536. See: https://kernel.org/doc/Documentation/sysctl/vm.txt Our recommendation is that this value can be safely doubled or even quadrupled where modern hardware is taken into consideration:
For this step, this is more important for hosts that have a larger amount of RAM. If you are setting up hosts with 256GB RAM or greater, this change is really worth considering. 8. Configure SOMAXCONNLinux SOMAXCONN parameter defines the maximum number of backlog value MarkLogic process allowed to pass to socket listen. Different Linux platforms (RHEL/CentOS) or even different versions of Linux may have different default SOMAXCONN value.
MarkLogic default backlog value for application servers is 512; However, Linux platform have lower SOMAXCONN value than MarkLogic requested higher backlog, then MarkLogic requested backlog will not be respected by Linux. MarkLogic when started as root user, will go through each application server and find the max backlog value, and set the Linux SOMAXCONN value to match the highest backlog value.
One can set the SOMAXCONN to max value of any application server backlog value manually using below.
9. Socket buffer rmem_max and wmem_maxLinux parameter defines the max send buffer size (wmem) and receive buffer size (rmem) for TCP ports. In other word, this parameter set the amount of memory that is allocated for each TCP socket when it is opened or created while transferring files. For more efficient parallel job performance, MarkLogic sets buffer values based on platform hardware RAM size during the startup when started as a root as below.
One can set the rmem_max and wmem_max values for platform with RAM>32GB manually using below.
10. Linux swappiness and Dirty background ratioMarkLogic sets Linux swappiness and dirty background ratio parameters during startup. When starting as non-root, Linux swappiness and dirty background ratio should be set as per KB "Linux Swappiness". Further readingIt is recommended that you also read this Knowledgebase article which covers running MarkLogic as a non-root user: Knowledgebase - Start and Stop MarkLogic Server as Non-root User Our documentation also covers running the main MarkLogic process (daemon by default) as a different user: Documentation: Configuring MarkLogic Server on UNIX Systems to Run as a Non-daemon User | |
|