Knowledgebase: Performance Tuning
Sizing E-nodes
27 October 2015 03:58 PM

Introduction

The performance and resource consumption of E-nodes is determined by the kind of queries executed in addtion to the distribution and amount of data. For example, if there are 4 forests in the cluster and the query is asking for only the top-10 results, then the E-node would receive a total of 4 x 10 results in order to determine the top-10 among these 40. If there are 8 forests, then the E-node would have to sort through 8 x 10 results.

Performance Test for Sizing E-Nodes:

To size E-nodes, it’s best to determine first how much workload a single E-node can handle, and then scale up accordingly.

Set up your performance test so it is at scale and so that it only talks to a single E-node. Start the Application Server settings with something like

  • threads = 32
  • backlog = 512
  • keep alive = 0

Crank up the number of threads for the test from low to high, and observe the amount of resources being used on the E-node (cpu, memory, network). Measure both response time and throughput during these tests.

  • When the number of threads are low, you should be getting the best response time. This is what the end user would experience when the site is not busy.
  • When the number of threads are high, you will see longer response time, but you should be getting more throughput.

As you increase the number of threads, you will eventually run out of resources on the E-node - most likely memory. The idea is to identify the number of active threads when the system's memory is exceeded, because that is the maximum number of threads that your E-node can handle.

Addtitional Tuning of E-nodes

Thrashing

  • If you notice thrashing before MarkLogic is able to reach a  memory consumption equilibrium, you will need to continue decreasing the threads so that the RAM/thread ratio is near the 'pmap total memory'/thread
  • The backlog setting can be used to queue up requests w/o chewing up significant resources.
  • Adjusting backlog along with some of the timeout settings might give a reasonable user experience comparable to, or even better than, what you may see with high thread counts. 

As you continue to decrease the thread count and make other adjustments, the mean time to failure will likely increase until the settings are such that equilibrium is reached before all the memory resources are consumed - at which time we do not expect to see any additional memory failures.

Swap, RAM & Cache for E-nodes

  • Make sure that the E-nodes have swap space equal to the size of RAM (if the node has less than 32GB of RAM) or 32 GB (if the node has 32GB or more of RAM)
  • For E-nodes, you can minimize the List Cache and Compressed Tree Cache  - set to 1GB each - in your group level configurations.
  • Your Expanded Tree Cache (group level parameter) should be at least equal to 1/8 of RAM, but you can further increase the Expanded Tree Cache so that all three caches (List, Compressed, Expanded) in combination are up to 1/3 of RAM.
  • Another important group configuration parameter is Expanded Tree Cache Partitions.  A good starting point is 2-3 GB per partition, but is should not be more than 12 GB per partition. The greater the number of partitions, the greater the capacity of handling concurrent query loads.

Growing your Cluster

As your application, data and usage changes over time, it is important to periodically revisit your cluster sizings and re-run your performance tests.

 

(2 vote(s))
Helpful
Not helpful

Comments (0)