Summary

Disk utilization is an important part of the hosts ecosystem.  The results of filling the file system can have disastrous effects on server performance and data integrity.  It is very important to ensure that your host always has an appropriate amount of free disk space. 

Detection

When the file system runs out of space there will be a dramatic decrease in query performance and merges will cease.  There will also be a number entries in the ErrorLog.txt file that look like these:

SVC-FILWRT: File write error: write 'filename': No space left on device

Error in merge of forest [Forest-Name]: XDMP-MERGESPACE: Not merging due to disk space limitations, need=xxxMB, have=xxxMB

Mitigation

The best practice is to ensure that the total physical disk space available is sufficient to store all your forest data.  If you happen into a situation where you are dangerously low on disk space the following methods can be used to correct the situation.

1. Move/Remove any unwanted files from the file system. This might include cleaning up log files that have grown very large.
2. Add additional storage to the host.  This may require that you move the location of forest data.  Please see Moving Forests Across Storage Devices for more details.

In the event that the data directory containing the forest data and the directory containing the MarkLogic config files are on the same partition (which is common on Windows installations), it is possible to encounter a unique situation. In this situation, if the file system is completely full, MarkLogic will be unable to write config files.  If this is the case, you will not be able to perform any task in the Admin UI.  For this situation you will need to manually move your forest data if you cannot free enough disk space to allow for writing of configuration files.  This is a special case and is not recommended for situation where the Admin UI can be used.  Please refer to "Moving Forests Across Storage Devices" for our recommended process.

Step 1. Stop MarkLogic

Step 2. Move the forest data to another location.  Be sure to maintain permissions assigned to the forest data.

Step 3. Start MarkLogic

Step 4.  Detach your forest from the DB

• In the Admin UI navigate to Configure -> Databases -> [Your-Database] -> Forests
• Uncheck the box next to your forest and click "OK".

Step 5. Create a new forest, specifying the new storage location

• In the Admin UI navigate to Configure -> Forests
• Click the "Create" tab and fill in the appropriate information

Step 6. Copy all data from the old forest directory into the new forest directory

Step 7.  Attach the new forest to the DB

• In the Admin UI navigate to Configure -> Database -> [Your-Database] -> Forests
• Check the box next tot the newly created database and click "ok".

Step 8. Ensure there were no errors while mounting the new forest

• In the Admin UI navigate to Configure -> Database -> [Your-Database] and click the status tab
• It is also a good idea to look for any errors in the error log (/var/opt/MarkLogic/Logs/ErrorLog.txt)

Step 9. Provided there were no issues with the new forest, you can now delete the old forest

• In the Admin UI navigate to Configure -> Forests -> [Old-Forest]
• On the "Configure" tab select delete.
• The old forest data will need to be deleted manually.

Database and Forest Size References:

MarkLogic Installation Guide: Disk Space Requirements 

Knowledge Base Article: Beginning in MarkLogic 7, the 3x disk space requirement can be reduced if configured and managed.