Configuration Migration of MarkLogic Server using Gradle and ml-gradle plug in
11 May 2021 02:32 PM
As the Configuration Manager has been deprecated starting with MarkLogic version 9.0-5, there is a common question on the ways how the configuration of database or an application server from an old version of MarkLogic instance to new version of MarkLogic server or between any two versions of MarkLogic server post 9.0-4
As a pre-requisite, have the compatible gradle (6.x) and the latest ml-gradle plugin(latest version is 4.1.1) installed and configured on the client (local machine or a machine from where the gradle project has to run) machine.
The entire process is divided into two major parts Exporting resource configuration from the source cluster and Importing the resource configuration onto the destination cluster.
1. Exporting resource configuration from the source cluster/host:
On the machine where gradle is installed and the plug-in is configured, create a project as suggested in https://github.com/marklogic-community/ml-gradle#start-using-ml-gradle
In the example steps below the source project is /Migration
1.1 Creating the new project with the source details:
While creating this new project, please provide the host MarkLogic server host, username, password, REST port, multiple environment details in the command line and once the project creation is successful, you can verify the Source server details in the gradle.properties file.
Once this build was successful, you can see the below directory structure created under the project directory:
1.2 Exporting the configuration of required resources:
Once the new project is created, export the required resources from the source host/cluster by creating a properties file(Not in the project directory but some other directory) as suggested in the documentation with all the resources details that need to be exported to the destination cluster. In that properties file, specify the names of the resources (Databases, Forests, app servers etc..)using the keys mentioned below with comma-delimited values:
For example, a sample properties file looks like below:
Once the file is created, run the below:
Once this build was successful, the below directory structure is created under the project directory which includes the list of resources that have been exported and their config files (Example screenshot below):
With this step finished, the export of required resources from the source cluster is created. This export is now ready to be imported with these configurations(resources) into the new/destination cluster.
2. Importing Resources and the configuration on new/Destination host/Cluster:
For importing resource configuration on to the destination host/cluster, again create a new project and use the export that has been created in step 1.2 Exporting the configuration of required resources. Once these configuration files are copied to the new project, make the necessary modification to reflect the new cluster (Like hosts and other dependencies) and then deploy the configuration into the new project.
2.1 Creating a new project for the import with the Destination Host/cluster details:
While creating this new project, provide the destination MarkLogic server host, username, password, REST port, multiple environment details in the command line and once the project creation is successful, please verify the destination server details in the gradle.properties file. In the example steps below the source project is /ml10pro
Once the project is created, you can observe the below directory structure created:
2.2 Copying the required configuration files from Source project to destination project:
In this step, copy the configuration files that have been created by exporting the resource configuration from the source server in step “ 1.2 Exporting the configuration of required resources”
After copying, the directory structure in this project looks like below:
Please make sure that after copying configuration files from source to destination, review each and every configuration file and make the necessary changes for example, the host details should be updated to Destination server host details. Similarly, perform any other changes that are needed per the requirement.
For example, under ~/ml10pro/src/main/ml-config/forests/<database>/<forestname>.xml file you see the entry:
"host" : "Sourceserver_IP_Adress",
change the host details to reflect the destination host details. So after changing, it should look like:
"host" : "Destination_IP_Adress",
Similarly, For each forest, please define the host details of the specific node that is required.
For example for forest 1, if it has to be on node 1, define forest1.xml with
"host" : "node1_host",
Similarly, any other configuration parameters that have to be updated, it has to be updated in that specific resource.xml file under the destination ml-config directory.
As this involves modifying the configuration files, it is advised to have back up and maintain version control(like GitHub or svn) to track back the modifications.
If there is a requirement to deploy the same configuration to multiple environments (like PROD, QA, TEST) all that is needed is to have gradle.properties files created for a different environment where this configuration needs to be deployed. As explained in step 2.1 Creating a new project for the import with the Destination Host/cluster details, the property values for different environments need to be provided while creating the project so that the gradle.properties file for different environments are created.
2.3 Importing the configuration (Running mlDeploy):
In this step, import the configuration that has been copied/exported from a resource. After making sure that the configuration files are all copied from the source and then modified for the correct host details and other required changes, run the below:
Once the build is successful, go to the admin console of the destination server and verify that all the required configurations have been imported from the source server.
For more information, refer to our documentation and knowledge base articles: