Problem Statement

You have an application running on a particular cluster (the source cluster), devcluster and you wish to port that application to an new cluster (the target cluster) testcluster. Porting the application can be divided into two tasks: configuring the target cluster and copying the code and data. This article is only about porting the configuration.

In an ideal world, the application is managed in an "infrastructure as code" manner: all of the configuration information about that cluster is codified in scripts and payloads stored in version control and able to be "replayed" at will. (One way to assure that this is the case is to configure testing for the application in a CI environment that begins by using the deployment scripts to configure the cluster.)

But in the real world, it's all too common for some amount of "tinkering" to have been performed in the Admin UI or via ad hoc calls to the Rest Management API (RMA). And even if that hasn't happened, it's not generally possible to be certain that's the case, so you still have to worry that it might have happened.

Migrating the application

The central theme in doing this "by hand" is that RMA payloads are re-playable. That is, the payload you GET for the properties of a resource is the same as the payload that you PUT to update the properties of that resource.

If you were going to migrate an application by hand, you'd proceed along these lines.

Determine what needs to be migrated

An application consists (more or less by definition) of one or more application servers. Application servers have databases associated with them (those databases may have additional database associations). Databases have forests.

A sufficiently complex application might have application servers divided into different groups of hosts.

Applications may also have users (for example, each application server has a default user; often, but not always, "​nobody​").

Users, in turn, have roles, and roles may have roles and privileges. Code may have amps that use privileges.

That covers most of the bases, but beware that apps can have additional configuration that should be reviewed: security artifacts (certificates, external securities, protected paths or collections, etc.), mime types, etc.

Get Source Configuration

Using RMA, you can get the properties of all of these resources:

• Application servers

  Hypothetically, the App-Services application server.

curl --anyauth -u admin:admin \
   http://localhost:8002/manage/v2/servers/App-Services/properties?group-id=Default

• Groups

  Hypothetically, the Default group.

curl --anyauth -u admin:admin \
   http://localhost:8002/manage/v2/groups/Default/properties

• Databases

  Hypothetically, the Documents database.

curl --anyauth -u admin:admin \
   http://localhost:8002/manage/v2/databases/Documents/properties

• Users

  Hypothetically, the ndw user.

curl --anyauth -u admin:admin \
   http://localhost:8002/manage/v2/users/ndw/properties

• Roles

  Hypothetically, the app-admin role.

curl --anyauth -u admin:admin \
   http://localhost:8002/manage/v2/roles/app-admin/properties

• Privileges

  Hypothetically, the app-writer execute privilege.

curl --anyauth -u admin:admin \
   "http://localhost:8002/manage/v2/privileges/app-writer/properties?kind=execute"

And the create-document URI privilege.

curl --anyauth -u admin:admin \
   "http://localhost:8002/manage/v2/privileges/create-document/properties?kind=uri"

• Amps

  Hypothetically, my-amped-function in /foo.xqy in the Modules
  database using the namespace http://example.com/.

curl --anyauth -u admin:admin \
   "http://localhost:8002/manage/v2/amps/my-amped-function/properties\
   ?modules-database=Modules\
   &document-uri=/foo.xqy\
   &namespace=http://example.com"

Create Target Configuration

Some of the properties of a MarkLogic resource may be references to other resources. For example, an application server refers to databases and a role can refer to a privilege. Consequently, if you just attempt to POST all of the property payloads, you may not succeed. The references can, in fact, be circular so that no sequence will succeed.

The easiest way to get around this problem is to simply create all of the resources using minimal configurations: Create the forests (make sure you put them on the right hosts and configure them appropriately). Create the databases, application servers, roles, and privileges. Create the amps. If you need to create other resources (security artifacts, mime types, etc.) create those.

Finally, PUT the property payloads you collected from the source cluster onto the target cluster. This will update the properties of each application server, database, etc. to be the same as the source cluster.

Related Reading

MarkLogic Documentation - Scripting Cluster Management

MarkLogic Knowledgebase - Transferring data between MarkLogic Server clusters

MarkLogic Knowledgebase - Best Practices for exporting and importing data in bulk

MarkLogic Knowledgebase - Deployment and Continuous Integration Tools