Customizing MarkLogic on AWS with Packer and Terraform
31 August 2020 10:05 AM
Packer from HashiCorp is an open source provisioning tool, allowing for the automated creation of machine images, extending the ability to manage infrastructure to machine images. Packer supports a number of different image types including AWS, Azure, Docker, VirtualBox and VMWare.
These powerful tools can be used together to deploy a MarkLogic Cluster to AWS using the MarkLogic CloudFormation Template, using a customized Amazon Machine Image (AMI). The MarkLogic CloudFormation Template is the preferred method recommended by MarkLogic for building out MarkLogic clusters within AWS. By default the MarkLogic CloudFormation Template uses the official MarkLogic AMIs.
While this guide will cover a some portions of Terraform, the primary focus will be using Packer to customize an official MarkLogic AMI. For more detailed information on Terraform, we recommend reading Deploying MarkLogic to AWS with Terraform, which includes more detailed information on using Terraform, as well as the example files referenced later in this article.
Setting Up Packer
For the purpose of this example, I will assume that you have already installed the AWS CLI, with the correct credentials, and you have installed Packer.
A Packer template is a JSON configuration file that is used to define the image that we want to build. Templates have a number of keys available for defining the machine image, but the most commonly used ones are builders, provisioners and post-processors.
Creating a Template
For our example, we are going to take the official MarkLogic AMI and apply some customizations before creating a new image.
Variables help make the build more flexible, so we will utilize a seperate variables file,
Creating Our Template
Now that we have some of the specific build details defined, we can create our template,
In our example, we are using the
For our custom image, we've determined that we need an additional piece of software installed, which we will do inside a script. We've named the script
Executing Our Build
Now that we've completed setting up our build, it's time to use packer to create the image.
Here you can see that we are telling packer to do a build using
The last part of the build output will print out the details of our new image:
Terraform and the MarkLogic CloudFormation Template
At this point we have our image and want to use it when deploying the MarkLogic CloudFormation Template. Unfortunately there is no simple way to do this, as the MarkLogic CloudFormation Template does not have the option to specify a custom AMI. Fortunately Terraform has some functions available that we can use to make the changes to the Template.
First we want to add a couple entries to our existing Terraform variables file.
The first variable,
So we are filtering the available AMIs, only looking at ones that are owned by our own account (
Updates to Terraform Root Module
Now we are ready to make a couple of updates to our Terraform root module file to integrate the new AMI into our deployment. In our last example, we used the MarkLogic CloudFormation template from its S3 bucket. For this deployment, we are going to use a local copy of the template,
When we updated the variables in our Terraform variable file, we created the variable
Deploying with Terraform
Now we are ready to run Terraform to deploy our cluster. First we want to double check that the template looks correct before we attempt to create the CloudFormation stack. The output of terraform plan will show the CloudFormation template that will be deployed. Check the output to make sure that the value for ImageId shows our desired AMI
Once we have confirmed our new AMI is being referenced, we can then run terraform apply to create a new stack using the template. This can be validated by opening a command line on one of the new hosts, and checking to see if Git is installed, and if
At this point, we have now customized the official MarkLogic AMI to create our own AMI using Packer. We have then used Terraform to update the MarkLogic CloudFormation Template and to deploy a CloudFormation stack based on the updated template.