Deploying MarkLogic in AWS with Ansible

Summary

Ansible, owned by Red Hat, is an open source provisioning, configuration and application deployment tool that many organizations use to manage their infrastructure as code. Unlike options such as Chef and Puppet, it is agentless, utilizing SSH to communicate between servers. Ansible also does not need a central host for orchestration, it can run from nearly any server, desktop or laptop. It supports many different platforms and services allowing for the deployment and configuration of on-site physical infrastructure, as well as cloud and virtual infrastructure such as AWS, Azure, VSphere, and more.

Ansible uses YAML as its configuration management language, making it easier to read than other formats. Ansible also uses Jinja2 for templating to enable dynamic expressions and access to variables.

Ansible is a flexible tool can be used to deploy a MarkLogic Cluster to AWS using the MarkLogic CloudFormation Template. The MarkLogic CloudFormation Template is the preferred method recommended by MarkLogic for building out MarkLogic clusters within AWS.

Setting Up Ansible

For the purpose of this example, I will assume that you have already installed Ansible, the AWS CLI, and the necessary python packages needed for Ansible to talk to AWS. If you need some help getting started, Free Code Camp has a good tutorial on setting up Ansible with AWS.

Inventory Files

Ansible uses Inventory files to help determine which servers to perform work on. They can also be used to customize settings to indiviual servers or groups of servers. For our example, we have setup our local system with all the prerequisites, so we need to tell Ansible how to treat the local connections. For this demonstration, here is my inventory, which I've named hosts

[local]

localhost              ansible_connection=local

Ansible Modules

Ansible modules are discreet units of code that are executed on a target. The target can be the local system, or a remote node. The modules can be executed from the command line, as an ad-hoc command, or as part of a playbook.

Ansible Playbooks

Playbooks are Ansible's configuration, deployment and orchestration language. Playbooks are how the power of Ansible, and its modules is extended from basic configuration, or manangment, all the way to complex, multi-tier infrastructure deployments.

Chosing a Template

MarkLogic provides two templates for creating a managed cluster in AWS.

1. MarkLogic cluster in new VPC
2. MarkLogic cluster in an existing VPC

I've chosen to deploy my cluster to an VPC. When deploying to an existing VPC, you will need to gather the VPC ID, as well as the Subnet IDs for the public and private subnets.

Defining Variables

The MarkLogic CF Template takes a number of input variables, including the region, availability zones, instance types, EC2 keys, encryption keys, licenses and more. We have to define our variables so they can be used as part of the resource.

Variables in Ansible can be declared in a separate file, which allows for deployment flexibility.

Here is a snippet from our variables file:

# vars file for marklogic template and version

ml_version: '10.0-latest'

template_file_name: 'mlcluster.template'

template_base_url: 'https://marklogic-template-releases.s3.amazonaws.com/'

# CF Template Deployment Variables

aws_region: 'us-east-1'

stack_name: 'Dev-Cluster-An3'

IAMRole: 'MarkLogic'

AdminUser: 'admin'

...

Using the CloudFormation Module

So now we need to create our playbook, and choose the module that will allow us to deploy a CloudFormation template to create a new stack. The cloudformation module allows us to create a CloudFormation stack.

Next, we need to define the cloudformation configuration options, setting the variables that will be passed in when the stack is created.

# Use a template from a URL

- name: Ansible Test

  hosts: local

  vars_files:

    - ml-cluster-vars.yml

  tasks:

    - cloudformation:

        stack_name: "{{ stack_name }}"

        state: "present"

        region: "{{ aws_region }}"

        capabilities: "CAPABILITY_IAM"

        disable_rollback: true

        template_url: "{{ template_base_url+ml_version+'/'+ template_file_name }}"

      args:

        template_parameters:

          IAMRole: "{{ IAMRole }}"

          AdminUser: "{{ AdminUser }}"

          AdminPass: "{{ AdminPass }}"

          Licensee: "{{ Licensee }}"

          LicenseKey: "{{ LicenseKey }}"

          KeyName: "{{ KeyName }}"

          VolumeSize: "{{ VolumeSize }}"

          VolumeType: "{{ VolumeType }}"

          VolumeEncryption: "{{ VolumeEncryption }}"

          VolumeEncryptionKey: "{{ VolumeEncryptionKey }}"

          InstanceType: "{{ InstanceType }}"

          SpotPrice: "{{ SpotPrice }}"

          AZ: "{{ AZ | join(', ') }}"

          LogSNS: "{{ LogSNS }}"

          NumberOfZones: "{{ NumberOfZones }}"

          NodesPerZone: "{{ NodesPerZone }}"

          VPC: "{{ VPC }}"

          PrivateSubnets: "{{ PrivateSubnets | join(', ') }}"

          PublicSubnets: "{{ PublicSubnets | join(', ') }}"

        tags:

          Stack: "ansible-test"

Deploying the cluster

Now that we have defined our variables created our playbook, it's time for the actual deployment.

ansible-playbook -i hosts ml-cluster-playbook.yml -vvv

The -i option allows us to reference the inventory file we created. As the playbook runs, it will output as it starts and finishes tasks in the playbook.

PLAY [Ansible Test] ************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************

ok: [localhost]

TASK [cloudformation] **********************************************************************************************************

changed: [localhost]

When the playbook finishes running, it will print out a recap which shows the overall results of the play.

PLAY RECAP *********************************************************************************************************************

localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

This recap tells us that 2 tasks ran successfully, resulted in 1 change, and no failed tasks, which is our sign that things worked.

If we want to see more information as the playbook runs we can add one of the verbose flags (-vor -vvv) to provide more information about the parameters the script is running, and the results.

Now we can check the AWS Console to see our stack:

And we can also use the ELB to login to the Admin UI

Wrapping Up

We have now deployed a 3 node cluster to an existing VPC using Ansible. The cluster is now ready to have our Data Hub, or other application installed.  We can now use the git module to get our application code, and deploy our code using ml-gradle.