SSH to AWS MarkLogic Managed Cluster using a bastion host
18 November 2019 04:30 PM
The recommended way to run MarkLogic on AWS is to use the "managed" Cloud Formation template provided by MarkLogic:
The documentation for it is here:
By default, the MarkLogic nodes are hidden in Private Subnets of a VPC and the only way to access them from the Internet is via the Elastic Load Balancer.
This is optimal as it distributed the load and shields from common attack vectors.
However, for some types of maintenance it may be useful, or even necessary to SSH directly into individual MarkLogic nodes.
Examples where this is necessary:
1. Configuring Huge Pages size so that it is correct for the instance size/amount of RAM: https://help.marklogic.com/Knowledgebase/Article/View/420/0/group-level-cache-settings-based-on-ram
2. Manual MarkLogic upgrade where a new AMI is not yet available (for example for emergency hotfix): https://help.marklogic.com/Knowledgebase/Article/View/561/0/manual-upgrade-for-marklogic-aws-ami
To enable SSH access to MarkLogic nodes you need to:
I. Create an intermediate EC2 host, commonly known as 'bastion' or 'jump' host.
II. Put it in the correct VPC and correct (public) subnet and ensure that it has public / Internet-facing IP address
III. Adjust security settings so that SSH connections to bastion host as well SSH connection from bastion to MarkLogic nodes are allowed and launch the bastion instance.
IV. Additionally, you will need to configure SSH key forwarding or a similar solution so that you don't need to store your private key on the bastion host.
I. Creating the EC2 instance in AWS Console:
1. The EC2 instance needs to be in the same region as the MarkLogic Cluster so the starting console URL will be something like this (depending on the region and your account):
2. The instance OS can be any Linux of your choice and the default Amazon Linux 2 AMI is fine for this. For most scenarios the jump host does not need to be powerful so any OS that is free tier eligible is recommended:
3.Choose instance size. For most scenarios (including SSH for admin access), the free tier t2.micro is the most cost-effective instance:
4. Don't launch the instance just yet - go to Step 3 of the Launch Wizard ("Step 3: Configure Instance Details").
II. Put the bastion host in the correct VPC and subnet and configure public IP:
The crucial steps here are:
1. Choose the same VPC that your cluster is in. You can find the correct VPC by reviewing the resources under the Cloud Formation template section of the AWS console or by checking the details of the MarkLogic EC2 nodes.
2. Choose the correct subnet - you should navigate to the VPC section of the AWS Console, and see which of the subnets of the MarkLogic Cluster has an Internet Gateway in its route table.
3. Ensure that "Auto-assign Public IP" setting is set to "enable" - this will automatically configure a number of AWS settings so that you won't have to assign Elastic IP, routing etc. manually.
4.Ensure that you have sufficient IAM permissions to be able to create the EC2 instance and update security rules (to allow SSH traffic)
III. Configure security settings so that SSH connections are allowed and launch:
1. Go to "Step 6: Configure Security Group" of the AWS Launch Wizard. By default, AWS will suggest creating "launch" security group that opens SSH incoming to any IP address. You can adjust as necessary to allow only a certain IP address range, for example.
Additionally, you may need to review the security group setting for your MarkLogic cluster so that SSH connections from bastion host are allowed.
2.Go to "Step 7: Review Instance Launch" and press "Launch". At this step you need to choose a correct SSH key pair for the region or create a new one. You will need this SSH key to connect to the bastion host.
3. Once the EC2 instance launches, review its details to find out the public IP address.
IV. Configure SSH key forwarding so that you don't have permanently store your private SSH on the bastion host. Please review your options and alternatives here (for example using ProxyCommand) as key forwarding temporarily stores the private key on the bastion host, so anyone with root access to the bastion host could hijack your MarkLogic private key (when logged in at the same time as you).
1. Add the private key, to SSH agent:
ssh-add -K myPrivateKey.pem
2. Test the connection (with SSH agent forwarding) to the bastion host using:
ssh -A ec2-user@<bastion-IP-address>
3. Once you're connected ssh from the bastion to a MarkLogic node:
ssh ec2-user@<MarkLogic-instance-IP-address or DNS-entry>
For strictly AWS infrastructure issues (VPC, subnets, security groups) please contact AWS support. For any MarkLogic related issues please contact MarkLogic support via: