How to Use Dynamic Inventory for AWS with Ansible?

Do you know how Ansible works? Well, before we walk you through how to setup dynamic inventory, here’s some food for thoughts on Ansible Configuration Management System

A user using configuration management system will often want to save inventory in a different software system. As described in inventory, a basic text-based system is provided by Ansible. Some examples include pulling inventory from a cloud provider, LDAP, Cobbler, and a piece of expensive enterprise CMDB software.

These options are supported by Ansible through an external inventory system. Some of these such as EC2/Eucalyptus, Rackspace Cloud, and OpenStack are already a part of the contrib/inventory directory.

Maintaining an inventory file might not be the best approach if you use Amazon Web Services EC2. This is because the hosts may vary over time, they can be managed by external applications, or you might be using AWS autoscaling. In such a case, you can use the EC2 external inventory script.

Setting up EC2 External Inventory Script With Ansible

One way to setup an ec2 external inventory script is to copy the script to /etc/Ansible/ec2.py and chmod +x it. You will also need to copy the ec2.ini file to /etc/Ansible/ec2.ini. Once done, you can run Ansible as you would normally do.

For making a successful API call to AWS, you will need to configure Boto (the Python interface to AWS). There are a variety of methods available, but the simplest is to export the following environment variables:

export AWS_ACCESS_KEY_ID=’AK123′
export AWS_SECRET_ACCESS_KEY=’abc123′

Now, you will need to set up a few more environment variables to the inventory management script such as 

$ export ANSIBLE_HOSTS=/etc/ansible/ec2.py This variable tells Ansible to use the dynamic EC2 script instead of a static /etc/ansible/hosts file. Open up ec2.py in a text editor and make sure that the path to ec2.ini config file is defined correctly at the top of the script:

$ export EC2_INI_PATH=/etc/ansible/ec2.ini This variable tells ec2.py where the ec2.ini config file is located.

You can test the script to make sure your config is correct:

cd /etc/ansible
./ec2.py –list

After some time, you should be able to see the entire EC2 inventory across all regions in JSON. You can have a look at the scripts of EC2.py and EC2.ini

Understanding Tags and Host variables

Let us first understand how tags in external inventory work followed by how we can use them. Here’s an example playbook command:

/etc/ansible/playbooks$ ansible-playbook docker-update.yml –extra-vars    “variable_host=tag_Ecs_true:tag_User_ec2:&tag_Environment_stage”

In the above example:

1. ansible-playbook is the standard command to run a playbook
2. docker-update.yml is the name of our playbook inside the playbooks folder
3. –extra-vars is a parameter which tells the main command to take the following string as an input for the playbook file
4. variable_host is the variable that is being called inside the playbook to place a value for the hosts
5. tag_Ecs_true:tag_User_ec2:&tag_Environment_stage: These are tags and a way to tell playbook where to run the tasks, which user and key file to use to access the hosts

Whenever Ansible sees tag_Ecs_true, it tries to find all the ec2s in AWS account that has the tag Name ‘Ecs’ and Value ‘true’.

Simultaneously, Ansible also searches a group variable (inside group_vars folder) file with the name tag_Ecs_true for any host variable that we may have provided in that file. The above stands true for all and any ‘tag_abc_xyz’ provided along with the ansible-playbook command.

Let’s break these tags and see what they mean

Key file management: tag_Environment_stage — We need to tag all the ec2s in our account with the environment tag and respective value. We make use of this tag for providing the location of the key file required to access those ec2s. As explained earlier, Ansible will search inside the group_vars folder for a file named tag_Environment_stage, where we have already mentioned ansible_ssh_private_key_file: ~/.ssh/environment-staging.pem

This is nothing but the location of the key file. Similarly, we can manage all the key files.

User management: tag_User_ec2 — Now, Ansible will again search inside the group_vars folder for a file named tag_User_ec2, where we have already mentioned ansible_ssh_user: ec2-user. Thus, it uses ec2-user to access the hosts.

Where to run the playbook: tag_Ecs_true — This enables Ansible to look for all the hosts with the tag Ecs:true Name:value pair. We need to tag all the ECS instances with this tag so that we can make use of it whenever we need to run a playbook on them.

Patterns and Combinations of Tags

We will need various combinations of tags for which we can make use of standard Ansible patterns.

In the previous example, tag_Ecs_true:tag_User_ec2:&tag_Environment_stage, ‘:’ means that all the hosts in group tag_Ecs_true and tag_User_ec2 are to be managed if they are in the group tag_Environment_stage also ‘&’.

Please refer patterns document for a detailed explanation about patterns.

Handling Python Version Dependency 

• Ansible server checks for a specific version of python in all the target hosts to be able to run the playbooks or perform any action. As we are aware of the various OS families we are using in our infrastructure we don’t want to get stuck with python version dependencies.

In order to achieve this, we use Ansible pre_tasks (inside every playbook) which are actions that run prior to the actual role or tasks that we want to perform on any host. Before even calling the pre_tasks, we explicitly stop the ‘setup’ process of the playbook where it gathers facts about the hosts. It is in the pre_tasks, that we mention about a version of python which needs to be installed on each host where a playbook runs.

Once this is done, the ‘setup’ is explicitly called and then the execution of actual roles/tasks are performed by the playbook.

Here is an example playbook that updates docker version:

– hosts: “{{ variable_host | default(‘web’)}}”

gather_facts: no

pre_tasks:

– name: update on ubuntu

raw: sudo apt-get -y update

ignore_errors: true

– name: ‘install python2 on ubuntu’

raw: sudo apt-get -y install python-simplejson

ignore_errors: true

notify:

– Gathering facts

– name: update on Centos/Redhat

raw: sudo yum -y update

ignore_errors: true

– name: ‘install python2 on Centos/Redhat’

raw: sudo yum -y install python

ignore_errors: true

notify:

– Gathering facts

roles:

– role: docker-update-role

handlers:

– name: Gathering facts

setup:

• Another important addition in each playbook is provision for dynamic hosts: – hosts: “{{ variable_host | default(‘web’)}}”

We know our hosts are variable and we are not maintaining a static inventory of hosts. This is the reason why we don’t want our playbooks to be hosts dependent. Therefore, in all the playbooks we need to make the above entry.

Hope this blog was useful and you will be able to use dynamic inventory for AWS with Ansible.