In this quite extensive post I will walk you through the process of creating from scratch a box in EC2 ready to use for deploying your Rails app using Ansible. In the process I will show how to write a simple module that, while not necessary, will illustrate some points as well.
Keep in mind that this is an example based on how we work in our company and that, at the same time, I am not specialised in devops or system administration, so the following example can have its pitfalls. It is also my first work in Ansible so there are probably more efficient ways to do some of the stuff.
At the end of the process we will have a box with the following elements:
- desired ruby version
All this is built on CentOS 64 bit machines.
Finally, I’d like to mention that this work has been possible thanks to the amazing Ansible community on freenode, which has been extremely helpful and patient when I needed guidance.
All this code can be found in my public rails_server_playbook github repository, in which I will be adding improvements or fixing mistakes.
At work we have (almost) all of our infrastructure in AWS. We essentially use EC2 instances to host a series of applications and services, and have an RDS MySQL database. We do not require fancy things, and in fact we only use a tiny amount of features of the range AWS has to offer. We also don’t need to be constantly creating and terminating new instances for autoscaling.
The main problem before we decided to give Ansible a try was that the process for updating our boxes was extremely painful and inefficient. We had this old AMI template which we kept upgrading every time we needed something changed on it, and then reprovisioned our servers with the new template. Applying those changes to tens of machines was tedious and error prone. There was also the fact that the existing documentation on the packages installed and configuration files changed was scarce, making changes even more difficult for the fear of breaking something and the annoyance of having to be reverse engineering everything.
To put a stop to this, we decided it was time to automate all this, and by that time Ansible was just becoming trendy. It also seemed to fit all our needs:
Also, the community around it seemed great, so we decided to give it a go. This project would let us not only automate the whole server provisioning, but also have a comprehensive documentation (under version control) of what was installed on the boxes.
So basically the scenario I will describe assumes that:
- We have a known list of machines we want in our infrastructure
- Every machine can be either for staging or production
- Every machine has an associated and known elastic ip (which is eventually linked to one or multiple domain names)
- While some of the machine characteristics are particular, some of them are going to be shared among all of them
- Most of the machines will be used to run Rails apps on them. Most of the times the same app
With that in mind, let me explain how our Ansible playbooks work.
Creating the instances
The instance creation is centralised in a role called
ec2_creation. The role is fairly simple.
The instance configuration is on the file
roles/ec2_creation/vars/main.yml. This file uses some kind of a hierarchical model. The variable
default_values contains shared values among all of the instances. Then we have two more variables:
production, each one containing specific configuration for each environment.
Here’s what the file looks like:
The trick behind this is that we will pass our
ansible-playbook command the extra variables
--extra-vars "rails_env=staging site=example.com" and then the
instance_values variable will contain everything we need to create the instance.
The main provisioning file, which we call
provisioning.yml, has several parts, the first one is like this:
This is what will create the actual instance. Let me explain the parameters:
hosts: localhost-- we setup the hosts as localhost because the actual task will be run on the local machine.
connection: local-- same as above, we do not need any special connection to connect to localhost.
gather_facts: false-- no need to gather any facts.
roles: ec2_creation-- this will basically tell the playbook to apply the role
ec2_creation, which contains the individual tasks to create the instance.
ec2_creation role has two tasks, which you can see in the
main.yml file on its tasks folder:
The first task will connect to EC2 and query the current instances to see if what we want to create exists or not. In order to do this we use a custom made module named
ec2_instances that you can check on the appendix if you’re interested in knowing how it works. For the time being the only thing needed to know is that we will register the output of this module to a variable for later use. The code for this task is as follows:
We pass the module a single parameter region, in this case hardcoded to
"eu-west-1", you can use a variable if you prefer it.
register instruction will save the output of the module to a variable named
ec2_instances that we will be using later.
Finally, there’s a second task that will just output into the console the information retrieved. I use the
debug task often when I’m not sure what information each variable holds.
Once we have the information on the existing instances, we invoke the tasks on the
create_instance.yml file, which is a bit more complex:
The first one is yet another debug statement to show the values which will be used to create the instance.
The second one is the one that creates the instance. It uses the
ec2 module, and most of the parameters are self explanatory, so I will focus on several that I find need some more attention:
count: 1-- in this case, as mentioned, we only need one box per app.
wait: yes-- will wait until the instance is booting to return.
instance_tags-- this parameter is very important, as we will use the
Nametag of the instances to uniquely identify them.
register: ec2_info-- we register the details of the newly created instance in this variable because we will need this information on a later task.
when-- this one is also important because it will determine whether we will actually create the instance or not. If you remember the previous step, we connected to EC2 to get the existing instances and save that information on the variable
ec2_instances. This variable has a dictionary of the instances on EC2 indexed by the value of their
Nametag. So in our case,
ec2_instances.instances[instance_values['name']]will hold the information of the instance on EC2 with the name of the instance we want to create. If that information is there, it means the instance exists, so we do not create it. The way used to check for the dictionary having the key is a bit unorthodox and I'm open to a more elegant solution on the comments, but what we basically do is try to evaluate it and default it to the empty string using Jinja2 in case it's undefined. We compare this result to the empty string and if both values are equal it means that the initial evaluation failed to find the key as it had to use the default value (see Defaulting Undefined Variables).
The next task on the list will wait until the instance is up and listening to port 22 before doing anything else. Note that as a host we pass information from the registered variable:
ec2_info.instances.public_dns_name and that we only execute the task if the previous step has created the instance with
The reason to wait until ssh is up and listening to connections is that we will need to access the new box via ssh to run the rest of the playbook.
Finally, on the last task (ignoring the debug one) we add this newly created instance to a group of hosts we name
ec2_hosts. The actual ip is in the
ec2_info.instances.public_ip variable, and we also add some more information on the host that we will use later, like the EC2 instance id.
And that is all the
ec2_creation role will do. Next on the list for the main provisioning playbook is the application of the
passenger roles, which will install and configure everything needed on the box.
Configuring the newly created box
Once we have the machine up and running, what comes next is a standard set of tasks for ansible. I grouped those tasks in a role called
common that has the usual role structure and elements in separate folders:
- tasks -- the different tasks to perform.
- vars -- useful variables used along the role.
- files -- static config files for the target machine.
- templates -- template files that need some stuff replaced.
- handlers -- triggers for various things.
Now this common role is not yet a 100% complete and chances are for a fully working setup some more things will need to be added (like development yum packages for compiling certain ruby gems), but it’s a good start as a skeleton.
The application of the common role in the main provisioning playbook is done by adding this to the provisioning.yml file:
The common role has its tasks separared in several files that are included in the right order in the main.yml file:
Note that we will create a user
rails that will be the user running the applications, and that we also have the
ec2-user user provided by the Amazon Linux AMI that has
In order for this to work, you have to make sure you can connect to the newly created instance with the
ec2-user by adding your EC2 key into your ssh-agent.
To avoid repeating myself in each task in which we do this, note that most of them require the
sudo modifier so the commands are run with superuser permissions.
The hostname task
The fist thing we’ll do is set up the machine hostname. We will use a pattern to define our machine hostnames, and that pattern will be
stag.example.com). The task uses the
hostname module and is pretty straightforward:
The sudoers task
In here we will add some configuration to the sudoers system. This is the task code:
In here we
copy a file into the remote machine and assign it the correct owner and permissions. The file is located in the
roles/common/files/cloud-init path and has this contents:
What this does is allow both the
rails users to be able to run
sudo commands without having to type in the password. This will be handy for us to run commands with superuser privileges, but keep in mind the security implications of it.
The rails_user task
The next step is creating the
rails user. The file actually contains more than one task:
We start by using the user module to create the user.
Once the user is created we setup the ssh authorized keys so we can log in with this user using an arbitrary number of ssh keys we want (in our case we use one key per developer plus the ones we need for deployments). We do this on two stages: the first one clears the
~/.ssh/authorized_keys to get rid of old keys using a
shell command, and then we use the
authorized_key module and use the loop pattern so we can add as many keys as we want. The public keys are on the
Another ssh key related task is needed too. In our architecture, all frontends using the
rails user share the same ssh key as well. This simplifies a lot connection in between all our ec2 machines. In this case, though, because we need both the private and public keys, we do not store it in a repository, but on a special machine (which also holds sensitive password information on rails yaml files, for example) that in this example would be
securehost.com. So in order to copy the keys securely, we run a local command that will use
scp to transfer the keys from the secure host to the new box. It’s important to notice the flag
-rp3 on the command, which will transfer the files using the local machine as a gateway. Otherwise it would try to connect using ssh between both hosts, which would not yet be possible precisely because the new machine lacks the keys to connect to the secure host (this of course assumes the shell from which you’re running the playbook has ssh access to the secure host).
After that we finish our ssh maintenance with two more things. First we ensure the newly copied keys have the right permissions with the
file module, and lastly we copy the ssh config we want the user to use from
~/.ssh/config. This config has just the line
StrictHostKeyChecking no which will ignore fingerprint changes when connecting to ssh hosts. This is, again, a security compromise made based on the fact that we reprovision boxes often.
The bash task
At the end of the
rails_user.yml you’ll notice we included the
bash.yml file. This will configure some bash options for the new user and create some folders that we will need later. The file has the following contents:
What we do here is create the
.bashrc.d folder in the user home folder, which will hold additional bash configuration files. We then copy the
.bashrc config file from
This will make sure every file in the newly created plugins folder will get sourced upon login. And finally we add one file to the plugins folder. In this case it’s not just a plain file but a template, located in
This will simply make sure the machine has the correct
RAILS_ENV environment variable set up.
The packages task
This is a really simple task that will simply install some packages in the system using the
yum package manager module:
The packages to install are gathered from a variable named
packages that is defined in
roles/common/vars/main.yml (it also holds more variables to be used in other tasks):
By default it will install the bare software to later build rbenv, but it is a good place to add other packages that you may need for other purposes.
The apache task
To host the apps we will use apache. The software has already been installed in a previous task, but we still need to add a configuration file to it. This will be done with the following task:
The config file, that you can find on
roles/common/files/apache_custom.conf has the following contents:
And it will just let us add customised virtual hosts into our
rails user home folder, for each of our apps.
The rbenv task
And finally, we install the
rbenv ruby version manager.
This is a little more involved task with several steps on it, and all the information to do this is on the
rbenv web page and is just adapted to our structure:
We begin by copying a bash plugin that will make sure that
rbenv is properly set up upon login. For this we use a template in
If you need details on this check the documentation on
rbenv where it explains why it’s needed. The template uses the variable
rbenv_root that contains the folder in which
rbenv will be installed.
After this we clone the
rbenv repository into the installation folder using the
Once we have
rbenv, we also clone the
ruby-build plugin, that will allow us to build the ruby versions that we need for our systems to run the applications.
Now we are ready to build the ruby we need. But before that we check that it’s actually not been already built, to avoid extra work. We do this by running the command
rbenv versions | grep and registering the result into the
ruby_installed variable, that we will use later as a conditional.
The next task builds ruby and has two special things:
The first one is that it has a conditional, so it will only be run when the variable we registered before is false with the line
when: ruby_installed|failed. The second one is that it has a
notify tag that will trigger a handler with the line
notify: rbenv rehash.
This has to be done because of the
rbenv architecture, that requires you to run a special command every time you install a new command line tool to a managed ruby.
We can do this with ansible handlers. This will let us call the special handler
rbenv rehash when certain conditions are met (like a task being executed) without having to repeat the same set of things on different places.
In our case, this handler is set up in the file
And is a very simple shell command.
So now that we have the version of ruby we want installed, we make it the default ruby interpreter for
We then update the
And last, but not least, install the
And that is all for the
Installing Phusion Passenger
The playbook also includes the role
passenger, which will, as you may expect, get a working passenger installation done.
The role is divided into two tasks, listed in the
The first thing we do is install the passenger gem and compile, if needed, the necessary libraries:
The first task is pretty self explanatory. It’s important to note that we need to call the
rbenv rehash handler, as the gem will install new binaries that otherwise would not be accessible to
After that, we check if, by any change, we already installed and compiled passenger before. The way to do it is to run the command
passenger-install-apache2-module --snippet and getting the part of the output that points us to the library that it’s built. Then we do a
test -f of that file to check if it exists. We register the result on the
passenger_compiled variable for later use.
In the case
passenger_compiled fails we need to compile the module. We can achieve this easily by running the shell command on the config above. Note that we pass the
--auto modifier so it doesn’t need any interaction from the user.
That will leave us with everything installed and on place. Now apache needs to be told to use this new module, which we do in the
The first thing to do is capture the config snippet from the
passenger-install-apache2-module --snippet command and save it to
passenger_snippet. Then we create a new apache config file with its contents on
/etc/httpd/conf.modules.d/02-passenger.conf. All files in the folder
/etc/httpd/conf.modules.d/ will be automatically loaded by apache assuming you have not changed the main config file. Finally, we copy another file with some passenger defaults to
Feel free to use your own values for this.
And that is all. After this you only need to work on your own apache configurations and deployment scripts to get things up and running.
Associating the elastic ip to the new box
In the main provisioning task, there is a final task that will use the ec2_eip module to associate the elastic ip to the new box:
ec2_instances bespoke module
If you are interested in the module that was built for the purpose of getting a list of your inventory on AWS, you can find it on the
library/ec2_instances file. The
library folder is the place to put modules not present in ansible. It is heavily based on other ec2 modules already found in the core, and it’s basically a wrapper around the python boto library:
The important bit is on the last
try/except block, in which me make a request using boto and then craft a response that only includes instances in which there is a tag with the
'Name' key and the status of the instance is not
In the repository you will find a couple of roles that you may find useful for installing a redis database engine (or just the client).
Please feel free to comment on mistakes, improvements or any other questions you may have on this.
Congratulations for reading this to the end :)