Vagrant: What, Why, and How

This article will help walk you through using Vagrant to manage your virtual machine instances, and explain how you can take advantage of Puppet to provision various resources, like PHP and PostgreSQL.

Introduction

Developers have a huge selection of ways to build their web development environment.

Developers have a huge selection of ways to build their web development environment. You can use "local" options, such as installing pre-built "all-in-one" server stacks, such as Zend Server, XAMPP, MAMP, WAMP, etc., or you can install the components yourself from source, or via package management systems like Homebrew, Apt, and Yum.

This can build up as you work on various projects with: PHP 5.3 and PHP 5.4, MySQL, SQLite, MongoDB, Postgres, PEAR, PHPUnit, Rails 3.1, Memcached, Redis, Gearman, NodeJS, etc. If you upgrade or you computer dies, you'll have to start all over again.

You could have a "remote" setup, using a server on the network with Samba shares, or a SSH server mounted with a tool like ExpanDrive. The latter option can lead to latency on file reads/writes, which are extremely annoying. You could use SSH + Vim for everything, which is quick, but that only works if you want to use Vim for everything.

Development vs. Production

While you may be happy with how you are doing things right now, how many of you have heard (or said) "Well it works on my computer". This is horribly common and it happens when environments differ by even the most trivial detail.

It is extremely important to make sure that your development environment is identical to the production environment, and matches staging and testing servers if you have those too.

That might sound easy if you just think about installing Apache, PHP and some copy of MySQL, but there are a million factors to think about. If you are developing on OSX and deploying to an Ubuntu system, then you will notice fun issues with file capitalization. This is common in CodeIgniter, when somebody has a library with a lowercase first letter. It will load fine on OSX, but will break when deployed to production. Your development process might have just lost you some business, all because of some trivial OS difference that nobody thought of until it was too late.

Development = Production

Forcing developers to use the same OS is going to lead to problems.

So what is the solution? Force all of your developers to throw out their different tools and develop on the exact same model of laptop? If your developers are all getting brand new Macbooks, then you might not get too many complaints, but then you'd need to use OSX Server for everything.

You could use Linux for everything, but then you have to fight over which distribution to use. Forcing developers to use the same OS is going to lead to problems, reduced productivity and promoting nerd-fighting.

Virtualisation is the answer and it is nothing new, but when people think of virtualisation they often think of performance issues and their fans spinning wildly while their laptop desperately tries to run two operating systems.

This can still be the case trying to run Windows on a low-powered machine, but these days, an average Mac has 4 GB of RAM out of the box, which is more than enough to power an Ubuntu server installation running in command line mode and all of your usual development tools (IDE, browser, debugging tools, etc). There are a few options for virtualisation, but I prefer VirtualBox from Oracle (which is free). This software does all the heavy lifting for the virtualisation, then a tool called Vagrant manages the instances.

Step 1 - Installing VirtualBox

First Download VirtualBox and install it. On *nix systems (Mac OSX, Linux, etc), you will need to modify your .bash_profile (or .zsh_profile) to extend your $PATH variable:

PATH=$PATH:/Applications/VirtualBox.app/Contents/MacOS/
export PATH

This will allow Vagrant to know where VirtualBox is installed, and will, of course, vary for different operating systems.

Step 2 - Install Vagrant

You can download a vagrant build for your operating system, or install it as a gem if one is not available:

$ gem install vagrant

Step 3 - Create an Instance

Make somewhere for your vagrant setups to live:

mkdir -p ~/Vagrant/test
cd ~/Vagrant/test

We'll be using Ubuntu 12.04 LTS (Precise Pangolin), which already has a "box" set up.

vagrant box add precise32 http://files.vagrantup.com/precise32.box

You see here the argument "precise32" which is a nickname for the URL. You can now create an instance, which will download this .box.

vagrant init precise32
vagrant up

It will now be running. Easy! If you want to get into this instance, via SSH, use this command:

vagrant ssh

Step 5 - Configuration

You will have a file, called Vagrantfile, which contains configuration for this instance:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant::Config.run do |config|

    config.vm.box = "precise32"
    config.vm.box_url = "http://files.vagrantup.com/precise32.box"

    # Assign this VM to a host-only network IP, allowing you to access it
    # via the IP. Host-only networks can talk to the host machine as well as
    # any other machines on the same network, but cannot be accessed (through this
    # network interface) by any external networks.
    config.vm.network :hostonly, "192.168.33.10"

    # Set the default project share to use nfs
    config.vm.share_folder("v-web", "/vagrant/www", "./www", :nfs => true)
    config.vm.share_folder("v-db", "/vagrant/db", "./db", :nfs => true)

    # Forward a port from the guest to the host, which allows for outside
    # computers to access the VM, whereas host only networking does not.
    config.vm.forward_port 80, 8080

    # Set the Timezone to something useful
    config.vm.provision :shell, :inline => "echo \"Europe/London\" | sudo tee /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"

    # Update the server
    config.vm.provision :shell, :inline => "apt-get update --fix-missing"

    # Enable Puppet
    config.vm.provision :puppet do |puppet|
        puppet.facter = { "fqdn" => "local.pyrocms", "hostname" => "www" } 
        puppet.manifests_path = "puppet/manifests"
        puppet.manifest_file  = "ubuntu-apache2-pgsql-php5.pp"
        puppet.module_path  = "puppet/modules"
    end

end

This, if you hadn't noticed, is Ruby syntax; so you can get fairly creative with this file, but this is the basics.

It will show which nickname to use and have the URL in-case the nickname is not set up locally (good for sharing around).

The share_folder lines are really useful for mapping a folder in the instance to a local folder. Using nfs => true the instance will be able to write and change permissions of the files, which is useful if you are, for example, trying to install a CMS in there.

Port forwarding allows you to access your instance on http://localhost:8080 and, of course, change that to a different port if that conflicts.

This configuration file will also set the timezone to Europe/London, then run apt-get update, which should force your system to be up-to-date whenever it is booted. If you skip this config item, you may find multiple packages refuse to install as the references are now out of date.

When you change config, you can reload the instance to use it:

vagrant reload

Now that our servers are running and ready to go, we need to install some software. We're not just going to apt-get install a bunch of packages via the command line, we're going to "provision" our servers.

Step 4 - Provisioning

Server Provisioning is not something many developers need to think about.

Server Provisioning is not something many developers need to think about, as it is normally left to sysadmins. The idea is to make some record of what software and configuration has been made on a server so you can create new development environments, new staging servers that replicate production, or make another production server to load balance between the two.

Old-school Provisioning

How sysadmins handle this varies, but all sorts of solutions have been used in the past – from keeping a wiki of commands run (which can get big and outdated quickly) and the awesome approach of having a "multi-terminal", where you type commands into one window and it replicates the same commands on another 7 servers at the same time. These methods are all terrible.

One solution would be to build your own .box file, or create .iso backups so new servers can just be based on that, but maintaining those images creates a lot of extra work, and no matter how hard you try, those development machines will become out of sync as time goes on.

Modern Provisioning

There are two popular systems at the moment, called Puppet and Chef.

There are two popular systems at the moment, called Puppet and Chef. Both have been around for years, but have started becoming a lot more popular with the increase of the DevOps development method. The ideas for both are similar and you should investigate both systems, but this tutorial will focus exclusively on Puppet.

Essentially, instead of running a bunch of commands and hoping everything works fine, you build a manifest for Puppet explaining everything that you need to ensure has been done. When you run a command in the terminal, you are basically saying to the computer:

"Install Apache"

With Puppet, we would say:

"Ensure Apache is installed"

Or, instead of:

"Make a new folder, called /var/www and set permissions to www-data:www-data"

With Puppet, we'd say:

"Ensure /var/www exists and has permissions matching www-data:www-data"

The difference here is that these manifests can be run multiple times (on a cron job hourly or daily) to keep things up to date, and there won't be unexpected results from something trying to install twice.

It will also help you test that everything is running as expected, as any of these rules failing will throw errors that are easier to track than greping a huge quantity of bash command results. Puppet will throw big red errors letting you know that PHP did not install, or a specific module could not be configured.

Manifests and Modules

Manifests are slightly confusing at first, but, after a while,, they start to make sense.

To review a basic example:

file {'testfile':
  path    => '/tmp/testfile',
  ensure  => present,
  mode    => 0640,
  content => "I'm a test file.",
}

No need to explain what is happening here, right?

This file can be later referred to in your manifest as "testfile" which means it can be listed as a dependency for other actions.

For further examples, we'll refer to the PyroCMS Puppet manifests on GitHub.

include apache

$docroot = '/vagrant/www/pyrocms/'
$db_location = "/vagrant/db/pyrocms.sqlite"

# Apache setup
class {'apache::php': }

apache::vhost { 'local.pyrocms':
    priority => '20',
    port => '80',
    docroot => $docroot,
    configure_firewall => false,
}

a2mod { 'rewrite': ensure => present; }

This includes the "apache" module, sets up some variables, runs the extra "apache::php" manifest in the apache module, sets up a virtual host then ensures that "mod_rewrite" is enabled.

All of these classes are defined in the Apache module that we included.

Moving on, we also want to install PHP:

include php

php::module { ['xdebug', 'pgsql', 'curl', 'gd'] : 
    notify => [ Service['httpd'], ],
}
php::conf { [ 'pdo', 'pdo_pgsql']:
    require => Package['php5-pgsql'],
    notify  => Service['httpd'],
}

This chunk of manifest will install the PHP extensions we need, then the notify option will let Apache know that you have installed new configuration, meaning it will restart.

include postgresql

class {'postgresql::server': }

postgresql::db { 'pyrocms':
    owner     => 'pyrocms',
    password => 'password',
}

This will set up a postgres server, create a database, called "pyrocms" and make sure a user, called "pyrocms" exists with the password provided.

Nearly finished! The last step is to ensure that you have writable files and folders set correctly:

file { $docroot:
    ensure  => 'directory',
}

file { "${docroot}system/cms/config/config.php":
    ensure  => "present",
    mode    => "0666",
    require => File[$docroot],
}

$writeable_dirs = ["${docroot}system/cms/cache/", "${docroot}system/cms/config/", "${docroot}addons/", "${docroot}assets/cache/", "${docroot}uploads/"]

file { $writeable_dirs:
    ensure => "directory",
    mode   => '0777',
    require => File[$docroot],
}

This will ensure that the Apache document root is there, the config file is set to 0666, and a few writable folders are set to 777.

There we have it!

To run all of this, simply reboot your vagrant instance, or run:

vagrant provision

If everything has worked correctly, you should see lots of blue text signaling that everything is being installed, but, if something goes wrong, you will see red. Google those errors and try again.

The modules used here are: Apache, Postgres, PHP and you can see the whole thing in action by cloning the PyroCMS Vagrant repo:

git clone --recursive git://github.com/pyrocms/devops-vagrant.git ~/vagrant/pyrocms
cd ~/vagrant/pyrocms
vagrant up

Point your browser to http://localhost:8089/ and you should see the installer. Easy stuff, huh?

Summary

In this article, we used Vagrant, VirtualBox and Puppet to not only set one server instance for us to work with, but we've create a test suite for our server to ensure that everything is running, installed and configured properly.

We have also created a checklist for requirements, and will, in the future, be able to create any number of identical servers in minutes, not hours!