Andrew Wilkinson

Random Ramblings on Programming

Archive for the ‘deployment’ Category

Simplifying Django dependencies with virtualenv

with one comment

books in a stack (a stack of books) by austinevanvirtualenv is a tool for simplifying dependency management in Python applications. As the name suggests, virtualenv creates a virtual environment which makes it easy to install Python packages without needing root privileges to do so.

To use the packages installed in a virtual environment you run the activate script in the bin directory of the virtual environment. This is fine when you’re working on the command line, but you don’t want to have to remember this step when running the debug server, and it’s hard to get that to work when the site is deployed under mod_wsgi.

To make things easier you can add the appropriate directory from the virtual environment to Python’s path as part of, or the appropriate fcgi or wsgi control script.

import os
import sys
import site

root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))

site.addsitedir(os.path.join(root_dir, 've/lib/python%i.%i/site-packages' % (sys.version_info[0], sys.version_info[1])))

Just add the code above to the top of your file and the ve virtual environment will always be activated when you run the script.

Photo of books in a stack (a stack of books) by austinevan.

Written by Andrew Wilkinson

August 24, 2010 at 1:23 pm

Continuous Integration Testing

with 7 comments

rom whence the brick came by mythotoAt work we recently set up Buildbot to replace an in-house continuous integration tool that never really took off. I’ve used Buildbot before, in a hobby capacity, but using it my day job has really brought home to me how important not only testing is, but testing continuously.

Buildbot is pretty simple to explain. Every time you make a commit to your source control system this triggers buildbot to run a series of steps over your code. What these steps are is totally configurable by you. Typically though, you’ll configure it to check out your code, compile it and run unit tests.

I’m a big fan of test driven and issue driven development. With these two methodologies when you want to make a change you record the issue in your bug tracking software, then write a test that detects the deficiency in your code. Once you’ve done that you can start actually making the change that you wanted to make in the first place. By writing down the issue you’re forced into thinking about the change that you want to make, and creating the test ensures that you know when it’s complete.

TDD and IDD are useful things to aim for, but in reality you’ll probably fall short sometimes. Not everything can be unit tested easily, or you might be under significant time pressures that force you to cut corners. One of the projects I manage has a test suite that takes 25 minutes to run. If I’m getting phone calls urging me to fix a problem as soon as possible then I can’t wait for the tests to run before I commit the change.

Because builtbot runs without user intervention you don’t need to remember to do anything. Buildbot is also a nag. If you introduce a bug with a change you quickly get an email letting you know that there is a problem, and if you set those emails up to go to your team then you also have peer pressure to get that problem fixed.

I spend most of my time building Django based sites. For these sites I set up six steps for Buildbot to follow. The first step is the checkout step, then a step to copy a settings file into place. Django stores settings in a Python file and these settings need to be configured for each server that the site is run on. I typically have a file stored in the repository and this step copies this file to so the following commands will run with the correct settings. The next step is to download and install the dependencies of the site. I’m going to go into more detail on how I manage settings and dependencies in a future post, but Buildbot just runs the correct commands.

The first three steps that Buildbot runs are just configuration steps. The first of the real test is a simple compilation step. It’s rare that a change gets committed without it ever being run, but it’s still worthwhile to check that the code compiles. It is possible that when merging changes from different branches errors can creep in. Fortunately Python makes it easy to check that all the files in a directory structure compile using the compileall module. This can be run using the command python -m compileall -q which return an exit code of zero if all the files compile, and nonzero if there is an error. This means that if there is an error Buildbot will detect it, and mark the build as failed.

The next step I run is the unit tests. This where the bulk of the testing occurs. I’m not going to go into details on how to write unit tests, but assuming you have them this assures you that your tests pass after every commit. Because Buildbot can be easily configured to build on several different machines on each commit you can easily test that a change you made on a Python 2.5 machine also works on a Python 2.4 machine.

The final step I run is PyLint. PyLint checks that your code complies with a set of coding standards and simple checks. If your unit tests don’t have a 100% coverage (and frankly, whose does?) then PyLint will hep detect that there are no obvious flaws in your code. PyLint takes a bit of time to configure, especially as it doesn’t like some of the metaclass magic that Django uses, but once you have a configuration file written PyLint is an excellent final check.

Hopefully this has given you a flavour of what Buildbot can do for you, and if you’re in a team or are building a cross-platform piece of software then Buildbot really needs to be there, watching your back.

Photo of from whence the brick came by mythoto.

Written by Andrew Wilkinson

June 30, 2010 at 2:16 pm

Posted in deployment

Tagged with , , ,

Using Fabric For Deployment

with 2 comments

robot invasion by donsoloIn a previous post I discussed what you want from an automatic deployment system. In this post I’ll discuss how use to solve the repeatability and scalability requirements that I set out.

Fabric is a tool which lets you write scripts to automate repetitive tasks. So far, so bash like. What sets Fabric apart is the tools it gives you to run commands on remote servers. Fabric allows you to run the same commands on multiple machines, and to move files between the hosts easily.

To get started with Fabric you’ll need to install it, but a simple sudo easy_install fabric should be enough to get you up and running. The Fabric website has excellent documentation, including a tutorial, but before I discuss how to integrate Fabric with your Django deployment process, lets go over the basics.

A Fabric control file is a Python file named In it, you define a series functions, one for each command that you want to run on the remote servers.

from fabric.context_managers import cd
from fabric.operations import sudo

env.hosts = ['host1', 'host2']

def update():
    with cd('/data/site'):
        sudo('svn up')
        sudo('/etc/init.d/apache2 graceful')

We’ve defined the function update which can be run by typing fab update in the same directory as When run, Fabric will connect in turn to host1 and host2 and run svn up in /data/site and then restart Apache.

Typically I define two functions. An update command, like that above is used to update the site where is has previously been deployed. A deploy command is used to checkout the site onto a new machine. Fabric lets you override the host list on the command line using the -H option. To deploy one of my sites on a new box I just have to type fab -H new-machine deploy and the box is set up for me.

Fabric helps you fulfil a few of the requirements for a perfect deployment system. It is scaleable, as to extend your system to a new machine you only need to add the hostname to the env.hosts list. It is also repeatable, providing you put every command you need to run to update your site into your fabfile.

With an automated deployment system in place we can now move on to looking a dependency, settings and database change management, but those are subjects for a future post.

Photo of robot invasion by donsolo.

Written by Andrew Wilkinson

June 16, 2010 at 1:18 pm