Perspective Unspoken

My perspective on Git, Docker, Python, Django, PHP and other stuff

Deploying with Fabric and Chef Solo (Part 1)

Fabric is a widely used library for SSH’ing into remote servers. It’s one of those libraries that really respect the zen of Python and makes for beautiful and seamless remote operations. Chef, now is well respected in the world of DevOps for being thee tool used for deployments and setting up of servers. At work we had to build a script to manage the deployment of the server so we ended using both.

Why both? Well, Fabric is awesome for handling the SSH part and moving around, creating directories, and executing random commands. Chef on the other hand now, is great for setting up applications because there are so many cookbooks out there. So it means, with little or no effort, we could set up our stack using cookbooks someone has already built.

Our stack looks like this:

  • Nginx (public facing server)
  • Gunicorn (reverse proxy server)
  • Redis (you know, for cache)
  • Rabbit MQ (Celery’s Broker)
  • Celery
  • Django (of course)
  • Log Rotate – used to rotate our logs and compress them.

In this tutorial, I’m going to cover how to cover using Fabric to start the deployment process, then hand over to Chef to get most of these applications installed.

Sew that Fabric!

Using Fabric, I created some basic functions to I’d need to install and update apt, clone our project and install Rabbit MQ (I ended up doing it here because at the time I had lots of issues with the RabbitMQ cookbook).

So the first step was to install a couple packages using Fabric. I installed the basic packages needed for building stuff on Linux like make and build-essential, and then git and emacs. Then I also did some symlinks necessary for PIL to work. Finally, we cloned our project. Typically to execute a function on the remote server, you pass that function to execute as the first argument and arguments for the function are passed after.

After that it was basically time to hand over to Chef to install the rest of our stack!

Cooking with Chef!

So we actually used Fabric to install Chef. This is where things got a little interesting. Chef-Solo uses two configuration files that are critical. There’s a solo.rb file that tells Chef where your cookbooks are and there’s a web.json file that holds JSON configuration for the Chef nodes. In our code below we check if the chef-solo binary exists by trying to get the version number. If not, there’s a script we pull down with Curl and execute. Then thereafter we generate the solo.rb file. The last line in our function calls chef-solo and passes it the web.json and solo.rb files.

The web.json is also worth perusing, it had configuration for python, postgres and our project. More importantly though, it stored the run list. The run list tells Chef the list of tasks it needs to execute.

The last thing in that run list is “recipe[project]”, which is the name of our recipe. The directory that stored our cookbooks looked like this:

  • cookbooks
    • project
      • recipes
        • default.rb – our main Ruby script that does the work
      • templates
        • default
          • – template for local settings
          • nginx-project.conf.erb – Nginx template
          • project.logrotate.erb – configuration for Log Rotate
          • – Run script for Celery
          • – Run script template for Gunicorn
          • update-project.conf.erb – Template for Upstart for the project
          • update-project-celery.conf.erb – Template for Upstart for celery

I’m going to describe our main default.rb file in stages.


First we had to define some variables and a function or two. The syntax #{variable} is used to reference a Ruby variable inside a string. Additionally, node[“variable”] is how we access data stored in our web.json file. We added a few variables to grab values from our web.json file so we can use shorter variable names later.

At the end of our script we created services for our project, but at the start we try to stop those services if they exist. We use the ignore_failure attribute so this fails silently when we’re deploying for the first time.

Next, we use the psql function we created earlier to create our database user. This was done at the time using this function because the Cookbook wasn’t treating me too nicely.

Next we created our virtualenv and installed all requirements.

Setup templates

Next, we proceeded to setup templates for the different services. Nginx uses one simple configuration file. We were able to set that up easily using a little bit of information from the web.json file using the template resource provided by Chef. We simply indicate the source, action and provide a list of variables that we want to pass on to the template. Finally, we symlink from /etc/nginx/sites-available/project to /etc/nginx/sites-enabled.

Using the nginx-project.conf.erb, we configured a reverse proxy server. Nginx accepts connections from the outside world and then proxies those requests through to Gunicorn. We add aliases for /media, /static and /admin/media. All URLs starting with one of those aliases are served directly by Nginx. All other requests are passed through to Gunicorn. In addition, we also raised the timeouts for reading data from Gunicorn (proxy_read_timeout) to be 60¬†seconds. That’s the time Nginx will wait for a response from Gunicorn. The proxy_connect_timeout controls how long Nginx will wait to connect before the error_page is served. The¬†client_max_body_size dictates the maximum file size that Nginx will accept to send to Gunicorn. This controls how big a file a person can upload.

Well… now you have to stay tuned for Part 2. In Part 2, we’ll talk about configuring upstart for the project (so we can say service project start/stop), log rotation and syncing and migrating.








jaywhy13 • January 19, 2016

Previous Post

Next Post