Automating everyday system routine with Fabric (Python)

Lets talk about your console work. I use it on everyday basis. I need to log on to  my deployment/stage/whatever, server and do some redundancy. E.g. download some logs, clean up some caches and/or redeploy something. Here is the occasion, when Fabric comes handy. You can eliminate all the redundancy and cover tons of operations you need to do every day, using console, shortcutting them to one of your simple commands, like: fab deploy_production -H root@mydeployment.com interested? Let's move on then. Notie this command is not a masterpiece, but must give you the understanding of usual workflow.

1. Installation and purposes

Let's go ahead, installing Fabric and automating some simple operations. You can read official docs about alternative methods. But installation is fairly simple. And is the matter of typing:

$pip install fabric

Nothing more special is required. You will need fabric at system wide scale. I install it locally with superuser privilegies. Note I'm assuming installation of this on my MacBook and not on a production server or any remote destination. Fabric really is a shell wrapper with commands queue that executes your console tasks. Something similar to Django unit tests, if you know what I mean. E.g. Developing a fabric script is a process of copy-paste your console interactions and here you go... Similar to BASH/SH scripts. But why use fabric then?

How would you handle commands exit status in shell or bash? Write some error handling right? And this one comes out of the box with it ;)
Let's not just talk, but write some

2. Example usage

To write your commands you need to make a file called fabfile.py with syntax that is described in fabric documentation. But it's almost "pure" python. So everything must be understood easily.

Fabric supports all the python stuff. And good IMHO practice it to split your task to steps and execute them one by one. You can use def method to define them. E.g.:

"""Test script for article"""

from fabric.api import run

def set_up():
    """does some setup automation"""
    run('setup something')

def run():
    """main run method, that now calls set_up() method"""
    set_up()

Now you can run a command fab --list in this directory. Output will be like so. Note how it converted all your comments into handy help text explanation for your script.

leopard-2:article garmon$ fab --list
Test script for article

Available commands:

    run     main run method, that now calls set_up() method
    set_up  does some setup automation
leopard-2:article garmon$

Now we have got a handy script to setup something. But let's not get away with it and show you some real life example. It will also be converted to hide some (possible) sensitive information...

3. Real life example

"""Example script for article with deployment parts"""

from fabric.api import run, sudo, cd, get, local, lcd, prefix

def cleanup_environment(folder_path):
    """ deletes old deployment instance"""
    sudo('rm -rf %s' % folder_path)

def install(folder_path):
    """ Installs an instance of my website into production deployment server with recreating a virtual environment"""
    with cd(folder_path):
        sudo('virtualenv ve --no-site-packages')
    with prefix('source %sve/bin/activate' % folder_path):
        sudo('pip install mysite')

def restart():
    """Restarts a service of my website"""
    sudo('service mysite restart')

def stop():
    """Stops a service of my website"""
    sudo('service mysite stop')

def run():
    """Runs a service of my website"""
    sudo('service mysite run')

def download_logs(folder_path):
    """ Download logs files and db from server to local machine current directory
        This will download all the logs from deployment server
        into your current directory (locally) with current timestamp"""
    import datetime
    time_stamp = datetime.datetime.strftime(datetime.datetime.now(), "%Y-%m-%d-%HH-%MM-%SS")
    with cd(folder_path):
        sudo('tar -czvf /tmp/dms_logs.tgz ./log/dms.log*', user='user')
    local('mkdir ./%s/' % time_stamp)
    with lcd(time_stamp):
        get("/tmp/dms_logs.tgz", "./dms_logs.tgz")
        local("tar -xzvf dms_logs.tgz")
        local("rm -f dms_logs.tgz")
    sudo("rm -f /tmp/dms_logs.tgz")

def main():
    """Redeploy your Django powered website"""
    deployment_path = '/srv/www/django'
    download_logs(deployment_path)
    stop()
    cleanup_environment(deployment_path)
    install()
    run()

This script has main commands for me to run make redeployment of my website into a production web server, with archiving all the data (logs in this example) into a directory with current timestamp.

You must run it with a parameter -H specified. this parameter represents a host that our little script will ssh to. and is something like username@192.168.1.1, or something you can ssh to, using command ssh. E.g. -H stage-server in case you can locally type ssh stage-server. So the full command to run this will be something like: fab main -H my_deployment_server

Most fo the code in this script is given as an example. And may contain some logical errors, because it is taken from different parts of a real life deployment commands. But I really hope it will help you to master your own deployment with this handy tool.

Comments and suggestions are welcome...