Jul 31, 2014

5 most common Rsync command usage examples

rsync is a command for remote sync.

It is used to synchronise one location to another in a simple way. Location is meant to be local directory, server or remote web server or whatever accessible by ssh.

Advantages of using rsync over other tools is speed and bandwidth requirements. First time rsync copies entire contents of the directory provided and increments changes over next sync times.

Command syntax:

$ rsync options source destination

1. Synchronising local directories:

$ rsync -zvr /tmp/logs/ /tmp/logs1/
building file list ... done
created directory /tmp/logs1

sent 98 bytes  received 48 bytes  292.00 bytes/sec
total size is 0  speedup is 0.00
Used rsync command options here:

  • -z is for compression
  • -v is for verbose output
  • -r is for recursive directory scanning
By default rsync does not preserve timestamps. 

2. Preserve timestamps and permissions during sync:

$ rsync -azvr /tmp/logs/ /tmp/logs1/
building file list ... done

sent 122 bytes  received 48 bytes  340.00 bytes/sec
total size is 0  speedup is 0.00
Option -a preserves symbolic links, timestamps, user permissions and ownership.

3. Synchronize from Local to Remote:

$ rsync -avz /tmp/logs/ root@
root@'s password: 
building file list ... done
created directory /home/user/logs

sent 122 bytes  received 48 bytes  68.00 bytes/sec
total size is 0  speedup is 0.00
It is required to specify username and ip-address of the remote server, while doing synchronization. It is also required to specify the destination directory on the remote server. The format is username@machinename:path. Sometimes depending on your credentials and ssh authentication method you may need to enter a password. Like in this example.

4. Synchronize from Remote to Local:

$ rsync -avz root@ ~/temp/
root@'s password: 
receiving file list ... done

sent 290 bytes  received 11002 bytes  3226.29 bytes/sec
total size is 16956702  speedup is 1501.66
This example is opposite to previous. we have synchronized a list of only changed files here. So speedup is relatively high.
5. View the rsync Progress during Transfer:
$ rsync -avz --progress root@ ~/temp/
root@'s password: 
receiving file list ... 
284 files to consider
        1372 100%    1.31MB/s    0:00:00 (xfer#1, to-check=252/284)
        1024 100% 1000.00kB/s    0:00:00 (xfer#2, to-check=251/284)
        8658 100%    8.26MB/s    0:00:00 (xfer#3, to-check=250/284)
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=249/284)
        1024 100%  500.00kB/s    0:00:00 (xfer#5, to-check=248/284)
        8632 100%    2.74MB/s    0:00:00 (xfer#6, to-check=247/284)

sent 158 bytes  received 9666 bytes  3929.60 bytes/sec
total size is 16956814  speedup is 1726.06
Running with --progress option showcase detailed progress of server interaction during load operations.

Thats it. Hope you find this article helpful to you. Comments? Suggestions?

Jul 29, 2014

Django adding custom widget to Django Admin

Sometimes you need to get out of standard behaviour of your Django admin. It means expanding its functionality with custom things. In our case we have a model that we want to add a custom html element. Button that must load something via AJAX from another place to be exact.
We have a typical Django polls application for simplicity. Here is its structure:

First of all we need to create a structure. So here comes our model:
# models.py
from django.db import models

class Poll(models.Model):
    question = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')
We have to have our django application admin configuration to display that model:
# admin.py
from django.contrib import admin
from polls.models import Poll

It will look in Django admin like so:
You can see polls available here in admin. You can do all the typical things with Polls model. Add/Delete/Edit... Using standard Django features. Imagine we have something on that polls model that we need to expand. Say, we need a button to visit external site here, made like an additional edit field. Imagine we need to take something on external website. We can post something to that website from our filled django admin form. To do that we need to create a custom form field. I have not found any standard widgets to do that. It is required to write our own field and attach it to the Django admin form.
We do not need own field here. Just a widget. Widgets are standard Django types. It is possible to read wide description of them in the official documentation: Django Widgets. I wont discuss here best ways to implement this task and there might be better ones. However I have chosen a form widget for simplicity.

It is required to have a model expansion form and a custom widget in order to do so. Registration fo that for m in django admin setup is also required. Lets implement this.
Starting with a form and a widget:
# forms.py
from django import forms
from django.utils.safestring import mark_safe
from django.template.loader import render_to_string

from polls.models import Poll

class ButtonWidget(forms.Widget):
    template_name = 'auth_button_widget.html'

    def render(self, name, value, attrs=None):
        context = {
            'url': '/'
        return mark_safe(render_to_string(self.template_name, context))

class PollsForm(forms.ModelForm):
    button = forms.CharField(widget=ButtonWidget)

    class Meta:
        model = Poll
It is located in a newly created file - forms.py at the django app - polls directory. Here we have a custom written widget that inherits a typical default Django widget (forms.Widget). we have provided it a template - auth_button_widget.html and a custom render() method that provides extra context for that template.
Next comes the PollsForm class with a custom button field that uses that widget. Note here Meta for that model is specified, indicating we are adding that field to the Polls model add/edit form.
Template, residing in all django templates directory will look like so:
{# auth_button_widget.html #}
<a href="{{ url }}">Go Button</a>
We have our form and widget and a template to render that form element. Time to use that form.

Using this form means modifying our admin.py file and registering custom model admin there. It will use our custom form and a widget. Resulting admin.py will look like this:
# admin.py
from django.contrib import admin
from polls.models import Poll
from polls.forms import PollsForm

class PollsAdmin(admin.ModelAdmin):
    form = PollsForm

admin.site.register(Poll, PollsAdmin)
Resulting change will add a custom form to edit or add the Polls model, containing our custom button field. It will look something like so:

Notice the button we have add here. Its just a simple HTML element that needs to be styled and animated by some CSS and JS. It can be done adding Media to our form widget. Resulting new widget will look like this:
# forms.py (partially)
class ButtonWidget(forms.Widget):
    template_name = 'auth_button_widget.html'

    class Media:
        js = (
        css = {
            'all': (

    def render(self, name, value, attrs=None):
        context = {
            'url': '/'
        return mark_safe(render_to_string(self.template_name, context))
Its a registration of static files to use in your widget. They will represent a scripting and styling for that field and will be included to the page of django admin form automatically.
For this to function you have to have django static files configured

Content of scripts and styles are out of the scope of this article. Main idea here is to highlight a proper way to add and write a custom widget from the backend perspective.

Comments, suggestions?

Jul 14, 2014

Virtualenv for Django

One of main features and troubles for Django projects is the Django version you are using. Example for this would be the occasion of starting project for Django 1.4.x version. Trying to run it under Django 1.7.x would lead to various troubles, counting template changes, misconfiguration errors and so on.

Solution is to use virtual environment. The tool for this is called Virtualenv. It is a wrapper for a python interpreter. It enables you to have multiple versions of python packages that run independently.

The Virtualenv creates you a standalone environment. Basically it copies and keeps a system Python of yours in a specified directory. All the python packages will be installed there. (In case you have not forgot to activate it first, of course)

Lets get started then. We need Virtualenv itself for the first. There are many ways to install it depending on your system. Most obvious to use pip. Your installation command will look something like that:
pip install virtualenv
This installs the tool itself to your system. this only must be done once.

Assuming we have it now and going forward to usage basics. We need to create a virtual environment for our particular project. Usual rule for most of developers is to have a single virtual environment for a standalone project.
To create one for ours we need to be in the directory where we want it to reside. (Virtual environment is not movable by default). So choose wisely.
I suggest it be somewhere in the project directory. The directory that is up one step from the project one is ok too. Assuming we have a project that is called "example". I will enter the directory of that project:
$ cd example
garmoncheg:~/Projects/example$ ls -la
total 8
drwxr-xr-x  4 garmoncheg  staff   136B Jul 14 14:26 ./
drwxr-xr-x  3 garmoncheg  staff   102B Jul 14 14:26 ../
drwxr-xr-x  6 garmoncheg  staff   204B Jul 14 14:26 example/
-rw-r--r--  1 garmoncheg  staff   250B Jul 14 14:26 manage.py
We have here a manage.py file to run our project with and a project files directory. Seems like a nice place to create a virtual environment at. Time to do it with executing a command:
$ virtualenv venv
New python executable in venv/bin/python
Installing setuptools............done.
Installing pip...............done.
We have commanded our virtualenv to create a directory for our new project environment and call it venv. You could use any name here. it will indicate your projects name or whatever your fantasy is up to. Mine project directory now looks like this:
$ ls -l
total 9
drwxr-xr-x  6 garmoncheg  staff   204B Jul 14 14:26 example/
-rw-r--r--  1 garmoncheg staff   250B Jul 14 14:26 manage.py
drwxr-xr-x  6 garmoncheg  staff   204B Jul 14 14:32 venv/
I have a mange.py script here, example directory with project files and a venv directory containig my environment files.
Now we have created a directory for environment and put our environment in it. However system is not aware we are using this environment. We must activate it now. Doing that is easy enough.
Assuming you are in the same project directory and called the virtual environment we are creating "venv". Execute a command:
$ source venv/bin/activate
This will tell the system we are using the interpreter set we have just created. It is indicated with (venv) at the start of your terminal prompt. this means we are "inside" the environment now.
Note we are in the same directory we where before.

Lets familiarise ourselves with main virtualenv functionality now. Whats the main point of it?
Simple. All the "pip install something" made now in this terminal will install a Python package inside this venv folder, leaving the main system python files intact. You can have as many environment as you wish. You can install whatever Python package you want inside of that virtual environment.

Now lets go through basic commands. Lets try to run our project now, assuming you have followed my previous instructions. To do that just type:
(venv)$ ls
total 9
drwxr-xr-x  6 garmoncheg  staff   204B Jul 14 14:26 example/
-rw-r--r--  1 garmoncheg  staff   250B Jul 14 14:26 manage.py
drwxr-xr-x  6 garmoncheg  staff   204B Jul 14 14:32 venv/
(venv)garmoncheg:~/Projects/example$ python manage.py runserver
Traceback (most recent call last):
  File "manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
Whoops. We can not do that because there is no Django installed. Even though you have had it in system it does not seem to be present now.
All the credits to virtualenv activation command. (source path/to/activate/script). We have to have it installed in the environment now. However it is more convenient to look for ourselves there is no Dango in environment. Lets execute:
(venv)garmoncheg:~/Projects/example$ pip freeze
Tadaam. We only have wsgiref in our console environment now. Lets install django:
(venv)garmoncheg:~/Projects/example$ pip install django
Downloading/unpacking django
  Downloading Django-1.6.5.tar.gz (6.6MB): 6.6MB downloaded
  Running setup.py egg_info for package django
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Installing collected packages: django
  Running setup.py install for django
    changing mode of build/scripts-2.7/django-admin.py from 644 to 755
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /Users/leopard/Developer/Python/Me/Tests/Projects/example/venv/bin/django-admin.py to 755
Successfully installed django
Cleaning up...
Tadaa. We have our django for the project now. Lets make sure with pip freeze:
(venv)garmoncheg:~/Projects/example$ pip freeze
And our project now can run inside a wrapper. Lets test with:
(venv)leopard@garmoncheg:~/Projects/example$ python manage.py runserver

Validating models...

0 errors found
July 14, 2014 - 12:00:03
Django version 1.6.5, using settings 'example.settings'
Starting development server at
Quit the server with CONTROL-C.
Yes it works.

Cons here that we must activate a virtual environment each time we want to seriously interact with our project. So even to run it we need to activate it first. but it is really the only option to work with multiple project in a system in an easy way.

To return the terminal to a proper state (back to system Python interpreter) you could just close it or deactivate the environment:
(venv)garmoncheg:~/Projects/example$ deactivate
Tadaa. (venv) preceding the terminal prompt is gone and we are back to the system interpreter.

Hope this helps someone. Suggestions? Helped? Please comment! I will really appreciate all the info.