Python / Django UnicodeEncodeError hacks.

Python plays with unicode nicely nowadays. Bt what if you must deal with old time formats conversion, or ASCII files exporting for e.g. You may also use software that is out of date but is too long to rewrite...  Here often errors occur. I have received mine at copy-pasting from MS Word into Django admin UI  by stupid users. Most of the website played nicely with this fancy characters, but exporting to CSV failed due to non ASCII characters support. Google said nothing special. Python docs about unicode usage briefly cover this type of events. So here is the result of some hours of experiments. I've decided to rewrite some of the python functionality to create decode function with behavior for my needs. Hopefully they will shorten you some time with those collisions you may get in your Django apps...

Anyway I've started to receive errors like:

Exception Type:  UnicodeEncodeError
Exception Value: 'ascii' codec can't encode character u'\u2013' in position 17: ordinal not in range(128)

So I had a list of u'' values that contained special characters " ordinal not in range(128) ". Requires no imports... Pure python:

        values == [
          u'Some fancy text \u2013 something', 
          u'some normal, easy convertible text', 
          u'some more normal text'
          ]      
        HACK: entry cleanup for special characters (Fixing Bug #...)
        # entry cleanup for special characters
        i = 0
        for value in values:
            try:
                # if string can be encoded to 'ascii' pass
                unicode(value).encode('ascii')
            except UnicideEncodeError:
                val_temp = unicode(value)
                # cleaning up string with escaping non convertible characters
                result = []
                for symbol in val_temp:
                    try:
                        symbol.encode('ascii')
                        result.append(symbol)
                    except UnicodeEncodeError:
                        pass
                # rewriting wrong value in values array
                val_temp = ''.join(result)
                values[i] = val_temp
                pass
            i = i+1
        # normally work with our list... it's safe now...
        values == [
          u'Some fancy text  something', 
          u'some normal, easy convertible text', 
          u'some more normal text'
          ]

This code is a bit complicated due to mine specific task and has iterations in iterations etc... But it's from a working app and checked working. However here is the theoretical example that must clean up a single string:

value = u'Some fancy text \u2013 something'
try:
    # if string can be encoded to 'ascii' pass
    value.encode('ascii')
except:
    # cleaning up string with escaping non convertible characters
    result = []
    for symbol in val_temp:
        try:
            symbol.encode('ascii')
            result.append(symbol)
        except UnicodeEncodeError:
            pass
    # rewriting our variable with safe one
    value = ''.join(result)
    pass
# normally work with our unicode string... it's safe now...
value = u'Some fancy text  something

So the technique here is simple. We are checking if this unicode string can be converted to 'ascii' python encoding without errors we simply passing through. And if it's not... Converting it to 'ascii' string symbol by symbol. Symbols that will fail will be gracefully omitted. You can create a function from all of this, like 'my_decode_cleanup' or something and use whenever needed...

Hope this will help you to save some precious time during your python development.

Helped? I'm wrong somewhere? Please comment!

Installing CouchDB for Mac OS X. Nuances.

Our project will have a new architecture changes in future. We've chosen from different tech. But now we 're experimenting with CouchDB a bit. It may become a part of our new architectural Changes. I'll try to cover most of newbies observations with CouchDB for Python(Django) programmer, planning to use it in your project.

First article will be, unsurprisingly, installing CouchDB under Mac OS X. It, sure, has some inches that you need to scratch. So let's roll...

CouchDB is not a simple SQL database like you used to install before. It has own server to accept connections. And, in fact, you can work with database from your browser GUI (without other server proxy, like Django or ...). Anyway it will not be our target. Building JS UI is quite easier with CouchDB and changes your logic dramatically. To build UI with Python we need some proxy parts. Anyway, again, CouchDB is a standalone server, that runs on your server's port and talks to your APP/Browser/whatever... through JSON API. It's quite a bad thing, because it's a bit slow, but a good thing, because it's extremely easy to implement. Most programming languages have adaptors to JSON.  CouchDB is quite a new tech. And it is highly developed and "pushed through" tech for now...

Ok enough preface... Actual install is described in official DOCS HERE. I'm using "mac ports" for my project already. So it's my choice of install method by all means. I'll cover docs here in brief to be consistent:

1. Install it through mac ports.

$ sudo port install couchdb

Will start huge and massive install of required ports for this DB.

2. Good idea would be to update Mac ports. If you have some ports installed already, like I do:

$ sudo port upgrade couchdb

Will upgrade only ports required by "couchdb" port.
It's also a good idea to selfupdate ports if you already have them installed:

$ sudo port selfupdate

This must be sufficient for all of our needs.

At this point you must be able to run CouchDB from command line running:

$ couchdb

command in Terminal. If it runs with error, no worries. Read on. There is a known issue!

3.  Bug occurs in MacPorts itself with "couchdb" package. It does not create some paths in required dir. You need to manually create those paths and give CouchDB permissions to write there. Mac Ports Track has the solution. Or you can manually (better IMHO) make sure all your permissions and directories exist.
You need to make sure those directories exist, and create them if they don't:

/opt/local/var/lib/couchdb/
/opt/local/var/log/couchdb/
/opt/local/etc/couchdb/

and have permission to work with CouchDB:

$ sudo chown -R couchdb:couchdb /opt/local/var/lib/couchdb/
$ sudo chown -R couchdb:couchdb /opt/local/var/log/couchdb/
$ sudo chown -R couchdb:couchdb /opt/local/etc/couchdb/

(Those paths are ok for a standart "mac ports" install.. If your directories differ... than change those paths accordingly)
After those actions, running couchdb from console should led to normal startup sequence output:

$ couchdb
Apache CouchDB 1.1.1-incubating (LogLevel=info)
Apache CouchDB is starting.
Apache CouchDB has started. Time to relax.

This will signal that your CouchDB install is almost OK and you may check that it is running:

http://127.0.0.1:5984/

In your browser. It must Show output JSON. Something like:

{"couchdb":"Welcome","version":"1.1.1"}

You may also examine it's web admin UI, typing in browser:

http://127.0.0.1:5984/_utils/index.html

Should show handy Admin UI. You may examine/edit DB by hand there... (Something similar to PHP MyAdmin or whatever you used to use on SQL DB's)

4. Add autorun command (optionally). It's good to run every time you need it for experiments, etc... But if you plan to Debug test apps on that platform it's good idea to make it start with system every toime it runs. To make that you need to type this command into console:

$ sudo launchctl load -w /opt/local/Library/LaunchDaemons/org.apache.couchdb.plist

THE END.

Comments and suggestions are welcome!
In future I plan to make some articles on actual befriending CouchDB with Python. Stay tuned and thanks for reading that far.

Establishing Dev environment with PyCharm + Virtualenv for Django development

I'm a fan of IDE's. Also I'm a fan of GUI's. I also use console where needed, but why waist time typing commands when you can just point and click. I've been using Eclipse + PyDev for almost a year and recently switched to PyCharm.

Why I prefer PyCharm:
- Template Debugging. (you can set a normal breakpoints in templates)
- Easy Virtualenv connection. (Set an interpreter from your virtualenv and you're ready to roll)
- Has excellent set of most common CVS integrations (Git, SVN, Redmine, etc...)
- Has looots of tiny tasty things for coding and proper code highlighting over JS and HTML out of the box)
- Has proper and shiny themes out of the box.
So I'm a blind minded blond, it seems now. But design of this IDE is really attractive IMHO :)
- OH and it's quite quick and has ALL the functions I need.

Here is a brief instruction to setup PyCharm IDE to work with my typical Django project.

1. First thing you need is a PyCharm installation. You can get it here (official website) for a 30 day trial for free. And this project has way of free licensing for Open Source projects. For everyone else it's 100$. Guess what? I'm open-source developer :).

2. I guess you'll be ok to install it by yourself. Nothing too special here. Just a normal setup depending on your platform. (I use Mac OS X Lion. And for me it's just a "drag to ~/Applications folder").
3. Now that you have it installed you can dive into it's magic. lets setup a new project like you do most of the times from console first.

$ cd developer
$ mkdir testproject
$ cd testproject
$ virtualenv testvenv --no-site-packages
New python executable in testvenv/bin/python
Installing setuptools............done.
Installing pip...............done.
$ source testvenv/bin/activate
(testvenv)username:testproject leopard$ 
$ pip install django
Downloading/unpacking django
  Downloading Django-1.3.1.tar.gz (6.5Mb):  53%  3.5Mb
...
$ django-admin.py startproject mysite
$ cd mysite
$ ls -l
total 32
-rw-r--r--  1 leopard  staff     0 Jan 10 11:13 __init__.py
-rw-r--r--  1 leopard  staff   503 Jan 10 11:13 manage.py
-rw-r--r--  1 leopard  staff  5031 Jan 10 11:13 settings.py
-rw-r--r--  1 leopard  staff   565 Jan 10 11:13 urls.py
$

Now that we have created typical project with virtual environment and a project dir inside it.

4. Let's connect it to PyCharm. Press "Open Directory" in either ("File > Open Directory") main menu or on this fancy welcome screen.

Actually have to notice here. Instead of "Project" architecture, PyCharm is sticks himself to folders. He lays there his ".idea" hidden directory with project configurations, run configurations etc. PyCharm stores all it's settings there. So, if you need to open new project, press "Open Directory" and select new one. You can not select Projects in the Projects explorer tree like it is in Eclipse PyDev. Which I see a bit of annoying.

After those actions you must get something like this:

A complete project directory. And a virtual environment directory. After opening it start's to update indexes. IDE indexes all python and actually all known files and scans their source. So code autocompletion is not available until it's finished:

5. Now that we have connected PyCharm to our Django Project... We need to setup interpreter. For PyCharm to work correctly with virtualenv and rund project in the specific environment... We simply need to specify Python executable (from our virtualenv) for this project and enable Django support.

It is all done in main menu ("PyCharm > Preferences"). We now need to select "Django support" and add make sure it's checked "enabled". Nw we need to specify manage.py and settings.py scripts. They will bee needed later for setting up run configurations.

6. After this we need to setup our virtual environment to be used with this IDE. PyCharm is quite sophisticated and does almost everything by himself. You basically need to add a new Python interpreter. PyCharm detects that it's a virtual environment and links and indexex all by himself. You need to select "Python Interpreter" in those preferences panel and press "Add New" > "Select other" in the top right corner. After that you need to point out to your "virtual-environment directory > bin > Python executable". PyCharm will scan for environment and connect it to where needed.

Select newly added interpreter and press "OK". TADAAA!!!!

AND we're almost ready to roll...

7. Creating Run configuration.

Now we have our project all set up and connected. We need to setup a proper run for our environment. Notice button "testproject" with Django logo appeared in the main buttons tab at the top. Pres a drop down menu and select "Edit configurations". Imho this part is a masterpiece here. It has nothing more than I usually need to run even sophisticated projects.

I'm specifying the "Host" to be run on local IP: 127.0.0.1 or "localhost" if you prefer so :). And i'm checking "No reload" option to manually reload server after some code changed. I prefer it like so to avoid various server hangouts and need to reboot entire Mac if something goes wrong. And now we're ready toclick "OK". And that's basically It.

You can now press "Run" button and check your browser showing 127.0.0.1:8000 shows Django test page. Or go directly to testing delicious debugger by running "Debug" button. 

Comments are welcome!