Tuesday, June 3, 2008

Creating a full-text indexed, ajaxian website - part 1

To my dismay, there is no complete guide how to get a nice, full-text indexed website up and running easily with some of my favorite tools. Therefore, I am giving you a guide here how to use the Eclipse IDE to create a nice Django WDF-based site on top of a MySQL DB engine together with the Sphinx full-text indexing and search engine and some simple AJAX calls using the jQuery JavaScript library to have autocompletion on the searches. I am using Apple OS X Leopard, so my examples might be slightly biased towards that system, but I will keep the OS-specific parts to a minimum.

1. Installation


I will not walk you through the installation procedures of these things (most stuff is easy and straightforward) except if there is some special attention to be payed; Python 2.5 should be installed on OS X 10.5 already.
  1. EasyEclipse (much quicker to install and maintain than the "real" Eclipse); Plugins you will need are: "Eclipse Web tools editors", "Eclipse Data Tools", and "PyDev". Additionally, choose any other tools you might deem necessary. Note that not all plugins are available for EasyEclipse 1.3 still, so you might want to use 1.2.2 if you need plugins not available for 1.3 (available ones are marked with a *) and/or check that first.

  2. Django (check out the SVN version, it provides much more functionality than 0.96 stable; see installation instructions)

  3. MySQL (get version 5.0, last stable - if you want a newer version, make sure MySQLdb and Sphinx work nicely with it; this is tricky, follow the installation instructions for your OS closely)

  4. Sphinx (get version 0.9.7 to work nicely with django-sphinx or check the latest supported Sphinx version)

  5. jQuery (latest available version usually works best, no guarantees...)

Next, you will have to get the "glue" between Django and Sphinx, Django and SQL, and the jQuery Autocompletion plugin:
  1. django-sphinx - a Django module to create search interfaces for your Django object models.

  2. MySQLdb - the Python DB client for MySQL; Important: installing this library on OS X Leopard will cause troubles, you need to edit some lines in the code - see below.

  3. jQuery Autocomplete - used to do autocompletion with AJAX calls in jQuery


1.1. Some hints


Installing Django, MySQL and MySQLdb on Leopard can be a bit of a hassle, at least the MySQLdb part. Here is a complete setup guide. Also, Python installations (specifically, site-packages) on Leopard can choose one of two possible locations:
  • /Library/Python/2.5/

  • /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/

The first is just an ancient artifact, the real place for your Python stuff is the second. I got so annoyed by this, that I finally moved everything in the fist path to the real location, deleted the directory "2.5" and then did:

ln -s /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5 /Library/Python/2.5

Voila, everything always goes to the same place.

1.2. Tutorials you might want to look at


If you are new to the world of databases or need a short introduction to MySQL, work through this tutorial to get you up and running.

There are some guides by the PyDev developer which you should read to get Python to work in Eclipse:

You might want to read through some other tutorials to, but these two are essential.

Finally, if you are completely new to Django or even the whole world of WDF (web development frameworks), I strongly advise you to work through the four pages of the Django tutorial. Also, there is an excellent Eclipse/PyDev/Django tutorial here.

2. Setting up a Django project in Eclipse


Following those guides, set up a new PyDev project, say "Django Sandbox", together with the source folder ('src') for Python 2.5. Now we will create the Django base files in the project using the terminal. Change to your Workspace folder you set up for Eclipse and go into the src folder of the project (ECLIPSE_WORKSPACE/Django\ Sandbox/src). For the next command to work, you should have the files in $PYTHONPATH/django/bin/ in your $PATH variable (see the Django installation instructions), or call the command directly.

django-admin.py startproject sandbox

To see the newly created files in Eclipse, right-click on the src folder and choose "Refresh". Check in the terminal everything is working:

python manage.py runserver

should produce:

Validating models...

0 errors found
Django version 0.97-pre-SVN-7568, using settings 'sandbox.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.


Next, we get the development server running in Eclipse. Choose "Run->Run..." from the menu, and select "Python Run" in the Run window, then press the "New" button. Name it "Django Sandbox default", associate it with the project and choose manage.py in the src/sandbox folder as the main module. Change to the Arguments tab and add runserver --noreload as the program arguments. If you now press the "Run" button, you should get just the same output in the Eclipse Console as we saw before in the terminal. If this is working, you basically have just managed to set up your IDE to develop Django sites in Eclipse. If not, have a look at the Django tutorial and the PyDev/Eclipse/Django tutorial linked above for more detailed information. Next we need to tell Django where our database backend is py editing settings.py in Eclipse:

DATABASE ENGINE = 'mysql'
DATABASE_NAME = 'django_sandbox'
DATABASE_USER = '`echo $USER`'


Password, host and port can stay empty usually, if we have set up a user on localhost with no required password and the default MySQL port. Also, as this is a nice little test system, we will disable all default Django apps, by commenting the array values in the INSTALLED_APPS variable in settings.py we do not need (all of them). Next we actually need to create the database in MySQL (start MySQL up in you System Preferences if you did the default installation without autostarting MySQL). You could log into your database on the terminal and run:

me@home:~[1]$ mysql -u $USER
mysql> CREATE DATABASE django_sandbox CHARACTER SET 'utf8';


But, we are using Eclipse, so we will be doing all our database work from Eclipse, too. In case you so far have never connected to your database from Eclipse, see this post on how to quickly set up a "connector" (probably you will have to download the appropriate JDBC driver for MySQL).
  1. Change to the database view in Eclipse ("Window->Open Perspective->Other..."). In the "Data Source Explorer" navigator window to the left you can now see your connector from the submenu "Databases" (I called my connector "MySQL localhost").

  2. Go into properties (right-click the new connector and choose "Properties") and make sure you are connecting as a user with the ability to created databases and make sure it connects to the database "mysql". Finish by connecting to your "mysql" database in the server by right-clicking on the connector back in the navigator window.

  3. Next, we create a simple SQL file to create databases from the menu "File->New->Other...", select "SQL-Development->SQL File" and associate it with our project ("Django Sandbox"). I named the file "Create Database", using connection profile type "Generic JDBC_1.x", name "MySQL localhost" (or whatever you called your connector). Choose "mysql" as the database.

  4. Now copy the CREATE DATABASE statement from above into the new SQL file editor view (without the "mysql>" part, obviously...) and save. If you are connected to the mysql database, all you need to do now is right-click in the editor and select "Execute All".

If all went well, in the lower part of the pane, you should see a nice success message in the "SQL Results" window. I do understand this sounds slightly overblown to create a database, but it should give you a good feeling how to do all you database work from inside Eclipse. (screenshot will follow). Finally, you can create a new database connector in the Database view to your newly created django_sandbox database which you will use later on to inspect your database, etc..

That is it! You have started and set up a new Django project in Eclipse, created the database, and connected to the database from inside Eclipse. Not much maybe, but this is quite a milestone. Congratulations! In the next part we will create a Django application, add some dummy models and fill the database with data to index.