Simple Python Plot X,Y for the Web

Motivation

I have a python script which produces a 2-D array. What is a (very) simple way to expose this script over the web and present a fancy graph?

or

I have a bunch of simple data files which I'd like to parse for 2-D graphs on the web, and I'd like to use python...

Brief Discussion of Technologies

There are at least a couple of technologies which can allow developers to put python code to use on the web. The first to come to mind are:
  1. A CGI-like web interface, such as mod_python for Apache
  2. A python web framework, such as TurboGears, which is used on at least a few projects at the NRAO

Further, there are far too many libraries for the web which could potentially ease the process of making a sharp UI and plotting simple data.

To make things simple for the user, let's assume that we would like to support any modern browser with Javascript enabled, and nothing more.

Approach

I will approach this problem in two phases:
  1. Allow users to call the python script from the web, and simply print the 2-D data to a web page.
  2. Take that 2-D data and plot it using a Javascript plotting tool.

The TurboGears Approach

Assuming you can easily get a hold of a TurboGears installation, it's relatively painless to get a new project up and running. It should be easy to embed a python script into a TurboGears project, and once we get started, we might want to expand our functionality with various TurboGears features.

NOTE: This is not meant to be a stand-alone exhaustive TurboGears tutorial. Instead, it complements the core documentation and highlights each step I took to create a simple python web plot x,y utility. See the respective documentation for details on TurboGears, genshi, and other tools.

Get TurboGears

For this run, I am using TurboGears 1.0 on RedHat EL4 with a custom-built python installation, but you can get up and running on any python installation. Python users can install new modules easily using easy_install (see easy_install page for installation instructions). Like many modules which are installed with easy_install, TurboGears is installed by default to your python installation's site-packages directory. So if you have sudo access or a custom-built python installation (for the easiest installation, you need write access to the site-packages directory), easy_install can get you up and running quickly with new python modules.

easy_install TurboGears

I've found that, for the most part, easy_install to-date is excellent for quickly adding python modules and their dependencies. However, sometimes things fail on install and you're left all alone. In extreme cases, I've had to download the module from its respective homepage and adjust the setup.py to fit my installation, but usually I've only had to run a few extra easy_install commands.

easy_install TurboGears
# TurboGears installation failed on attempt to install dependency simplejson
easy_install simplejson
easy_install TurboGears
# Success!

Note that TurboGears often allows you to choose your own tools in place of the defaults. For example, for *HTML/XML templating, I often like to use genshi over the TurboGears 1.0 default, kid.

easy_install genshi

Further, should a package yell at you about proper versions, you can use easy_install to install a different version of a package.

easy_install SQLAlchemy==0.4
easy_install SQLObject==0.9

If at any point you have questions about TurboGears, see the official documentation at http://docs.turbogears.org/. Once installed, you can check out the tutorials, especially the "20 Minute Wiki" (which might take 60 - 90 minutes for very new users).

Start a TurboGears Project

From a shell on Linux with python configured with TurboGears, genshi, and other goodies, run:
$ tg-admin quickstart

tg-admin is a TurboGears admin command-line utility which provides a lot of handy commands. This command will ask 3 questions, then create a project directory with the base files you need to get going. I only answer that I'd like to call the project easyplotxy, then I simply hit enter to accept the defaults on the next two prompts.

Enter project name: easyplotxy
Enter package name [easyplotxy]:
Do you need Identity (usernames/passwords) in this project? [no]
Selected and ...
TurboGears is producing a lot of files...

You will now find a directory called easyplotxy. Make easyplotxy your working directory, and run start-easyplotxy.py:

./start-easyplotxy.py

On success, you should see a line to the effect of:

2008-02-01 09:20:42,041 cherrypy.msg INFO HTTP: Serving HTTP on http://localhost:8080/

Pointing your web browser to http://localhost:8080/, you'll notice that you have a TurboGears project running on port 8080.

NOTE: If you change application code, the TurboGears server is set to reload by default. Assuming you don't make any errors, you'll find that the server will stay up-to-date with each change we make.

What Are These Files?

TurboGears implements a Model-View-Controller design pattern. In essence:
  • Model - the database driving the content of the pages
  • View - the output which will appear in the web browser
  • Controller - the glue which puts it all together

Key files to note are:
  • easyplotxy/start-easyplotxy.py - the launch script for easyplotxy
  • easyplotxy/dev.cfg - the default config file for easyplotxy; for example, you can change the server port here
  • easyplotxy/easyplotxy/model.py - easyplotxy database model
  • easyplotxy/easyplotxy/templates/ - easyplotxy directory for view templates
  • easyplotxy/easyplotxy/controllers.py - easyplotxy controllers
  • easyplotxy/easyplotxy/static/ - easyplotxy directory for static files, such as Javascript, images, CSS, etc.

We are going to look primarily at the controllers and templates files.

We won't be using the model/database functionality at first, but it's good to know that it's there if in the future we should want to store our favorite data and/or provide a catalog of a set of data files.

A Fresh Start

Let's setup some boilerplate to give ourselves a fresh start.

Clean out controllers.py:

from turbogears import controllers, expose, flash
# from model import *

class Root(controllers.RootController):
    @expose(template="genshi:easyplotxy.templates.index")
    def index(self):
        return dict()

In the template directory, create index.html and site.html. This may look like a lot of code, but it will isolate the changes that we will make as we go.

index.html:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
      xml:lang="en" lang="en"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      xmlns:py="http://genshi.edgewall.org/">
  <xi:include href="./site.html" />

  <?python
    pageTitle = "Hello, World!"

    def getTitle():
        return pageTitle
  ?>

  <head>
    <title>${getTitle()}</title>
  </head>

  <body>
    <h2>${getTitle()}</h2>
  </body>
</html>

site.html:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      xmlns:py="http://genshi.edgewall.org/"
      py:strip="">

  <?python
    toolName = 'easyplotxy'

    def getToolName():
        return toolName

    def getTitle():
        pass
  ?>

  <!--! parse head of caller -->
  <py:match path="head" once="true">
    <head py:attrs="select('@*')">
      <title py:with="title = list(select('title/text()'))">
        ${getToolName()}<py:if test="title">: ${title}</py:if>
      </title>
      <link rel="stylesheet" href="/static/css/style.css" type="text/css" />
      <link href="/static/images/favicon.ico" rel="shortcut icon"
       type="image/x-icon" />
      ${select('*[local-name()!="title"]')}
    </head>
  </py:match>

  <!--! parse body of caller -->
  <py:match path="body" once="true">
    <body py:attrs="select('@*')">
      <div id="top">
        <!--! Insert site-level top-of-page content here -->
      </div>


      <div id="contents">
        <!--! contents of caller automatically goes here, don't edit -->
        ${select('*|text()')}
      </div>

      <div id="bottom">
        <!--! Insert site-level bottom-of-page content here -->
      </div>

    </body>
  </py:match>

</html>

The index.html file includes site.html file. We can put all site-/tool-level code in site.html (note comments where to put top-of-page and bottom-of-page contents) and all style code in style.css (easyplotxy/easyplotxy/static/css/style.css, created with project). We can copy the index.html file to another template and edit appropriately. If we made it this far, we are about to really speed things up!

Produce Dummy 2-D Data

Let's create some dummy data for {x | 0 <= x <= 10} and {y | -10 <= y <= 10}. Add to controllers.py:

import random

# ...

    @expose()
    def dummy(self):
        data = []
        for i in range(0,100):
            data.append([float(i) / 10, random.randrange(-10,10)])
        return dict(data=data)

The return dict(data=data) allows us to call on context data by the variable name data (we're not too creative are we?) in the template/view code. Here, however, we are performing a simple expose without any template. So, when we view http://localhost:8080/dummy we don't see any HTML, but the python dictionary which is passed to the view code.

{"tg_flash": null, "data": [[0.0, 2], [0.10000000000000001, -2], [0.20000000000000001, -2], [0.2999999...

Note: TurboGears uses 'tg_flash' for its flash operation, which we are not using here.

Now let's look at plotting...

A Look at Plotting

For X,Y plotting, we are looking to plot a 2-D line graph / scatter plot.

The dojo toolkit (not part of TurboGears, but a stand-alone Javascript toolkit) looks very compelling. It has dojoX which promises to give "unified 2D and 3D drawing" (http://dojotoolkit.org/projects/dojox). That sounds interesting, but dojo is a pretty big framework, and the dojox.charting 2D test didn't produce any charts when I loaded it into Firefox 2.0.0.6 on Linux.

Let's first look at something a bit more lightweight.

The flot tool has a simple zoom-able scatter plot example. Let's try that.

I downloaded flot at http://code.google.com/p/flot/downloads/list (v0.3) and unpacked it to easyplotxy/easyplotxy/static/javascript/flot.

Looking at the example, we need to eventually get data in the following format:
{label: "text", data: [[x, y]]}

Fortunately, this is intuitive in python. Even better, our dummy data is almost there. Let's remove the dummy method's expose decorator, and have it return a list to a new plot method. In controllers.py:

    def dummy(self):
        data = []
        for i in range(0,100):
            data.append([float(i) / 10, random.randrange(-10,10)])
        return data

    @expose()
    def plot(self):
        return dict(label="dummy data", data=self.dummy())

Now we can look at http://localhost:8080/plot and see what we will be passing into the view code we're about to write.

{"tg_flash": null, "label": "dummy data", "data": [[0.0, 3], [0.10000000000000001, 7], [0.20000000000000001, -8], [0.29999999999999999, -5], ...

We have the ingredients and the structure, let's press some buttons!

Plot!

Let's take the zoom-able example from earlier, strip out the test data, and apply our dummy data. We need to make 3 major changes:
  • point to the Javascript source files in our directory
  • replace getData with the data passed in from the plot method
  • make sure there are no hard-code min/max values for the plot

Start by copying the example to a new plot.html file in the templates directory. Change the script tags to point to our copies of the flot library. Remove the min and max tags from the various options settings; this is to allow flot to auto-scale our plots.

Now let's add the data passed in from the plot method. First, let's change dummy's return value from a list to a pure string. Then we can just drop the string into it proper place in the target HTML. Before we continue too far, we should also change the @expose() above def plot... in controllers.py to @expose(template="genshi:easyplotxy.templates.plot") ... once we are comfortable with the values that http://localhost:8080/plot is showing.

Here is the updated controllers.py:

from turbogears import controllers, expose, flash
import random

class Root(controllers.RootController):
    @expose(template="genshi:easyplotxy.templates.index")
    def index(self):
        # not yet used...
        return dict()

    def dummy(self):
        data = []
        for i in range(0,100):
            data.append(str([float(i) / 10, random.randrange(-10,10)]))
        return ", ".join(data)

    @expose(template="genshi:easyplotxy.templates.plot")
    def plot(self):
        return dict(label="dummy data", data=self.dummy())

Let's put data in it's place in our plot.html in the templates directory, and on success we see the following screenshot at http://localhost:8080/plot:

tg-flot-demo.png

Here is the full listing of plot.html:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
      xml:lang="en" lang="en"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      xmlns:py="http://genshi.edgewall.org/">
  <xi:include href="./site.html" />

  <?python
    pageTitle = "Flot Demo"

    def getTitle():
        return pageTitle
  ?>

  <head>
    <title>${getTitle()}</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <script language="javascript" type="text/javascript" src="/static/javascript/flot/excanvas.pack.js"></script>
    <script language="javascript" type="text/javascript" src="/static/javascript/flot/jquery.js"></script>
    <script language="javascript" type="text/javascript" src="/static/javascript/flot/jquery.flot.js"></script>
  </head>

  <body>
    <h2>${getTitle()}</h2>

    <div></div>

    <div style="float:left">
      <div id="placeholder" style="width:500px;height:300px"></div>
    </div>

    <div id="miniature" style="float:left;margin-left:20px;margin-top:50px">
      <div id="overview" style="width:166px;height:100px"></div>

      <p id="overviewLegend" style="margin-left:10px"></p>
    </div>

<script id="source" language="javascript" type="text/javascript">
$(function () {
    // setup plot
    function getData() {
        return [
            { label: "${label}", data: [${data}] }
        ];
    }

    var options = {
        legend: { show: false },
        lines: { show: true },
        points: { show: true },
        yaxis: { noTicks: 10 },
        selection: { mode: "xy" }
    };

    var plotData = getData();

    var plot = $.plot($("#placeholder"), plotData, options);

    // setup overview
    var overview = $.plot($("#overview"), plotData, {
        legend: { show: true, container: $("#overviewLegend") },
        lines: { show: true, lineWidth: 1 },
        shadowSize: 0,
        xaxis: { noTicks: 4 },
        yaxis: { noTicks: 3 },
        grid: { color: "#999" },
        selection: { mode: "xy" }
    });

    // now connect the two
    var internalSelection = false;

    $("#placeholder").bind("selected", function (event, area) {
        // do the zooming
        plot = $.plot($("#placeholder"), plotData,
                      $.extend(true, {}, options, {
                          xaxis: { min: area.x1, max: area.x2 },
                          yaxis: { min: area.y1, max: area.y2 }
                      }));

        if (internalSelection)
            return; // prevent eternal loop
        internalSelection = true;
        overview.setSelection(area);
        internalSelection = false;
    });

    $("#overview").bind("selected", function (event, area) {
        if (internalSelection)
            return;
        internalSelection = true;
        plot.setSelection(area);
        internalSelection = false;
    });
});
</script>

 </body>
</html>

Better Data Input

Let's extend this for better data input by adapting an RFI data example in place of our dummy data. Here is a new listing of controllers.py:

from turbogears import controllers, expose, flash, redirect, url
# from model import *
import random
import numpy as np
import string

class Root(controllers.RootController):
    @expose(template="genshi:easyplotxy.templates.index")
    def index(self):
        raise redirect(url('/plot'))

    def dummy(self):
        data = []
        for i in range(0,100):
            data.append(str([float(i) / 10, random.randrange(-10,10)]))
        return ", ".join(data)

    @expose(template="genshi:easyplotxy.templates.demo")
    def demo(self):
        return dict()

    @expose(template="genshi:easyplotxy.templates.plot")
    def plot(self):
        # self.write_file()
        meta, data = self.read_file()
        return dict(label="Intensity (Y) vs. Frequency (X)", meta=meta, data=data)

    def write_file(self):
        fd = file('rfi_example.dat', 'wb')
        fd.write('Antenna: 50-1000 MHz Omni\n')
        fd.write('Instrument: MS3456 Spectrum Analyzer\n')
        fd.write('Resolution: 30 kHz\n')
        fd.write('Integration Time: 0.0003 sec\n')
        fd.write('Frequency Units: MHz\n')
        fd.write('Intensiy Units: Kelvin\n')
        freq = np.arange(500.0, 550.0, 0.05, dtype=np.float32)
        temp = np.random.normal(loc=5.0, scale=1.0, size=len(freq)).astype(np.float32)
        temp[20:900:53] = 200.0
        temp[29:900:73] = 500.0
        fd.write('Number of Points: %d\n'%(len(freq)))
        freq.tofile(fd)
        temp.tofile(fd)
        fd.close()

    def read_file(self):
        meta = {'order': []}
        fd = file('rfi_example.dat', 'rb')
        meta['Antenna'] = fd.readline()[:-1]; meta['order'].append('Antenna')
        meta['Instrument'] = fd.readline()[:-1]; meta['order'].append('Instrument')
        meta['Resolution'] = fd.readline()[:-1]; meta['order'].append('Resolution')
        meta['Integration Time'] = fd.readline()[:-1]; meta['order'].append('Integration Time')
        meta['Frequency Units'] = fd.readline()[:-1]; meta['order'].append('Frequency Units')
        meta['Intensiy Units'] = fd.readline()[:-1]; meta['order'].append('Intensiy Units')
        last_line = fd.readline()
        num_pts = string.atoi(string.split(last_line)[3])
        freq = np.fromfile(fd, dtype=np.float32, count=num_pts)
        temp = np.fromfile(fd, dtype=np.float32, count=num_pts)
        data = []
        for i in range(num_pts):
            data.append(str([freq[i], temp[i]]))
        return meta, ", ".join(data)

Also, let's "re-brand" the page with some better titles (in plot.html).

tg-rfi-demo.png

Tada!

-- RonDuPlain - 01 Feb 2008
Topic revision: r1 - 2008-02-01, RonDuPlain
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback