Tag Archives: cherrypy

Python Web Frameworks – Application Object

In response to my previous post, someone asked “… what’s the point? How one way is better than the other?” My response was that I’m “not trying to judge these frameworks from the perspective of a developer (at least not completely), but rather from the POV of ‘if I were writing a framework, which features are essential, which nice to have, which I can disregard'” (emphasis added). Of course, I have my own biases as a developer as to what I consider “nice” or essential.

With that clarification out of the way, let’s look at the next feature: the WSGI callable generally known as the application object.

Django doesn’t want developers to be concerned with this artifact. A Django application is the collection of model, views (controllers in standard MVC terminology) and templates (views) that gets run by the server. For those interested, the Django WSGI callable is an instance of WSGIHandler (in django.core.handlers.wsgi).

As best as I can tell, Twisted Web expects the developer to write the application object. It will get called from the WSGIResponse run() method in twisted.web.wsgi. As its documentation says, Twisted Web is “a framework for doing things with the web” but is “not a ‘web framework’ in the same sense” as Django, so it doesn’t fit well in this comparison.

Pyramid‘s Router class (in pyramid.router) is what implements the WSGI callable. The Router instance gets created from the Configurator class’s make_wsgi_app() method. Select Router attributes and methods:

  • logger
  • root_factory
  • routes_mapper
  • request_factory
  • registry (from args)
  • handle_request()

In CherryPy, the Application class is the application object, although it’s not generallly used directly. Instead a user root object is mounted on a Tree which is a registry of these applications. Major attributes and methods:

  • root (from args): root object (application handler)
  • namespaces, config: for configuration
  • wsgiapp(): a CPWSGIApp instance, to handle apps in a pipeline

Being a support library, Werkzeug doesn’t include an application object, but it has a sample in its Shortly example. Major attributes and methods:

  • dispatch_request(): dispatcher
  • jinja_env: template enviroment
  • url_map: Map() taking list of Rule()’s
  • error_404(): error handler
  • render_template(): template renderer

Like Django, Web2py isn’t interested in having developers worry about the application object. Its WSGI callable is the standalone wsgibase() function, in gluon.main.

The web.py application class (yes, not capitalized) implements its WSGI callable. Major attributes and methods:

  • autoreload (from args)
  • init_mapping (from args)
  • request()
  • handle(): matches path to mapping
  • run(): runs the developer server
  • notfound(), internalerror(): error handlers

Flask‘s Flask class is its application object, derived from flask.helpers._PackageBoundObject. Selected attributes and methods (in addition to those in Werkzeug’s example above):

  • static_url_path, static_folder (from args)
  • instance_path (from args or auto-determined)
  • view_functions
  • before/after_xxx_funcs (dicts)
  • logger
  • run()
  • route(): decorator for URL rules
  • error_handler_spec (a dict)
  • errorhandler(): decorator for error handler functions

Like Werkzeug, WebOb doesn’t include an application object, but it has a sample in its wiki example, most of it very specific to the example.

Bottle‘s Bottle class is its WSGI callable. Major attributes and methods:

  • routes (list) and router (class Router instance)
  • config (class ConfigDict instance)
  • mount(): to mount an app at a given point
  • match(): to search for matching routes
  • route(): decorator to bind a function to a URL

Pesto‘s DispatcherApp class is its application object. Major attributes and methods:

  • prefix (from args): a “mount” point
  • matchpattern(), match(): matches URLs (match is decorator)
  • urlfor(): handler/dispatcher to URL converter
  • gettarget(): returns a four-part tuple from the URI
  • status404_application(): error handler

Diva‘s Application class is the object of interest. Select attributes and methods:

  • config (from kwargs), configure()
  • routing_cfg and routing
  • locale_dir (from args)
  • template_dirs (from args)
  • templates(): template loader
  • prepare() and cleanup(): before and after methods

Summary: For developers who care about writing application code, Django and Web2py and to some extent Pyramid and CherryPy hide the existence of the WSGI callable, while other frameworks require that the programmer instantiate it or invoke it. If I were to write a framework, I’d want to make it easy on the developer as the former projects do, but wouldn’t keep it completely out of sight.

Python Web Frameworks – Development Server

One of the most frequent tasks of a web developer is running the app while it’s being written. So we’ll begin our examination of Python web frameworks by looking at the development servers provided and how does one run the app.

Django developers are used to invoking

python manage.py runserver

The runserver option runs Django’s development server which is invoked via the run() function in django.core.servers.basehttp. This in turn runs a WSGIServer which is derived from Python’s wsgiref’s simple_server.

Twisted Web has its own server based on its core reactor object. Here’s a simple usage example:

from twisted.web import server, resource
from twisted.internet import reactor

class HelloResource(resource.Resource):
    ...

reactor.listenTCP(8080, server.Site(HelloResource()))
reactor.run()

Pyramid uses Paste’s paste.httpserver which in turn is based on Python’s BaseHTTPServer. Sample usage:

from paste.httpserver import serve
from pyramid.config import Configurator

def hello_world(request):
   ... 

if __name__ == '__main__':
   config = Configurator()
   ...
   app = config.make_wsgi_app()
   serve(app, host='0.0.0.0')

In CherryPy, the normal server is cherrypy.engine which is invoked as:

cherrypy.engine.start()
cherrypy.engine.block()

cherrypy.engine is actually an instance of cherrypy.process.wspbus.Bus. A simpler alternative to the engine calls is to use:

cherrypy.quickstart(app())

Under the covers, CherryPy also defines a global server variable that is an instance of Server which implements the HTTP server, with help from cherrypy._cpwsgi_server.CPWSGIServer.

Werkzeug has a built-in server, used as follows:

from werkzeug.serving import run_simple
from myproject import make_app

app = make_app(...)
run_simple('localhost', 8080, app, use_reloader=True)

run_simple() invokes its own (werkzeug.serving) make_server which creates one of three types of XxxWSGIServer’s, which are derived from Python’s BaseHTTPServer.HTTPServer.

When Web2py is invoked, a GUI dialog is shown. The dialog is implemented in gluon.widget and its web2pyDialog.start method creates a main.HttpServer. gluon.main’s HttpServer class uses the Rocket server whose code is in gluon.rocket.

In web.py, to run an application you instantiate a web.application and invoke its run() method, e.g.:

app = web.application(urls, globals())
app.run()

This eventually runs a BaseHTTPServer.HTTPServer from the Python standard library.

Flask‘s WSGI application class (Flask) has a run() method:

run(host='127.0.0.1', port=5000, debug=None, **options)

This invokes the Werkzeug serving.run_simple function (see above).

WebOb doesn’t offer any server per se, but suggests using Paste’s httpserver.serve or wsgiref.simple_server.

Bottle has a standalone run() function:

run(app=None, server='wsgiref', host='127.0.0.1', port=8080,
    interval=1, reloader=False, quiet=False, plugins=None, **kargs)

This runs one of several web servers supported, by default, wsgiref, which uses the wsgiref.simple_server.

Pesto suggests using the Python wsgiref directly, e.g.:

import pesto
dispatcher = pesto.dispatcher_app()
...
httpd = make_server('', 8080, dispatcher)
httpd.serve_forever()

Diva uses Python’s WSGIServer from the standard library to implement its main() and serve() standalone functions. The former can be used as follows:

from diva.core import Application
from diva.server import main

class MyApp(Application): pass

main(MyApp())

Summary: With the exception of Twisted and CherryPy, all frameworks base their development servers on the standard library’s wsgiref or BaseHTTPServer, Paste or Rocket (which claims to be CherryPy-compatible). Lesson for prospective framework creators: Don’t write your own server. However, you may want to enhance your server with auto-reloading, threading, debugging. etc.

There are two approaches to running the app or server. Django and Web2py don’t want the developer to bother with writing app.run() or similar, the other frameworks expect the programmer to plug the various pieces together.

Best wishes for 2012 to all readers!

Python Web Frameworks – Candidates

I’ve been researching web frameworks, partly with a view to writing my own, and thought I’d share my findings. This first post will present the candidates briefly and subsequent posts will delve into particular features.

The candidates are presented based on the number of users as found on Ohloh, as a rough measure of popularity.  Each includes the number of downloads for the latest release on PyPI (sometimes misleading if a package had a recent release) and the salient parts of its advertised description (caveat emptor!). All of them are open source and written in Python, but some of them are WSGI libraries rather than full frameworks.

Django (843 users, 126,426 downloads): A “high-level … Web framework that encourages rapid development and clean, pragmatic design.”

Twisted Web (136 users, 14,413 downloads [both for Twisted]):  An “HTPP server that can be used as a library or run … stand-alone …, an HTML templating engine [and] an HTTP client library.”

Pyramid (aka Pylons on Ohloh, 125 users, 606 + 640 downloads): A “very general … web framework [designed] to make it easier for a developer to create an arbitrary web application.”

CherryPy (75 users, PyPI download info not available): A “minimalist … pythonic, object-oriented web framework.”

Werkzeug (31 users, 4225 downloads): A “WSGI utility library, … [it] is Simple … And Powerful.”

Web2py (21 users, 1157 downloads): A “full-stack framework for rapid development of fast, scalable, secure and portable database-driven web-based applications.”

Web.py (18 users, 11,708 downloads): A “web framework … that is as simple as it is powerful” (hmmm … see Werkzeug above).

Flask (16 users, 42,657 downloads): A “microframework … based on Werkzeug, Jinja 2 and good intentions.”

WebOb (16 users, 29,664 downloads): A “library that provides wrappers around the WSGI request environment, and an object to help create WSGI responses.”

Bottle (4 users, 163 downloads): A “fast, simple and lightweight WSGI micro web-framework.”

Pesto (no users, 421 downloads): Not a framework, but a library for “writing WSGI web applications.”

Diva (not on Ohloh or PyPI): This is Christopher Lenz’s  framework experimentation sandbox: “a lightweight web framework … built on top of WSGI and integrated with … Genshi.” My first, unpublished attempt at writing a framework was based on Diva.

Left Out: I had to draw the line somewhere. Compared to Richard Jones’ micro-framework battle, I included three of what he called “mega frameworks” and excluded three of his micro frameworks. For the record, here are the ones I left out: Aspen, Bobo, CubicWeb, Grok, Itty, Milla, Nagare, Nevow, Pump, Pyjamas, Pylons, Python Paste, Quixote, Spyce, Tornado, TurboGears, Webware, Zope 2 and Zope 3Bluebream. But that’s not all: see the Python.org wiki, WSGI.org and Wikipedia.

Unlike Richard, I will not attempt to develop an application with each framework. Instead, I’ll analyze specific features along the lines of Christopher’s Diva documentation.

Dueling Frameworks, revisited

Werkzeug

In the closing remarks of “Dueling Frameworks”, I hinted that I may continue to explore Flask. That road first led me to take a closer look at the recent state of Werkzeug.

As I had done with Flask and CherryPy, the natural first step was to port the minimalist WSGI database user interface (but taking advantage of the robustness and templating (Jinja2) enhancements) to Werkzeug. This turned out to be very straightforward (see GitHub tag v0.3.2). Key changes:

  • replace my Request/Response objects by Werkzeug’s own
  • add a request argument to the various handler methods
  • simplify the dispatch methods, using request.path and request.form
  • replace the DatabaseApp.__call__ by a much simpler wsgi_app
  • eliminate serve_css (static files are served by Werkzeug’s SharedDataMiddleware)
  • use Werkzeug’s development server.

The second modification (see tag v0.3.3) was to use Werkzeug’s Map and Rule classes to refactor the URL routing. What I liked about this, compared to Flask’s, is that I could keep the delegation of routing within the FilmHandler class.

The final version —for now (see tag v0.3.4), improves the URLs to make them more RESTful. While they’re not completely in agreement with Oliver Charles comments, they’re a step in the right direction. This is something I could not achieve with CherryPy (without bringing in or writing an external module).

The bottom line is that I’ve decided to use Werkzeug for continuing with this tutorial. Perhaps what won me over was its debugger. Also, Werkzeug is a library, not a framework or even a micro-framework, so it feels like a much lighter dependency than CherryPy or Flask. If there is a concern, it is that Werkzeug lacks support for Python 3. This may make a difference —in terms of “traction”— now that WebOb has been ported, but I’m hoping Werkzeug/Flask fans will correct that in the not too distant future.

Dueling Frameworks

Cue the music

In this corner, the time-tested CherryPy, standing at version 3.2, six years old, enabled for Python 3, sporting an HTTP 1.1-compliant WSGI webserver, support for other WSGI servers or adapters, a plugin mechanism, built-in tools and much more.

In the other corner, the newcomer Flask, at version 0.7.2, about one year old (but with older ancestry), claiming to be a “microframework … with good intentions” and “fun.”

In my post A Funny Thing Happened on the Way to the Webserver I mentioned I took a look at the second contender and liked what I saw. First impressions, as they say, can be deceiving. What may seem appealing is sometimes harder to use or disappointing when you get down to writing code and testing it.

Coincidentally, after I had decided on the “Dueling Banjos” theme, Audrey Roy pointed me1 to Richard Jones’ “Web micro-framework Battle!” which provided additional food for thought.

As you may expect, my selection criteria don’t exactly match Richard’s. Perhaps the most striking difference is how he implemented routing (URL mapping, page 14 of the presentation) for the baseline cgi+wsgiref implementation vs. my choice for doing dispatching in my minimalist WSGI database UI (see in particular, the dispatch methods in dbapp.py and film.py).

Like Richard, I don’t appreciate “magic” in a framework or application. but I’m less concerned with strictly RESTful URLs. Richard appears to prefer the decorator approach to URL mapping (as seen by his Bottle, Flask and Pesto examples, among others), but also used URL mapping tables (seen in the web.py, Werkzeug and baseline examples).

The latter approach reminds me of a generic menu interface I wrote eons ago in C. It may be OK for a vtable in C++, but I usually find it inappropriate for application level code. In fact, the mapping table seems “magical” since it doesn’t encapsulate the code with the corresponding URL.

Although Flask looked attractive due to its URL routing decorator, it now appears limiting because it can only be used on functions but not on class methods. (Note: I haven’t explored all of Flask, e.g., haven’t looked at its Blueprints).

As a result, refactoring the WSGI application to use CherryPy was quite straightforward, whereas changing it to accomodate Flask involved quite a bit of surgery. This is exemplified by these Git stats:

 cherrypy/film.py |   73 ++++++++++++++++-------------------------------------
 1 files changed, 22 insertions(+), 51 deletions(-)

 flask/film.py |  235 ++++++++++++++++++++++++++-------------------------------
 1 files changed, 107 insertions(+), 128 deletions(-)

While many modifications simply involved indentation and removing the self parameter, other changes were necessary. For example, as there can only be one mapping to a function named ‘index’, I had to rename the film.py ‘index’ function/method to ‘list’ to avoid conflict with dbapp.py. This does not bode well for extensibility: a real application would support multiple entities—films, actors, customers, etc.—with similar function/methods. Maybe this is what Blueprints are for, but why invent another wheel when Python classes are available?

There were other minor annoyances with Flask, like how to instantiate its application object without adding a subdirectory, or how to pass the database connection to the film.py functions without using a global variable.

I may continue to explore Flask, although at the moment CherryPy seems more suitable to my purposes. I’m also intrigued by some concepts of Richard’s winners—Pesto and Bottle. A Pesto-like dispatcher with support for class methods on top of CherryPy would probably make my day.

The code implementing the CherryPy database UI interface is on GitHub tagged as version v0.3.0 and the corresponding Flask code is also there tagged as v0.3.1.


1 BTW, +1 for Audrey’s wish list item: Python in the Browser (see pages 16-18 of her presentation).

A Minimalist WSGI Database User Interface

In a previous post, I presented a minimalist command line database user interface. Subsequently, I separated the business logic into its own module and presented an outline design for a web interface. Today I’ll go over a minimalist WSGI database interface. To try it out, do the following:

$ git clone git://github.com/jmafc/database-ui-tutorial.git dbui
$ cd dbui
$ createdb moviesdev
$ yamltodb moviesdev film.yaml | psql moviesdev
$ cd wsgi
$ python dbapp.py moviesdev

The pre-requisites (Git, PostgreSQL, Python, Psycopg2 and Pyrseas) have not changed. If you previously experimented with the command line interface, you’ll want to do a git pull and then the last two steps above. To browse the code, visit GitHub.

Caveat: I used Python’s standard library wsgiref package to develop the barest functionality that I needed for this tutorial. It’s not an attempt to create YAPWF (Yet Another Python Web Framework). With that said, since we’re dealing with a film database, let’s review the various protagonists and crew members.

Supporting Cast

server.py

This is about the most basic WSGI server you can write. For a real appplication, you’ll want to use CherryPy (perhaps fronted by nginx or Apache), Apache with mod_wsgiFlask or any number of other WSGI frameworks.

static/*.css

Two Cascading Style Sheets are provided: base.css and app.css. They implement a theme similar to Django’s admin application.

URL routing

Routing of requests occurs in the dispatch methods of classes DatabaseApp (in dbapp.py) and FilmHandler (in film.py), see Main Characters below. Everything is hard-coded with if/elif/else logic. In real life, you’d want to use something like CherryPy’s exposed objects, Routes or the routing features of your framework.

templating.py

You can’t get much more austere than this templating “engine.” It expects to find template files at a fixed location (see below) and a dictionary for string formatting. For a non-tutorial application, take a look at Mako, Jinja or the templating features of your web framework.

templates/*.html

The home.html template presents a trivial menu. The error/404.html file is used for Not Found responses. The templates/film subdirectory has four files: list.html displays all movies in an HTML table, new.html is used to accept input for a new record, edit.html to display an existing record and allow changes or deletion, and delete.html requests confirmation prior to executing a DELETE.

dblib.py

This simple database interface library is making a repeat appearance, but it has been revamped into a DbConnection class so that it can be easily shared by the main players.

bl.py

The business logic module is essentially unchanged, only adapted to use the DbConnection class.

errors.py

This just declares two Exception classes for use by the two main modules.

Main Characters and Crew

dbapp.py

Continuing with movie analogies, dbapp.py runs the show, like a film director. It defines two simple Request and Response classes and the main DatabaseApp class. The latter’s main attraction it its __call__ method, the heart of a WSGI interface. In as few lines as possible, it implements:

  • serving the normal HTML responses
  • serving the CSS files
  • serving HTTP 404 Not Found responses
  • simple redirects to its own URLs

film.py

This is the leading performer and accordingly it has the most lines. It first defines a FilmForm class which encapsulates the HTML form or HTTP query string data. Its validate method incorporates the field validation originally present in the edit function of cmdline/dbapp.py.

The FilmHandler class implements the methods to respond to each HTTP request. Consider what happens when the user selects the /film/new link from the menu or film listing pages. The dispatch methods then cause the FilmHandler new method to execute which renders the new.html form.

After the user enters the data and clicks Save, the dispatch methods invoke the create method. This uses FilmForm to extract and validate the data. If a validation error is present, the form is redisplayed with a simplified error mechanism. Otherwise, the business logic insert function is called. If a database error occurs, the form is redisplayed with the error from psycopg2 and PostgreSQL. If everything is OK, the transaction is committed and the application redirects to the film listing page.

For now, the application does not use any Javascript, so it’s overall interaction is not the most user friendly. However, if you’re new to WSGI, I hope you’ll take some time to review the application and that you’ll find it instructive.