Easy to use, flexible HTTP access method in Python

Python comes with 'batteries included' as they like to say. This means that libraries for most tasks you need to tackle are already part of a standard Python install. In order to issue HTTP requests, the most common libraries mentioned are urllib and urllib2. They offer the very convenient urlopen() function, which make it a snap to quickly retrieve online resources.

One thing that always bothered me, however, was the inflexibility of those methods. Want to use PUT instead of POST? Out of luck. Need authentication? Deal with pesky URLopener classes. Want timeouts for your requests? Use global socket options or use Python 2.6. Want to specify some custom headers? Deal with more classes. And so on.

All I want is a simple function like urlopen(), which can do all of these things for me and which also works on Python 2.5 (I'm often using Jython, which still is at 2.5). Of course, there is always the low level httplib, on top of which the two urllibs actually build. And while using that library requires a few more steps, you can use it to write yourself your powerful, simple version of urlopen().

So, for RESTx - an open source project to simply and easily create RESTful web services - I wrapped all the functionality I wanted into a convenient, compact function that uses httplib directly. Currently, it only supports HTTP's basic authentication, but hopefully that can be extended at some point in the future.

Here it is then for your enjoyment. Let me know if you think this is useful. Oh, and while you are here, please follow me on Twitter.

Update: Several people pointed out httplib2 to me, which offers a lot of the convenience I was looking for. However, it is not available in Jython, or Python 2.5 in general.

import httplib
import urllib
import socket
import urlparse

def http_access(method, url, data=None, headers=None, timeout=None, credentials=None):
"""
Access an HTTP resource with GET or POST.

@param method:         The method for the HTTP request: GET, POST, etc.
@type method:          string

@param url:            The URL to access.
@type url:             string

@param data:           If present it specifies the data for a POST or PUT request.
@type data:            Data to be sent or None.

@param headers:        A dictionary of additional HTTP request headers or None.
@type headers:         dict

@param timeout:        Timeout for the request in seconds, or None. Specified as
                      floating point value, so you can set sub-second timeouts.
@type timeout:         float

@param credentials:    A username/password tuple to support basic HTTP authentication.
@type credentials:     tuple

@return:               Code and response data tuple.
@rtype:                tuple

"""
(scheme, host_port, path, params, query, fragment) = urlparse.urlparse(url)
allpath     = url[url.index(host_port)+len(host_port):]
host, port  = urllib.splitport(host_port)

if not headers:
   headers = dict()

if credentials:
   accountname, password = credentials
   headers["Authorization"] = "Basic " + base64.encodestring('%s:%s' % (accountname, password))[:-1]

if scheme == 'https':
   conn = httplib.HTTPSConnection(host, port)
else:
   conn = httplib.HTTPConnection(host, port)

conn.request(method, allpath, data, headers)
conn.sock.settimeout(timeout)

try:
   resp   = conn.getresponse()
   code   = resp.status
   data   = resp.read()
except socket.timeout, e:
   return 408, httplib.responses[408]

return code, data



if __name__ == '__main__':
# Some usage example: Getting data in JSON format from a ficticious server with
# a timeout of 0.5 seconds.
status, data = http_access("GET", "http://localhost:8001", data = None,
                          headers = { "Accept" : "application/json" }, timeout=0.5 )

print "@@@ HTTP response code: ", status
print "@@@ Received data:      ", data

 

 

Mixed Jython/Python development and Google AppEngine oddities

Recently, I worked on a Python project, which I developed as a stand-alone WSGI application. I also experimented with running this app in Jython. In case you don't know: Jython is a neat Python implementation, which runs in the Java VM. The advantage of this is that you have access to all the usual Java infrastructure, framework and libraries, while being able to rapidly develop your program in the Python language. I ended up designing my application so that it could run both as Python as well as Jython project.

Shortly afterwards, I moved my WSGI application to Google's AppEngine, which is is a hosted environment for Python and Java apps. AppEngine can run pure WSGI applications quite easily, so this shouldn't be too much of a problem. Using Google's local development server (dev_appserver.py), I was able to run my application on my desktop in the emulated Google environment without problem.

I then used the provided command line utility (appcfg.py) to move my app to Google's actual servers... and nothing worked. I received a "500 Internal Server Error". Looking at the logs I found that my application was not able to import any of my own modules. While standard library items were imported without an issue, any module I had provided myself resulted in an import error.

What had happened? It took me a long time to figure this out, so I thought I share it with you: When running the Python application via Jython, a bunch of .class files were created, which is normal. However, when the application was uploaded via appcfg.py, those class files were uploaded as well (files ending in *.pyc and a few other extensions are ignored, but apparently anything else is considered to be part of your application).

So then, for reasons that aren't quite clear, the presence of .class files in your Python application seems to be really confusing for Google AppEngine.

The lesson? If you are developing an application that can run locally in Jython as well as on Google's AppEngine, make sure to clear out your *.class files before uploading. It can save you hours of grief.

Oh, and by the way, you should follow me on twitter here.