Easy to use, flexible HTTP access method in Python

Python comes with 'batteries included' as they like to say. This means that libraries for most tasks you need to tackle are already part of a standard Python install. In order to issue HTTP requests, the most common libraries mentioned are urllib and urllib2. They offer the very convenient urlopen() function, which make it a snap to quickly retrieve online resources.

One thing that always bothered me, however, was the inflexibility of those methods. Want to use PUT instead of POST? Out of luck. Need authentication? Deal with pesky URLopener classes. Want timeouts for your requests? Use global socket options or use Python 2.6. Want to specify some custom headers? Deal with more classes. And so on.

All I want is a simple function like urlopen(), which can do all of these things for me and which also works on Python 2.5 (I'm often using Jython, which still is at 2.5). Of course, there is always the low level httplib, on top of which the two urllibs actually build. And while using that library requires a few more steps, you can use it to write yourself your powerful, simple version of urlopen().

So, for RESTx - an open source project to simply and easily create RESTful web services - I wrapped all the functionality I wanted into a convenient, compact function that uses httplib directly. Currently, it only supports HTTP's basic authentication, but hopefully that can be extended at some point in the future.

Here it is then for your enjoyment. Let me know if you think this is useful. Oh, and while you are here, please follow me on Twitter.

Update: Several people pointed out httplib2 to me, which offers a lot of the convenience I was looking for. However, it is not available in Jython, or Python 2.5 in general.

import httplib
import urllib
import socket
import urlparse

def http_access(method, url, data=None, headers=None, timeout=None, credentials=None):
"""
Access an HTTP resource with GET or POST.

@param method:         The method for the HTTP request: GET, POST, etc.
@type method:          string

@param url:            The URL to access.
@type url:             string

@param data:           If present it specifies the data for a POST or PUT request.
@type data:            Data to be sent or None.

@param headers:        A dictionary of additional HTTP request headers or None.
@type headers:         dict

@param timeout:        Timeout for the request in seconds, or None. Specified as
                      floating point value, so you can set sub-second timeouts.
@type timeout:         float

@param credentials:    A username/password tuple to support basic HTTP authentication.
@type credentials:     tuple

@return:               Code and response data tuple.
@rtype:                tuple

"""
(scheme, host_port, path, params, query, fragment) = urlparse.urlparse(url)
allpath     = url[url.index(host_port)+len(host_port):]
host, port  = urllib.splitport(host_port)

if not headers:
   headers = dict()

if credentials:
   accountname, password = credentials
   headers["Authorization"] = "Basic " + base64.encodestring('%s:%s' % (accountname, password))[:-1]

if scheme == 'https':
   conn = httplib.HTTPSConnection(host, port)
else:
   conn = httplib.HTTPConnection(host, port)

conn.request(method, allpath, data, headers)
conn.sock.settimeout(timeout)

try:
   resp   = conn.getresponse()
   code   = resp.status
   data   = resp.read()
except socket.timeout, e:
   return 408, httplib.responses[408]

return code, data



if __name__ == '__main__':
# Some usage example: Getting data in JSON format from a ficticious server with
# a timeout of 0.5 seconds.
status, data = http_access("GET", "http://localhost:8001", data = None,
                          headers = { "Accept" : "application/json" }, timeout=0.5 )

print "@@@ HTTP response code: ", status
print "@@@ Received data:      ", data