tornado.httpclient — Non-blocking HTTP client

Blocking and non-blocking HTTP client interfaces.

This module defines a common interface shared by two implementations, simple_httpclient and curl_httpclient. Applications may either instantiate their chosen implementation class directly or use the AsyncHTTPClient class from this module, which selects an implementation that can be overridden with the AsyncHTTPClient.configure method.

The default implementation is simple_httpclient, and this is expected to be suitable for most users’ needs. However, some applications may wish to switch to curl_httpclient for reasons such as the following:

• curl_httpclient has some features not found in simple_httpclient, including support for HTTP proxies and the ability to use a specified network interface.
• curl_httpclient is more likely to be compatible with sites that are not-quite-compliant with the HTTP spec, or sites that use little-exercised features of HTTP.
• simple_httpclient only supports SSL on Python 2.6 and above.
• curl_httpclient is faster
• curl_httpclient was the default prior to Tornado 2.0.

Note that if you are using curl_httpclient, it is highly recommended that you use a recent version of libcurl and pycurl. Currently the minimum supported version is 7.18.2, and the recommended version is 7.21.1 or newer.

HTTP client interfaces

class tornado.httpclient.HTTPClient(async_client_class=None)

A blocking HTTP client.

This interface is provided for convenience and testing; most applications that are running an IOLoop will want to use AsyncHTTPClient instead. Typical usage looks like this:

http_client = httpclient.HTTPClient()
try:
    response = http_client.fetch("http://www.google.com/")
    print response.body
except httpclient.HTTPError, e:
    print "Error:", e

close()

Closes the HTTPClient, freeing any resources used.

fetch(request, **kwargs)

Executes a request, returning an HTTPResponse.

The request may be either a string URL or an HTTPRequest object. If it is a string, we construct an HTTPRequest using any additional kwargs: HTTPRequest(request, **kwargs)

If an error occurs during the fetch, we raise an HTTPError.

class tornado.httpclient.AsyncHTTPClient

An non-blocking HTTP client.

Example usage:

import ioloop

def handle_request(response):
    if response.error:
        print "Error:", response.error
    else:
        print response.body
    ioloop.IOLoop.instance().stop()

http_client = httpclient.AsyncHTTPClient()
http_client.fetch("http://www.google.com/", handle_request)
ioloop.IOLoop.instance().start()

The constructor for this class is magic in several respects: It actually creates an instance of an implementation-specific subclass, and instances are reused as a kind of pseudo-singleton (one per IOLoop). The keyword argument force_instance=True can be used to suppress this singleton behavior. Constructor arguments other than io_loop and force_instance are deprecated. The implementation subclass as well as arguments to its constructor can be set with the static method configure()

close()

Destroys this http client, freeing any file descriptors used. Not needed in normal use, but may be helpful in unittests that create and destroy http clients. No other methods may be called on the AsyncHTTPClient after close().

fetch(request, callback, **kwargs)

Executes a request, calling callback with an HTTPResponse.

The request may be either a string URL or an HTTPRequest object. If it is a string, we construct an HTTPRequest using any additional kwargs: HTTPRequest(request, **kwargs)

If an error occurs during the fetch, the HTTPResponse given to the callback has a non-None error attribute that contains the exception encountered during the request. You can call response.rethrow() to throw the exception (if any) in the callback.

static configure(impl, **kwargs)

Configures the AsyncHTTPClient subclass to use.

AsyncHTTPClient() actually creates an instance of a subclass. This method may be called with either a class object or the fully-qualified name of such a class (or None to use the default, SimpleAsyncHTTPClient)

If additional keyword arguments are given, they will be passed to the constructor of each subclass instance created. The keyword argument max_clients determines the maximum number of simultaneous fetch() operations that can execute in parallel on each IOLoop. Additional arguments may be supported depending on the implementation class in use.

Example:

AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")

Request objects

class tornado.httpclient.HTTPRequest(url, method='GET', headers=None, body=None, auth_username=None, auth_password=None, connect_timeout=20.0, request_timeout=20.0, if_modified_since=None, follow_redirects=True, max_redirects=5, user_agent=None, use_gzip=True, network_interface=None, streaming_callback=None, header_callback=None, prepare_curl_callback=None, proxy_host=None, proxy_port=None, proxy_username=None, proxy_password='', allow_nonstandard_methods=False, validate_cert=True, ca_certs=None, allow_ipv6=None, client_key=None, client_cert=None)

HTTP client request object.

Creates an HTTPRequest.

All parameters except url are optional.

Parameters:

• url (string) – URL to fetch
• method (string) – HTTP method, e.g. “GET” or “POST”
• headers (HTTPHeaders or dict) – Additional HTTP headers to pass on the request
• auth_username (string) – Username for HTTP “Basic” authentication
• auth_password (string) – Password for HTTP “Basic” authentication
• connect_timeout (float) – Timeout for initial connection in seconds
• request_timeout (float) – Timeout for entire request in seconds
• if_modified_since (datetime) – Timestamp for If-Modified-Since header
• follow_redirects (bool) – Should redirects be followed automatically or return the 3xx response?
• max_redirects (int) – Limit for follow_redirects
• user_agent (string) – String to send as User-Agent header
• use_gzip (bool) – Request gzip encoding from the server
• network_interface (string) – Network interface to use for request
• streaming_callback (callable) – If set, streaming_callback will be run with each chunk of data as it is received, and body and buffer will be empty in the final response.
• header_callback (callable) – If set, header_callback will be run with each header line as it is received, and headers will be empty in the final response.
• prepare_curl_callback (callable) – If set, will be called with a pycurl.Curl object to allow the application to make additional setopt calls.
• proxy_host (string) – HTTP proxy hostname. To use proxies, proxy_host and proxy_port must be set; proxy_username and proxy_pass are optional. Proxies are currently only support with curl_httpclient.
• proxy_port (int) – HTTP proxy port
• proxy_username (string) – HTTP proxy username
• proxy_password (string) – HTTP proxy password
• allow_nonstandard_methods (bool) – Allow unknown values for method argument?
• validate_cert (bool) – For HTTPS requests, validate the server’s certificate?
• ca_certs (string) – filename of CA certificates in PEM format, or None to use defaults. Note that in curl_httpclient, if any request uses a custom ca_certs file, they all must (they don’t have to all use the same ca_certs, but it’s not possible to mix requests with ca_certs and requests that use the defaults.
• allow_ipv6 (bool) – Use IPv6 when available? Default is false in simple_httpclient and true in curl_httpclient
• client_key (string) – Filename for client SSL key, if any
• client_cert (string) – Filename for client SSL certificate, if any

Response objects

class tornado.httpclient.HTTPResponse(request, code, headers={}, buffer=None, effective_url=None, error=None, request_time=None, time_info={})

HTTP Response object.

Attributes:

• request: HTTPRequest object

• code: numeric HTTP status code, e.g. 200 or 404

• headers: httputil.HTTPHeaders object

• buffer: cStringIO object for response body

• body: respose body as string (created on demand from self.buffer)

• error: Exception object, if any

• request_time: seconds from request start to finish

• time_info: dictionary of diagnostic timing information from the request.

  Available data are subject to change, but currently uses timings available from http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html, plus ‘queue’, which is the delay (if any) introduced by waiting for a slot under AsyncHTTPClient’s max_clients setting.

rethrow()

If there was an error on the request, raise an HTTPError.

Exceptions

exception tornado.httpclient.HTTPError(code, message=None, response=None)

Exception thrown for an unsuccessful HTTP request.

Attributes:

code - HTTP error integer error code, e.g. 404. Error code 599 is

used when no HTTP response was received, e.g. for a timeout.

response - HTTPResponse object, if any.

Note that if follow_redirects is False, redirects become HTTPErrors, and you can look at error.response.headers[‘Location’] to see the destination of the redirect.

Command-line interface

This module provides a simple command-line interface to fetch a url using Tornado’s HTTP client. Example usage:

# Fetch the url and print its body
python -m tornado.httpclient http://www.google.com

# Just print the headers
python -m tornado.httpclient --print_headers --print_body=false http://www.google.com