tornado.httpclient
— Asynchronous HTTP client¶
Blocking and non-blocking HTTP client interfaces.
This module defines a common interface shared by two implementations,
simple_httpclient
and curl_httpclient
. Applications may either
instantiate their chosen implementation class directly or use the
AsyncHTTPClient
class from this module, which selects an implementation
that can be overridden with the AsyncHTTPClient.configure
method.
The default implementation is simple_httpclient
, and this is expected
to be suitable for most users’ needs. However, some applications may wish
to switch to curl_httpclient
for reasons such as the following:
curl_httpclient
has some features not found insimple_httpclient
, including support for HTTP proxies and the ability to use a specified network interface.curl_httpclient
is more likely to be compatible with sites that are not-quite-compliant with the HTTP spec, or sites that use little-exercised features of HTTP.curl_httpclient
is faster.
Note that if you are using curl_httpclient
, it is highly
recommended that you use a recent version of libcurl
and
pycurl
. Currently the minimum supported version of libcurl is
7.22.0, and the minimum version of pycurl is 7.18.2. It is highly
recommended that your libcurl
installation is built with
asynchronous DNS resolver (threaded or c-ares), otherwise you may
encounter various problems with request timeouts (for more
information, see
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTCONNECTTIMEOUTMS
and comments in curl_httpclient.py).
To select curl_httpclient
, call AsyncHTTPClient.configure
at startup:
AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
HTTP client interfaces¶
- class tornado.httpclient.HTTPClient(async_client_class: Optional[Type[AsyncHTTPClient]] = None, **kwargs: Any)[source]¶
A blocking HTTP client.
This interface is provided to make it easier to share code between synchronous and asynchronous applications. Applications that are running an
IOLoop
must useAsyncHTTPClient
instead.Typical usage looks like this:
http_client = httpclient.HTTPClient() try: response = http_client.fetch("http://www.google.com/") print(response.body) except httpclient.HTTPError as e: # HTTPError is raised for non-200 responses; the response # can be found in e.response. print("Error: " + str(e)) except Exception as e: # Other errors are possible, such as IOError. print("Error: " + str(e)) http_client.close()
Changed in version 5.0: Due to limitations in
asyncio
, it is no longer possible to use the synchronousHTTPClient
while anIOLoop
is running. UseAsyncHTTPClient
instead.- fetch(request: Union[HTTPRequest, str], **kwargs: Any) HTTPResponse [source]¶
Executes a request, returning an
HTTPResponse
.The request may be either a string URL or an
HTTPRequest
object. If it is a string, we construct anHTTPRequest
using any additional kwargs:HTTPRequest(request, **kwargs)
If an error occurs during the fetch, we raise an
HTTPError
unless theraise_error
keyword argument is set to False.
- class tornado.httpclient.AsyncHTTPClient(force_instance: bool = False, **kwargs: Any)[source]¶
An non-blocking HTTP client.
Example usage:
async def f(): http_client = AsyncHTTPClient() try: response = await http_client.fetch("http://www.google.com") except Exception as e: print("Error: %s" % e) else: print(response.body)
The constructor for this class is magic in several respects: It actually creates an instance of an implementation-specific subclass, and instances are reused as a kind of pseudo-singleton (one per
IOLoop
). The keyword argumentforce_instance=True
can be used to suppress this singleton behavior. Unlessforce_instance=True
is used, no arguments should be passed to theAsyncHTTPClient
constructor. The implementation subclass as well as arguments to its constructor can be set with the static methodconfigure()
All
AsyncHTTPClient
implementations support adefaults
keyword argument, which can be used to set default values forHTTPRequest
attributes. For example:AsyncHTTPClient.configure( None, defaults=dict(user_agent="MyUserAgent")) # or with force_instance: client = AsyncHTTPClient(force_instance=True, defaults=dict(user_agent="MyUserAgent"))
Changed in version 5.0: The
io_loop
argument (deprecated since version 4.1) has been removed.- close() None [source]¶
Destroys this HTTP client, freeing any file descriptors used.
This method is not needed in normal use due to the way that
AsyncHTTPClient
objects are transparently reused.close()
is generally only necessary when either theIOLoop
is also being closed, or theforce_instance=True
argument was used when creating theAsyncHTTPClient
.No other methods may be called on the
AsyncHTTPClient
afterclose()
.
- fetch(request: Union[str, HTTPRequest], raise_error: bool = True, **kwargs: Any) Future[HTTPResponse] [source]¶
Executes a request, asynchronously returning an
HTTPResponse
.The request may be either a string URL or an
HTTPRequest
object. If it is a string, we construct anHTTPRequest
using any additional kwargs:HTTPRequest(request, **kwargs)
This method returns a
Future
whose result is anHTTPResponse
. By default, theFuture
will raise anHTTPError
if the request returned a non-200 response code (other errors may also be raised if the server could not be contacted). Instead, ifraise_error
is set to False, the response will always be returned regardless of the response code.If a
callback
is given, it will be invoked with theHTTPResponse
. In the callback interface,HTTPError
is not automatically raised. Instead, you must check the response’serror
attribute or call itsrethrow
method.
- classmethod configure(impl: Union[None, str, Type[Configurable]], **kwargs: Any) None [source]¶
Configures the
AsyncHTTPClient
subclass to use.AsyncHTTPClient()
actually creates an instance of a subclass. This method may be called with either a class object or the fully-qualified name of such a class (orNone
to use the default,SimpleAsyncHTTPClient
)If additional keyword arguments are given, they will be passed to the constructor of each subclass instance created. The keyword argument
max_clients
determines the maximum number of simultaneousfetch()
operations that can execute in parallel on eachIOLoop
. Additional arguments may be supported depending on the implementation class in use.Example:
AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
Request objects¶
- class tornado.httpclient.HTTPRequest(url: str, method: str = 'GET', headers: Optional[Union[Dict[str, str], HTTPHeaders]] = None, body: Optional[Union[bytes, str]] = None, auth_username: Optional[str] = None, auth_password: Optional[str] = None, auth_mode: Optional[str] = None, connect_timeout: Optional[float] = None, request_timeout: Optional[float] = None, if_modified_since: Optional[Union[float, datetime]] = None, follow_redirects: Optional[bool] = None, max_redirects: Optional[int] = None, user_agent: Optional[str] = None, use_gzip: Optional[bool] = None, network_interface: Optional[str] = None, streaming_callback: Optional[Callable[[bytes], None]] = None, header_callback: Optional[Callable[[str], None]] = None, prepare_curl_callback: Optional[Callable[[Any], None]] = None, proxy_host: Optional[str] = None, proxy_port: Optional[int] = None, proxy_username: Optional[str] = None, proxy_password: Optional[str] = None, proxy_auth_mode: Optional[str] = None, allow_nonstandard_methods: Optional[bool] = None, validate_cert: Optional[bool] = None, ca_certs: Optional[str] = None, allow_ipv6: Optional[bool] = None, client_key: Optional[str] = None, client_cert: Optional[str] = None, body_producer: Optional[Callable[[Callable[[bytes], None]], Future[None]]] = None, expect_100_continue: bool = False, decompress_response: Optional[bool] = None, ssl_options: Optional[Union[Dict[str, Any], SSLContext]] = None)[source]¶
HTTP client request object.
All parameters except
url
are optional.- Parameters
url (str) – URL to fetch
method (str) – HTTP method, e.g. “GET” or “POST”
headers (
HTTPHeaders
ordict
) – Additional HTTP headers to pass on the requestbody (
str
orbytes
) – HTTP request body as a string (byte or unicode; if unicode the utf-8 encoding will be used)body_producer (collections.abc.Callable) – Callable used for lazy/asynchronous request bodies. It is called with one argument, a
write
function, and should return aFuture
. It should call the write function with new data as it becomes available. The write function returns aFuture
which can be used for flow control. Only one ofbody
andbody_producer
may be specified.body_producer
is not supported oncurl_httpclient
. When usingbody_producer
it is recommended to pass aContent-Length
in the headers as otherwise chunked encoding will be used, and many servers do not support chunked encoding on requests. New in Tornado 4.0auth_username (str) – Username for HTTP authentication
auth_password (str) – Password for HTTP authentication
auth_mode (str) – Authentication mode; default is “basic”. Allowed values are implementation-defined;
curl_httpclient
supports “basic” and “digest”;simple_httpclient
only supports “basic”connect_timeout (float) – Timeout for initial connection in seconds, default 20 seconds (0 means no timeout)
request_timeout (float) – Timeout for entire request in seconds, default 20 seconds (0 means no timeout)
if_modified_since (
datetime
orfloat
) – Timestamp forIf-Modified-Since
headerfollow_redirects (bool) – Should redirects be followed automatically or return the 3xx response? Default True.
max_redirects (int) – Limit for
follow_redirects
, default 5.user_agent (str) – String to send as
User-Agent
headerdecompress_response (bool) – Request a compressed response from the server and decompress it after downloading. Default is True. New in Tornado 4.0.
use_gzip (bool) – Deprecated alias for
decompress_response
since Tornado 4.0.network_interface (str) – Network interface or source IP to use for request. See
curl_httpclient
note below.streaming_callback (collections.abc.Callable) – If set,
streaming_callback
will be run with each chunk of data as it is received, andHTTPResponse.body
andHTTPResponse.buffer
will be empty in the final response.header_callback (collections.abc.Callable) – If set,
header_callback
will be run with each header line as it is received (including the first line, e.g.HTTP/1.0 200 OK\r\n
, and a final line containing only\r\n
. All lines include the trailing newline characters).HTTPResponse.headers
will be empty in the final response. This is most useful in conjunction withstreaming_callback
, because it’s the only way to get access to header data while the request is in progress.prepare_curl_callback (collections.abc.Callable) – If set, will be called with a
pycurl.Curl
object to allow the application to make additionalsetopt
calls.proxy_host (str) – HTTP proxy hostname. To use proxies,
proxy_host
andproxy_port
must be set;proxy_username
,proxy_pass
andproxy_auth_mode
are optional. Proxies are currently only supported withcurl_httpclient
.proxy_port (int) – HTTP proxy port
proxy_username (str) – HTTP proxy username
proxy_password (str) – HTTP proxy password
proxy_auth_mode (str) – HTTP proxy Authentication mode; default is “basic”. supports “basic” and “digest”
allow_nonstandard_methods (bool) – Allow unknown values for
method
argument? Default is False.validate_cert (bool) – For HTTPS requests, validate the server’s certificate? Default is True.
ca_certs (str) – filename of CA certificates in PEM format, or None to use defaults. See note below when used with
curl_httpclient
.client_key (str) – Filename for client SSL key, if any. See note below when used with
curl_httpclient
.client_cert (str) – Filename for client SSL certificate, if any. See note below when used with
curl_httpclient
.ssl_options (ssl.SSLContext) –
ssl.SSLContext
object for use insimple_httpclient
(unsupported bycurl_httpclient
). Overridesvalidate_cert
,ca_certs
,client_key
, andclient_cert
.allow_ipv6 (bool) – Use IPv6 when available? Default is True.
expect_100_continue (bool) – If true, send the
Expect: 100-continue
header and wait for a continue response before sending the request body. Only supported withsimple_httpclient
.
Note
When using
curl_httpclient
certain options may be inherited by subsequent fetches becausepycurl
does not allow them to be cleanly reset. This applies to theca_certs
,client_key
,client_cert
, andnetwork_interface
arguments. If you use these options, you should pass them on every request (you don’t have to always use the same values, but it’s not possible to mix requests that specify these options with ones that use the defaults).New in version 3.1: The
auth_mode
argument.New in version 4.0: The
body_producer
andexpect_100_continue
arguments.New in version 4.2: The
ssl_options
argument.New in version 4.5: The
proxy_auth_mode
argument.
Response objects¶
- class tornado.httpclient.HTTPResponse(request: HTTPRequest, code: int, headers: Optional[HTTPHeaders] = None, buffer: Optional[BytesIO] = None, effective_url: Optional[str] = None, error: Optional[BaseException] = None, request_time: Optional[float] = None, time_info: Optional[Dict[str, float]] = None, reason: Optional[str] = None, start_time: Optional[float] = None)[source]¶
HTTP Response object.
Attributes:
request
: HTTPRequest objectcode
: numeric HTTP status code, e.g. 200 or 404reason
: human-readable reason phrase describing the status codeheaders
:tornado.httputil.HTTPHeaders
objecteffective_url
: final location of the resource after following any redirectsbuffer
:cStringIO
object for response bodybody
: response body as bytes (created on demand fromself.buffer
)error
: Exception object, if anyrequest_time
: seconds from request start to finish. Includes all network operations from DNS resolution to receiving the last byte of data. Does not include time spent in the queue (due to themax_clients
option). If redirects were followed, only includes the final request.start_time
: Time at which the HTTP operation started, based ontime.time
(not the monotonic clock used byIOLoop.time
). May beNone
if the request timed out while in the queue.time_info
: dictionary of diagnostic timing information from the request. Available data are subject to change, but currently uses timings available from http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html, plusqueue
, which is the delay (if any) introduced by waiting for a slot underAsyncHTTPClient
’smax_clients
setting.
New in version 5.1: Added the
start_time
attribute.Changed in version 5.1: The
request_time
attribute previously included time spent in the queue forsimple_httpclient
, but not incurl_httpclient
. Now queueing time is excluded in both implementations.request_time
is now more accurate forcurl_httpclient
because it uses a monotonic clock when available.
Exceptions¶
- exception tornado.httpclient.HTTPClientError(code: int, message: Optional[str] = None, response: Optional[HTTPResponse] = None)[source]¶
Exception thrown for an unsuccessful HTTP request.
Attributes:
code
- HTTP error integer error code, e.g. 404. Error code 599 is used when no HTTP response was received, e.g. for a timeout.response
-HTTPResponse
object, if any.
Note that if
follow_redirects
is False, redirects become HTTPErrors, and you can look aterror.response.headers['Location']
to see the destination of the redirect.Changed in version 5.1: Renamed from
HTTPError
toHTTPClientError
to avoid collisions withtornado.web.HTTPError
. The nametornado.httpclient.HTTPError
remains as an alias.
- exception tornado.httpclient.HTTPError¶
Alias for
HTTPClientError
.
Command-line interface¶
This module provides a simple command-line interface to fetch a url using Tornado’s HTTP client. Example usage:
# Fetch the url and print its body
python -m tornado.httpclient http://www.google.com
# Just print the headers
python -m tornado.httpclient --print_headers --print_body=false http://www.google.com
Implementations¶
- class tornado.simple_httpclient.SimpleAsyncHTTPClient(force_instance: bool = False, **kwargs: Any)[source]¶
Non-blocking HTTP client with no external dependencies.
This class implements an HTTP 1.1 client on top of Tornado’s IOStreams. Some features found in the curl-based AsyncHTTPClient are not yet supported. In particular, proxies are not supported, connections are not reused, and callers cannot select the network interface to be used.
This implementation supports the following arguments, which can be passed to
configure()
to control the global singleton, or to the constructor whenforce_instance=True
.max_clients
is the number of concurrent requests that can be in progress; when this limit is reached additional requests will be queued. Note that time spent waiting in this queue still counts against therequest_timeout
.defaults
is a dict of parameters that will be used as defaults on allHTTPRequest
objects submitted to this client.hostname_mapping
is a dictionary mapping hostnames to IP addresses. It can be used to make local DNS changes when modifying system-wide settings like/etc/hosts
is not possible or desirable (e.g. in unittests).resolver
is similar, but using theResolver
interface instead of a simple mapping.max_buffer_size
(default 100MB) is the number of bytes that can be read into memory at once.max_body_size
(defaults tomax_buffer_size
) is the largest response body that the client will accept. Without astreaming_callback
, the smaller of these two limits applies; with astreaming_callback
onlymax_body_size
does.Changed in version 4.2: Added the
max_body_size
argument.
- class tornado.curl_httpclient.CurlAsyncHTTPClient(max_clients=10, defaults=None)¶
libcurl
-based HTTP client.This implementation supports the following arguments, which can be passed to
configure()
to control the global singleton, or to the constructor whenforce_instance=True
.max_clients
is the number of concurrent requests that can be in progress; when this limit is reached additional requests will be queued.defaults
is a dict of parameters that will be used as defaults on allHTTPRequest
objects submitted to this client.
Example Code¶
A simple webspider shows how to fetch URLs concurrently.
The file uploader demo uses either HTTP POST or HTTP PUT to upload files to a server.