tornado.httpclient — Asynchronous HTTP client¶
Blocking and non-blocking HTTP client interfaces.
This module defines a common interface shared by two implementations,
curl_httpclient. Applications may either
instantiate their chosen implementation class directly or use the
AsyncHTTPClient class from this module, which selects an implementation
that can be overridden with the
The default implementation is
simple_httpclient, and this is expected
to be suitable for most users’ needs. However, some applications may wish
to switch to
curl_httpclient for reasons such as the following:
curl_httpclienthas some features not found in
simple_httpclient, including support for HTTP proxies and the ability to use a specified network interface.
curl_httpclientis more likely to be compatible with sites that are not-quite-compliant with the HTTP spec, or sites that use little-exercised features of HTTP.
Note that if you are using
curl_httpclient, it is highly
recommended that you use a recent version of
pycurl. Currently the minimum supported version of libcurl is
7.22.0, and the minimum version of pycurl is 7.18.2. It is highly
recommended that your
libcurl installation is built with
asynchronous DNS resolver (threaded or c-ares), otherwise you may
encounter various problems with request timeouts (for more
and comments in curl_httpclient.py).
AsyncHTTPClient.configure at startup:
HTTP client interfaces¶
HTTPClient(async_client_class: Type[AsyncHTTPClient] = None, **kwargs)¶
A blocking HTTP client.
Typical usage looks like this:
http_client = httpclient.HTTPClient() try: response = http_client.fetch("http://www.google.com/") print(response.body) except httpclient.HTTPError as e: # HTTPError is raised for non-200 responses; the response # can be found in e.response. print("Error: " + str(e)) except Exception as e: # Other errors are possible, such as IOError. print("Error: " + str(e)) http_client.close()
close() → None¶
Closes the HTTPClient, freeing any resources used.
fetch(request: Union[HTTPRequest, str], **kwargs) → tornado.httpclient.HTTPResponse¶
Executes a request, returning an
If an error occurs during the fetch, we raise an
raise_errorkeyword argument is set to False.
An non-blocking HTTP client.
async def f(): http_client = AsyncHTTPClient() try: response = await http_client.fetch("http://www.google.com") except Exception as e: print("Error: %s" % e) else: print(response.body)
The constructor for this class is magic in several respects: It actually creates an instance of an implementation-specific subclass, and instances are reused as a kind of pseudo-singleton (one per
IOLoop). The keyword argument
force_instance=Truecan be used to suppress this singleton behavior. Unless
force_instance=Trueis used, no arguments should be passed to the
AsyncHTTPClientconstructor. The implementation subclass as well as arguments to its constructor can be set with the static method
AsyncHTTPClient.configure( None, defaults=dict(user_agent="MyUserAgent")) # or with force_instance: client = AsyncHTTPClient(force_instance=True, defaults=dict(user_agent="MyUserAgent"))
Changed in version 5.0: The
io_loopargument (deprecated since version 4.1) has been removed.
close() → None¶
Destroys this HTTP client, freeing any file descriptors used.
This method is not needed in normal use due to the way that
AsyncHTTPClientobjects are transparently reused.
close()is generally only necessary when either the
IOLoopis also being closed, or the
force_instance=Trueargument was used when creating the
No other methods may be called on the
fetch(request: Union[str, HTTPRequest], raise_error: bool = True, **kwargs) → Awaitable[tornado.httpclient.HTTPResponse]¶
Executes a request, asynchronously returning an
This method returns a
Futurewhose result is an
HTTPResponse. By default, the
Futurewill raise an
HTTPErrorif the request returned a non-200 response code (other errors may also be raised if the server could not be contacted). Instead, if
raise_erroris set to False, the response will always be returned regardless of the response code.
callbackis given, it will be invoked with the
HTTPResponse. In the callback interface,
HTTPErroris not automatically raised. Instead, you must check the response’s
errorattribute or call its
configure(impl: Union[None, str, Type[tornado.util.Configurable]], **kwargs) → None¶
AsyncHTTPClientsubclass to use.
AsyncHTTPClient()actually creates an instance of a subclass. This method may be called with either a class object or the fully-qualified name of such a class (or
Noneto use the default,
If additional keyword arguments are given, they will be passed to the constructor of each subclass instance created. The keyword argument
max_clientsdetermines the maximum number of simultaneous
fetch()operations that can execute in parallel on each
IOLoop. Additional arguments may be supported depending on the implementation class in use.
HTTPRequest(url: str, method: str = 'GET', headers: Union[Dict[str, str], tornado.httputil.HTTPHeaders] = None, body: Union[bytes, str] = None, auth_username: str = None, auth_password: str = None, auth_mode: str = None, connect_timeout: float = None, request_timeout: float = None, if_modified_since: Union[float, datetime.datetime] = None, follow_redirects: bool = None, max_redirects: int = None, user_agent: str = None, use_gzip: bool = None, network_interface: str = None, streaming_callback: Callable[[bytes], None] = None, header_callback: Callable[[str], None] = None, prepare_curl_callback: Callable[[Any], None] = None, proxy_host: str = None, proxy_port: int = None, proxy_username: str = None, proxy_password: str = None, proxy_auth_mode: str = None, allow_nonstandard_methods: bool = None, validate_cert: bool = None, ca_certs: str = None, allow_ipv6: bool = None, client_key: str = None, client_cert: str = None, body_producer: Callable[[Callable[[bytes], None]], Future[None]] = None, expect_100_continue: bool = False, decompress_response: bool = None, ssl_options: Union[Dict[str, Any], ssl.SSLContext] = None)¶
HTTP client request object.
All parameters except
- url (str) – URL to fetch
- method (str) – HTTP method, e.g. “GET” or “POST”
- headers (
dict) – Additional HTTP headers to pass on the request
- body – HTTP request body as a string (byte or unicode; if unicode the utf-8 encoding will be used)
- body_producer – Callable used for lazy/asynchronous request bodies.
It is called with one argument, a
writefunction, and should return a
Future. It should call the write function with new data as it becomes available. The write function returns a
Futurewhich can be used for flow control. Only one of
body_producermay be specified.
body_produceris not supported on
curl_httpclient. When using
body_producerit is recommended to pass a
Content-Lengthin the headers as otherwise chunked encoding will be used, and many servers do not support chunked encoding on requests. New in Tornado 4.0
- auth_username (str) – Username for HTTP authentication
- auth_password (str) – Password for HTTP authentication
- auth_mode (str) – Authentication mode; default is “basic”.
Allowed values are implementation-defined;
curl_httpclientsupports “basic” and “digest”;
simple_httpclientonly supports “basic”
- connect_timeout (float) – Timeout for initial connection in seconds, default 20 seconds
- request_timeout (float) – Timeout for entire request in seconds, default 20 seconds
- if_modified_since (
float) – Timestamp for
- follow_redirects (bool) – Should redirects be followed automatically or return the 3xx response? Default True.
- max_redirects (int) – Limit for
follow_redirects, default 5.
- user_agent (str) – String to send as
- decompress_response (bool) – Request a compressed response from the server and decompress it after downloading. Default is True. New in Tornado 4.0.
- use_gzip (bool) – Deprecated alias for
decompress_responsesince Tornado 4.0.
- network_interface (str) – Network interface or source IP to use for request.
- streaming_callback (collections.abc.Callable) – If set,
streaming_callbackwill be run with each chunk of data as it is received, and
HTTPResponse.bufferwill be empty in the final response.
- header_callback (collections.abc.Callable) – If set,
header_callbackwill be run with each header line as it is received (including the first line, e.g.
HTTP/1.0 200 OK\r\n, and a final line containing only
\r\n. All lines include the trailing newline characters).
HTTPResponse.headerswill be empty in the final response. This is most useful in conjunction with
streaming_callback, because it’s the only way to get access to header data while the request is in progress.
- prepare_curl_callback (collections.abc.Callable) – If set, will be called with
pycurl.Curlobject to allow the application to make additional
- proxy_host (str) – HTTP proxy hostname. To use proxies,
proxy_portmust be set;
proxy_auth_modeare optional. Proxies are currently only supported with
- proxy_port (int) – HTTP proxy port
- proxy_username (str) – HTTP proxy username
- proxy_password (str) – HTTP proxy password
- proxy_auth_mode (str) – HTTP proxy Authentication mode; default is “basic”. supports “basic” and “digest”
- allow_nonstandard_methods (bool) – Allow unknown values for
methodargument? Default is False.
- validate_cert (bool) – For HTTPS requests, validate the server’s certificate? Default is True.
- ca_certs (str) – filename of CA certificates in PEM format,
or None to use defaults. See note below when used with
- client_key (str) – Filename for client SSL key, if any. See
note below when used with
- client_cert (str) – Filename for client SSL certificate, if any.
See note below when used with
- ssl_options (ssl.SSLContext) –
ssl.SSLContextobject for use in
- allow_ipv6 (bool) – Use IPv6 when available? Default is True.
- expect_100_continue (bool) – If true, send the
Expect: 100-continueheader and wait for a continue response before sending the request body. Only supported with
curl_httpclientcertain options may be inherited by subsequent fetches because
pycurldoes not allow them to be cleanly reset. This applies to the
network_interfacearguments. If you use these options, you should pass them on every request (you don’t have to always use the same values, but it’s not possible to mix requests that specify these options with ones that use the defaults).
New in version 3.1: The
New in version 4.0: The
New in version 4.2: The
New in version 4.5: The
HTTPResponse(request: tornado.httpclient.HTTPRequest, code: int, headers: tornado.httputil.HTTPHeaders = None, buffer: _io.BytesIO = None, effective_url: str = None, error: BaseException = None, request_time: float = None, time_info: Dict[str, float] = None, reason: str = None, start_time: float = None)¶
HTTP Response object.
request: HTTPRequest object
code: numeric HTTP status code, e.g. 200 or 404
reason: human-readable reason phrase describing the status code
effective_url: final location of the resource after following any redirects
cStringIOobject for response body
body: response body as bytes (created on demand from
error: Exception object, if any
request_time: seconds from request start to finish. Includes all network operations from DNS resolution to receiving the last byte of data. Does not include time spent in the queue (due to the
max_clientsoption). If redirects were followed, only includes the final request.
start_time: Time at which the HTTP operation started, based on
time.time(not the monotonic clock used by
IOLoop.time). May be
Noneif the request timed out while in the queue.
time_info: dictionary of diagnostic timing information from the request. Available data are subject to change, but currently uses timings available from http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html, plus
queue, which is the delay (if any) introduced by waiting for a slot under
New in version 5.1: Added the
Changed in version 5.1: The
request_timeattribute previously included time spent in the queue for
simple_httpclient, but not in
curl_httpclient. Now queueing time is excluded in both implementations.
request_timeis now more accurate for
curl_httpclientbecause it uses a monotonic clock when available.
HTTPClientError(code: int, message: str = None, response: tornado.httpclient.HTTPResponse = None)¶
Exception thrown for an unsuccessful HTTP request.
code- HTTP error integer error code, e.g. 404. Error code 599 is used when no HTTP response was received, e.g. for a timeout.
HTTPResponseobject, if any.
Note that if
follow_redirectsis False, redirects become HTTPErrors, and you can look at
error.response.headers['Location']to see the destination of the redirect.
Changed in version 5.1: Renamed from
HTTPClientErrorto avoid collisions with
tornado.web.HTTPError. The name
tornado.httpclient.HTTPErrorremains as an alias.
This module provides a simple command-line interface to fetch a url using Tornado’s HTTP client. Example usage:
# Fetch the url and print its body python -m tornado.httpclient http://www.google.com # Just print the headers python -m tornado.httpclient --print_headers --print_body=false http://www.google.com
Non-blocking HTTP client with no external dependencies.
This class implements an HTTP 1.1 client on top of Tornado’s IOStreams. Some features found in the curl-based AsyncHTTPClient are not yet supported. In particular, proxies are not supported, connections are not reused, and callers cannot select the network interface to be used.
initialize(max_clients: int = 10, hostname_mapping: Dict[str, str] = None, max_buffer_size: int = 104857600, resolver: tornado.netutil.Resolver = None, defaults: Dict[str, Any] = None, max_header_size: int = None, max_body_size: int = None) → None¶
Creates a AsyncHTTPClient.
Only a single AsyncHTTPClient instance exists per IOLoop in order to provide limitations on the number of pending connections.
force_instance=Truemay be used to suppress this behavior.
Note that because of this implicit reuse, unless
force_instanceis used, only the first call to the constructor actually uses its arguments. It is recommended to use the
configuremethod instead of the constructor to ensure that arguments take effect.
max_clientsis the number of concurrent requests that can be in progress; when this limit is reached additional requests will be queued. Note that time spent waiting in this queue still counts against the
hostname_mappingis a dictionary mapping hostnames to IP addresses. It can be used to make local DNS changes when modifying system-wide settings like
/etc/hostsis not possible or desirable (e.g. in unittests).
max_buffer_size(default 100MB) is the number of bytes that can be read into memory at once.
max_buffer_size) is the largest response body that the client will accept. Without a
streaming_callback, the smaller of these two limits applies; with a
Changed in version 4.2: Added the
libcurl-based HTTP client.