tornado.iostream — Convenient wrappers for non-blocking sockets

Utility classes to write to and read from non-blocking files and sockets.

Contents:

• BaseIOStream: Generic interface for reading and writing.

• IOStream: Implementation of BaseIOStream using non-blocking sockets.

• SSLIOStream: SSL-aware version of IOStream.

• PipeIOStream: Pipe-based IOStream implementation.

Base class

class tornado.iostream.BaseIOStream(max_buffer_size: Optional[int] = None, read_chunk_size: Optional[int] = None, max_write_buffer_size: Optional[int] = None)

A utility class to write to and read from a non-blocking file or socket.

We support a non-blocking write() and a family of read_*() methods. When the operation completes, the Awaitable will resolve with the data read (or None for write()). All outstanding Awaitables will resolve with a StreamClosedError when the stream is closed; BaseIOStream.set_close_callback can also be used to be notified of a closed stream.

When a stream is closed due to an error, the IOStream’s error attribute contains the exception object.

Subclasses must implement fileno, close_fd, write_to_fd, read_from_fd, and optionally get_fd_error.

BaseIOStream constructor.

Parameters

• max_buffer_size – Maximum amount of incoming data to buffer; defaults to 100MB.

• read_chunk_size – Amount of data to read at one time from the underlying transport; defaults to 64KB.

• max_write_buffer_size – Amount of outgoing data to buffer; defaults to unlimited.

Changed in version 4.0: Add the max_write_buffer_size parameter. Changed default read_chunk_size to 64KB.

Changed in version 5.0: The io_loop argument (deprecated since version 4.1) has been removed.

Main interface

BaseIOStream.write(data: Union[bytes, memoryview]) → Future[None]

Asynchronously write the given data to this stream.

This method returns a Future that resolves (with a result of None) when the write has been completed.

The data argument may be of type bytes or memoryview.

Changed in version 4.0: Now returns a Future if no callback is given.

Changed in version 4.5: Added support for memoryview arguments.

Changed in version 6.0: The callback argument was removed. Use the returned Future instead.

BaseIOStream.read_bytes(num_bytes: int, partial: bool = False) → Awaitable[bytes]

Asynchronously read a number of bytes.

If partial is true, data is returned as soon as we have any bytes to return (but never more than num_bytes)

Changed in version 4.0: Added the partial argument. The callback argument is now optional and a Future will be returned if it is omitted.

Changed in version 6.0: The callback and streaming_callback arguments have been removed. Use the returned Future (and partial=True for streaming_callback) instead.

BaseIOStream.read_into(buf: bytearray, partial: bool = False) → Awaitable[int]

Asynchronously read a number of bytes.

buf must be a writable buffer into which data will be read.

If partial is true, the callback is run as soon as any bytes have been read. Otherwise, it is run when the buf has been entirely filled with read data.

New in version 5.0.

Changed in version 6.0: The callback argument was removed. Use the returned Future instead.

BaseIOStream.read_until(delimiter: bytes, max_bytes: Optional[int] = None) → Awaitable[bytes]

Asynchronously read until we have found the given delimiter.

The result includes all the data read including the delimiter.

If max_bytes is not None, the connection will be closed if more than max_bytes bytes have been read and the delimiter is not found.

Changed in version 4.0: Added the max_bytes argument. The callback argument is now optional and a Future will be returned if it is omitted.

Changed in version 6.0: The callback argument was removed. Use the returned Future instead.

BaseIOStream.read_until_regex(regex: bytes, max_bytes: Optional[int] = None) → Awaitable[bytes]

Asynchronously read until we have matched the given regex.

The result includes the data that matches the regex and anything that came before it.

If max_bytes is not None, the connection will be closed if more than max_bytes bytes have been read and the regex is not satisfied.

Changed in version 4.0: Added the max_bytes argument. The callback argument is now optional and a Future will be returned if it is omitted.

Changed in version 6.0: The callback argument was removed. Use the returned Future instead.

BaseIOStream.read_until_close() → Awaitable[bytes]

Asynchronously reads all data from the socket until it is closed.

This will buffer all available data until max_buffer_size is reached. If flow control or cancellation are desired, use a loop with read_bytes(partial=True) instead.

Changed in version 4.0: The callback argument is now optional and a Future will be returned if it is omitted.

Changed in version 6.0: The callback and streaming_callback arguments have been removed. Use the returned Future (and read_bytes with partial=True for streaming_callback) instead.

BaseIOStream.close(exc_info: Union[None, bool, BaseException, Tuple[Optional[Type[BaseException]], Optional[BaseException], Optional[TracebackType]]] = False) → None

Close this stream.

If exc_info is true, set the error attribute to the current exception from sys.exc_info (or if exc_info is a tuple, use that instead of sys.exc_info).

BaseIOStream.set_close_callback(callback: Optional[Callable[[], None]]) → None

Call the given callback when the stream is closed.

This mostly is not necessary for applications that use the Future interface; all outstanding Futures will resolve with a StreamClosedError when the stream is closed. However, it is still useful as a way to signal that the stream has been closed while no other read or write is in progress.

Unlike other callback-based interfaces, set_close_callback was not removed in Tornado 6.0.

BaseIOStream.closed() → bool

Returns True if the stream has been closed.

BaseIOStream.reading() → bool

Returns True if we are currently reading from the stream.

BaseIOStream.writing() → bool

Returns True if we are currently writing to the stream.

BaseIOStream.set_nodelay(value: bool) → None

Sets the no-delay flag for this stream.

By default, data written to TCP streams may be held for a time to make the most efficient use of bandwidth (according to Nagle’s algorithm). The no-delay flag requests that data be written as soon as possible, even if doing so would consume additional bandwidth.

This flag is currently defined only for TCP-based IOStreams.

New in version 3.1.

Methods for subclasses

BaseIOStream.fileno() → Union[int, _Selectable]

Returns the file descriptor for this stream.

BaseIOStream.close_fd() → None

Closes the file underlying this stream.

close_fd is called by BaseIOStream and should not be called elsewhere; other users should call close instead.

BaseIOStream.write_to_fd(data: memoryview) → int

Attempts to write data to the underlying file.

Returns the number of bytes written.

BaseIOStream.read_from_fd(buf: Union[bytearray, memoryview]) → Optional[int]

Attempts to read from the underlying file.

Reads up to len(buf) bytes, storing them in the buffer. Returns the number of bytes read. Returns None if there was nothing to read (the socket returned EWOULDBLOCK or equivalent), and zero on EOF.

Changed in version 5.0: Interface redesigned to take a buffer and return a number of bytes instead of a freshly-allocated object.

BaseIOStream.get_fd_error() → Optional[Exception]

Returns information about any error on the underlying file.

This method is called after the IOLoop has signaled an error on the file descriptor, and should return an Exception (such as socket.error with additional information, or None if no such information is available.

Implementations

class tornado.iostream.IOStream(socket: socket, *args: Any, **kwargs: Any)

Socket-based IOStream implementation.

This class supports the read and write methods from BaseIOStream plus a connect method.

The socket parameter may either be connected or unconnected. For server operations the socket is the result of calling socket.accept. For client operations the socket is created with socket.socket, and may either be connected before passing it to the IOStream or connected with IOStream.connect.

A very simple (and broken) HTTP client using this class:

import tornado.ioloop
import tornado.iostream
import socket

async def main():
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
    stream = tornado.iostream.IOStream(s)
    await stream.connect(("friendfeed.com", 80))
    await stream.write(b"GET / HTTP/1.0\r\nHost: friendfeed.com\r\n\r\n")
    header_data = await stream.read_until(b"\r\n\r\n")
    headers = {}
    for line in header_data.split(b"\r\n"):
        parts = line.split(b":")
        if len(parts) == 2:
            headers[parts[0].strip()] = parts[1].strip()
    body_data = await stream.read_bytes(int(headers[b"Content-Length"]))
    print(body_data)
    stream.close()

if __name__ == '__main__':
    asyncio.run(main())

connect(address: Any, server_hostname: Optional[str] = None) → Future[_IOStreamType]

Connects the socket to a remote address without blocking.

May only be called if the socket passed to the constructor was not previously connected. The address parameter is in the same format as for socket.connect for the type of socket passed to the IOStream constructor, e.g. an (ip, port) tuple. Hostnames are accepted here, but will be resolved synchronously and block the IOLoop. If you have a hostname instead of an IP address, the TCPClient class is recommended instead of calling this method directly. TCPClient will do asynchronous DNS resolution and handle both IPv4 and IPv6.

If callback is specified, it will be called with no arguments when the connection is completed; if not this method returns a Future (whose result after a successful connection will be the stream itself).

In SSL mode, the server_hostname parameter will be used for certificate validation (unless disabled in the ssl_options) and SNI (if supported; requires Python 2.7.9+).

Note that it is safe to call IOStream.write while the connection is pending, in which case the data will be written as soon as the connection is ready. Calling IOStream read methods before the socket is connected works on some platforms but is non-portable.

Changed in version 4.0: If no callback is given, returns a Future.

Changed in version 4.2: SSL certificates are validated by default; pass ssl_options=dict(cert_reqs=ssl.CERT_NONE) or a suitably-configured ssl.SSLContext to the SSLIOStream constructor to disable.

Changed in version 6.0: The callback argument was removed. Use the returned Future instead.

start_tls(server_side: bool, ssl_options: Optional[Union[Dict[str, Any], SSLContext]] = None, server_hostname: Optional[str] = None) → Awaitable[SSLIOStream]

Convert this IOStream to an SSLIOStream.

This enables protocols that begin in clear-text mode and switch to SSL after some initial negotiation (such as the STARTTLS extension to SMTP and IMAP).

This method cannot be used if there are outstanding reads or writes on the stream, or if there is any data in the IOStream’s buffer (data in the operating system’s socket buffer is allowed). This means it must generally be used immediately after reading or writing the last clear-text data. It can also be used immediately after connecting, before any reads or writes.

The ssl_options argument may be either an ssl.SSLContext object or a dictionary of keyword arguments for the ssl.wrap_socket function. The server_hostname argument will be used for certificate validation unless disabled in the ssl_options.

This method returns a Future whose result is the new SSLIOStream. After this method has been called, any other operation on the original stream is undefined.

If a close callback is defined on this stream, it will be transferred to the new stream.

New in version 4.0.

Changed in version 4.2: SSL certificates are validated by default; pass ssl_options=dict(cert_reqs=ssl.CERT_NONE) or a suitably-configured ssl.SSLContext to disable.

class tornado.iostream.SSLIOStream(*args: Any, **kwargs: Any)

A utility class to write to and read from a non-blocking SSL socket.

If the socket passed to the constructor is already connected, it should be wrapped with:

ssl.wrap_socket(sock, do_handshake_on_connect=False, **kwargs)

before constructing the SSLIOStream. Unconnected sockets will be wrapped when IOStream.connect is finished.

The ssl_options keyword argument may either be an ssl.SSLContext object or a dictionary of keywords arguments for ssl.wrap_socket

wait_for_handshake() → Future[SSLIOStream]

Wait for the initial SSL handshake to complete.

If a callback is given, it will be called with no arguments once the handshake is complete; otherwise this method returns a Future which will resolve to the stream itself after the handshake is complete.

Once the handshake is complete, information such as the peer’s certificate and NPN/ALPN selections may be accessed on self.socket.

This method is intended for use on server-side streams or after using IOStream.start_tls; it should not be used with IOStream.connect (which already waits for the handshake to complete). It may only be called once per stream.

New in version 4.2.

Changed in version 6.0: The callback argument was removed. Use the returned Future instead.

class tornado.iostream.PipeIOStream(fd: int, *args: Any, **kwargs: Any)

Pipe-based IOStream implementation.

The constructor takes an integer file descriptor (such as one returned by os.pipe) rather than an open file object. Pipes are generally one-way, so a PipeIOStream can be used for reading or writing but not both.

PipeIOStream is only available on Unix-based platforms.

Exceptions

exception tornado.iostream.StreamBufferFullError

Exception raised by IOStream methods when the buffer is full.

exception tornado.iostream.StreamClosedError(real_error: Optional[BaseException] = None)

Exception raised by IOStream methods when the stream is closed.

Note that the close callback is scheduled to run after other callbacks on the stream (to allow for buffered data to be processed), so you may see this error before you see the close callback.

The real_error attribute contains the underlying error that caused the stream to close (if any).

Changed in version 4.3: Added the real_error attribute.

exception tornado.iostream.UnsatisfiableReadError

Exception raised when a read cannot be satisfied.

Raised by read_until and read_until_regex with a max_bytes argument.