30916df3d3
full diff: https://github.com/containerd/ttrpc/compare/v1.1.1...v1.2.2 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
12 KiB
Protocol Specification
The ttrpc protocol is client/server protocol to support multiple request streams over a single connection with lightweight framing. The client represents the process which initiated the underlying connection and the server is the process which accepted the connection. The protocol is currently defined as asymmetrical, with clients sending requests and servers sending responses. Both clients and servers are able to send stream data. The roles are also used in determining the stream identifiers, with client initiated streams using odd number identifiers and server initiated using even number. The protocol may be extended in the future to support server initiated streams, that is not supported in the latest version.
Purpose
The ttrpc protocol is designed to be lightweight and optimized for low latency and reliable connections between processes on the same host. The protocol does not include features for handling unreliable connections such as handshakes, resets, pings, or flow control. The protocol is designed to make low-overhead implementations as simple as possible. It is not intended as a suitable replacement for HTTP2/3 over the network.
Message Frame
Each Message Frame consists of a 10-byte message header followed by message data. The data length and stream ID are both big-endian 4-byte unsigned integers. The message type is an unsigned 1-byte integer. The flags are also an unsigned 1-byte integer and use is defined by the message type.
+---------------------------------------------------------------+
| Data Length (32) |
+---------------------------------------------------------------+
| Stream ID (32) |
+---------------+-----------------------------------------------+
| Msg Type (8) |
+---------------+
| Flags (8) |
+---------------+-----------------------------------------------+
| Data (*) |
+---------------------------------------------------------------+
The Data Length field represents the number of bytes in the Data field. The total frame size will always be Data Length + 10 bytes. The maximum data length is 4MB and any larger size should be rejected. Due to the maximum data size being less than 16MB, the first frame byte should always be zero. This first byte should be considered reserved for future use.
The Stream ID must be odd for client initiated streams and even for server initiated streams. Server initiated streams are not currently supported.
Mesage Types
| Message Type | Name | Description |
|---|---|---|
| 0x01 | Request | Initiates stream |
| 0x02 | Response | Final stream data and terminates |
| 0x03 | Data | Stream data |
Request
The request message is used to initiate stream and send along request data for
properly routing and handling the stream. The stream may indicate unary without
any inbound or outbound stream data with only a response is expected on the
stream. The request may also indicate the stream is still open for more data and
no response is expected until data is finished. If the remote indicates the
stream is closed, the request may be considered non-unary but without anymore
stream data sent. In the case of remote closed, the remote still expects to
receive a response or stream data. For compatibility with non streaming clients,
a request with empty flags indicates a unary request.
Request Flags
| Flag | Name | Description |
|---|---|---|
| 0x01 | remote closed |
Non-unary, but no more data expected from remote |
| 0x02 | remote open |
Non-unary, remote is still sending data |
Response
The response message is used to end a stream with data, an empty response, or an error. A response message is the only expected message after a unary request. A non-unary request does not require a response message if the server is sending back stream data. A non-unary stream may return a single response message but no other stream data may follow.
Response Flags
No response flags are defined at this time, flags should be empty.
Data
The data message is used to send data on an already initialized stream. Either
client or server may send data. A data message is not allowed on a unary stream.
A data message should not be sent after indicating remote closed to the peer.
The last data message on a stream must set the remote closed flag.
The no data flag is used to indicate that the data message does not include
any data. This is normally used with the remote closed flag to indicate the
stream is now closed without transmitting any data. Since ttrpc normally
transmits a single object per message, a zero length data message may be
interpreted as an empty object. For example, transmitting the number zero as a
protobuf message ends up with a data length of zero, but the message is still
considered data and should be processed.
Data Flags
| Flag | Name | Description |
|---|---|---|
| 0x01 | remote closed |
No more data expected from remote |
| 0x04 | no data |
This message does not have data |
Streaming
All ttrpc requests use streams to transfer data. Unary streams will only have
two messages sent per stream, a request from a client and a response from the
server. Non-unary streams, however, may send any numbers of messages from the
client and the server. This makes stream management more complicated than unary
streams since both client and server need to track additional state. To keep
this management as simple as possible, ttrpc minimizes the number of states and
uses two flags instead of control frames. Each stream has two states while a
stream is still alive: local closed and remote closed. Each peer considers
local and remote from their own perspective and sets flags from the other peer's
perspective. For example, if a client sends a data frame with the
remote closed flag, that is indicating that the client is now local closed
and the server will be remote closed. A unary operation does not need to send
these flags since each received message always indicates remote closed. Once a
peer is both local closed and remote closed, the stream is considered
finished and may be cleaned up.
Due to the asymmetric nature of the current protocol, a client should
always be in the local closed state before remote closed and a server should
always be in the remote closed state before local closed. This happens
because the client is always initiating requests and a client always expects a
final response back from a server to indicate the initiated request has been
fulfilled. This may mean server sends a final empty response to finish a stream
even after it has already completed sending data before the client.
Unary State Diagram
+--------+ +--------+
| Client | | Server |
+---+----+ +----+---+
| +---------+ |
local >---------------+ Request +--------------------> remote
closed | +---------+ | closed
| |
| +----------+ |
finished <--------------+ Response +--------------------< finished
| +----------+ |
| |
Non-Unary State Diagrams
RC: remote closed flag
RO: remote open flag
+--------+ +--------+
| Client | | Server |
+---+----+ +----+---+
| +--------------+ |
>-------------+ Request [RO] +----------------->
| +--------------+ |
| |
| +------+ |
>-----------------+ Data +--------------------->
| +------+ |
| |
| +-----------+ |
local >---------------+ Data [RC] +------------------> remote
closed | +-----------+ | closed
| |
| +----------+ |
finished <--------------+ Response +--------------------< finished
| +----------+ |
| |
+--------+ +--------+
| Client | | Server |
+---+----+ +----+---+
| +--------------+ |
local >-------------+ Request [RC] +-----------------> remote
closed | +--------------+ | closed
| |
| +------+ |
<-----------------+ Data +---------------------<
| +------+ |
| |
| +-----------+ |
finished <---------------+ Data [RC] +------------------< finished
| +-----------+ |
| |
+--------+ +--------+
| Client | | Server |
+---+----+ +----+---+
| +--------------+ |
>-------------+ Request [RO] +----------------->
| +--------------+ |
| |
| +------+ |
>-----------------+ Data +--------------------->
| +------+ |
| |
| +------+ |
<-----------------+ Data +---------------------<
| +------+ |
| |
| +------+ |
>-----------------+ Data +--------------------->
| +------+ |
| |
| +-----------+ |
local >---------------+ Data [RC] +------------------> remote
closed | +-----------+ | closed
| |
| +------+ |
<-----------------+ Data +---------------------<
| +------+ |
| |
| +-----------+ |
finished <---------------+ Data [RC] +------------------< finished
| +-----------+ |
| |
RPC
While this protocol is defined primarily to support Remote Procedure Calls, the protocol does not define the request and response types beyond the messages defined in the protocol. The implementation provides a default protobuf definition of request and response which may be used for cross language rpc. All implementations should at least define a request type which support routing by procedure name and a response type which supports call status.
Version History
| Version | Features |
|---|---|
| 1.0 | Unary requests only |
| 1.2 | Streaming support |