moby/libnetwork/docs/remote.md

Remote Drivers

The drivers.remote package provides the integration point for dynamically-registered drivers. Unlike the other driver packages, it does not provide a single implementation of a driver; rather, it provides a proxy for remote driver processes, which are registered and communicate with LibNetwork via the Docker plugin package.

For the semantics of driver methods, which correspond to the protocol below, please see the overall design.

LibNetwork integration with the Docker plugins package

When LibNetwork initializes the drivers.remote package with the Init() function, it passes a DriverCallback as a parameter, which implements RegisterDriver(). The remote driver package uses this interface to register remote drivers with LibNetwork's NetworkController, by supplying it in a plugins.Handle callback.

The callback is invoked when a driver is loaded with the plugins.Get API call. How that comes about is out of scope here (but it might be, for instance, when that driver is mentioned by the user).

This design ensures that the details of driver registration mechanism are owned by the remote driver package, and it doesn't expose any of the driver layer to the North of LibNetwork.

Implementation

The remote driver implementation uses a plugins.Client to communicate with the remote driver process. The driverapi.Driver methods are implemented as RPCs over the plugin client.

The payloads of these RPCs are mostly direct translations into JSON of the arguments given to the method. There are some exceptions to account for the use of the interfaces InterfaceInfo and JoinInfo, and data types that do not serialise to JSON well (e.g., net.IPNet). The protocol is detailed below under "Protocol".

Usage

A remote driver proxy follows all the rules of any other in-built driver and has exactly the same Driver interface exposed. LibNetwork will also support driver-specific options and user-supplied labels which may influence the behaviour of a remote driver process.

Protocol

The remote driver protocol is a set of RPCs, issued as HTTP POSTs with JSON payloads. The proxy issues requests, and the remote driver process is expected to respond usually with a JSON payload of its own, although in some cases these are empty maps.

Errors

If the remote process cannot decode, or otherwise detects a syntactic problem with the HTTP request or payload, it must respond with an HTTP error status (4xx or 5xx).

If the remote process http server receives a request for an unknown URI, it should respond with the HTTP StatusCode 404 Not Found. This allows LibNetwork to detect when a remote driver does not implement yet a newly added method, therefore not to deem the request as failed.

If the remote process can decode the request, but cannot complete the operation, it must send a response in the form

{
	"Err": string
}

The string value supplied may appear in logs, so should not include confidential information.

Handshake

When loaded, a remote driver process receives an HTTP POST on the URL /Plugin.Activate with no payload. It must respond with a manifest of the form

{
	"Implements": ["NetworkDriver"]
}

Other entries in the list value are allowed; "NetworkDriver" indicates that the plugin should be registered with LibNetwork as a driver.

Set capability

After Handshake, the remote driver will receive another POST message to the URL /NetworkDriver.GetCapabilities with no payload. The driver's response should have the form:

{
	"Scope":             "local"
	"ConnectivityScope": "global"
}

Value of "Scope" should be either "local" or "global" which indicates whether the resource allocations for this driver's network can be done only locally to the node or globally across the cluster of nodes. Any other value will fail driver's registration and return an error to the caller. Similarly, value of "ConnectivityScope" should be either "local" or "global" which indicates whether the driver's network can provide connectivity only locally to this node or globally across the cluster of nodes. If the value is missing, libnetwork will set it to the value of "Scope". should be either "local" or "global" which indicates

Create network

When the proxy is asked to create a network, the remote process shall receive a POST to the URL /NetworkDriver.CreateNetwork of the form

{
	"NetworkID": string,
	"IPv4Data" : [
	{
		"AddressSpace": string,
		"Pool": ipv4-cidr-string,
		"Gateway" : ipv4-cidr-string,
		"AuxAddresses": {
			"<identifier1>" : "<ipv4-address1>",
			"<identifier2>" : "<ipv4-address2>",
			...
		}
	},
	],
	"IPv6Data" : [
	{
		"AddressSpace": string,
		"Pool": ipv6-cidr-string,
		"Gateway" : ipv6-cidr-string,
		"AuxAddresses": {
			"<identifier1>" : "<ipv6-address1>",
			"<identifier2>" : "<ipv6-address2>",
			...
		}
	},
	],
	"Options": {
		...
	}
}

• NetworkID value is generated by LibNetwork which represents a unique network.
• Options value is the arbitrary map given to the proxy by LibNetwork.
• IPv4Data and IPv6Data are the ip-addressing data configured by the user and managed by IPAM driver. The network driver is expected to honor the ip-addressing data supplied by IPAM driver. The data include,
• AddressSpace : A unique string represents an isolated space for IP Addressing
• Pool : A range of IP Addresses represented in CIDR format address/mask. Since, the IPAM driver is responsible for allocating container ip-addresses, the network driver can make use of this information for the network plumbing purposes.
• Gateway : Optionally, the IPAM driver may provide a Gateway IP address in CIDR format for the subnet represented by the Pool. The network driver can make use of this information for the network plumbing purposes.
• AuxAddresses : A list of pre-allocated ip-addresses with an associated identifier as provided by the user to assist network driver if it requires specific ip-addresses for its operation.

The response indicating success is empty:

{}

Delete network

When a network owned by the remote driver is deleted, the remote process shall receive a POST to the URL /NetworkDriver.DeleteNetwork of the form

{
	"NetworkID": string
}

The success response is empty:

{}

Create endpoint

When the proxy is asked to create an endpoint, the remote process shall receive a POST to the URL /NetworkDriver.CreateEndpoint of the form

{
	"NetworkID": string,
	"EndpointID": string,
	"Options": {
		...
	},
	"Interface": {
		"Address": string,
		"AddressIPv6": string,
		"MacAddress": string
	}
}

The NetworkID is the generated identifier for the network to which the endpoint belongs; the EndpointID is a generated identifier for the endpoint.

Options is an arbitrary map as supplied to the proxy.

The Interface value is of the form given. The fields in the Interface may be empty; and the Interface itself may be empty. If supplied, Address is an IPv4 address and subnet in CIDR notation; e.g., "192.168.34.12/16". If supplied, AddressIPv6 is an IPv6 address and subnet in CIDR notation. MacAddress is a MAC address as a string; e.g., "6e:75:32:60:44:c9".

A success response is of the form

{
	"Interface": {
		"Address": string,
		"AddressIPv6": string,
		"MacAddress": string
	}
}

with values in the Interface as above. As far as the value of Interface is concerned, MacAddress and either or both of Address and AddressIPv6 must be given.

If the remote process was supplied a non-empty value in Interface, it must respond with an empty Interface value. LibNetwork will treat it as an error if it supplies a non-empty value and receives a non-empty value back, and roll back the operation.

Endpoint operational info

The proxy may be asked for "operational info" on an endpoint. When this happens, the remote process shall receive a POST to /NetworkDriver.EndpointOperInfo of the form

{
	"NetworkID": string,
	"EndpointID": string
}

where NetworkID and EndpointID have meanings as above. It must send a response of the form

{
	"Value": { ... }
}

where the value of the Value field is an arbitrary (possibly empty) map.

Delete endpoint

When an endpoint is deleted, the remote process shall receive a POST to the URL /NetworkDriver.DeleteEndpoint with a body of the form

{
	"NetworkID": string,
	"EndpointID": string
}

where NetworkID and EndpointID have meanings as above. A success response is empty:

{}

Join

When a sandbox is given an endpoint, the remote process shall receive a POST to the URL NetworkDriver.Join of the form

{
	"NetworkID": string,
	"EndpointID": string,
	"SandboxKey": string,
	"Options": { ... }
}

The NetworkID and EndpointID have meanings as above. The SandboxKey identifies the sandbox. Options is an arbitrary map as supplied to the proxy.

The response must have the form

{
	"InterfaceName": {
		SrcName: string,
		DstPrefix: string
	},
	"Gateway": string,
	"GatewayIPv6": string,
	"StaticRoutes": [{
		"Destination": string,
		"RouteType": int,
		"NextHop": string,
	}, ...]
}

Gateway is optional and if supplied is an IP address as a string; e.g., "192.168.0.1". GatewayIPv6 is optional and if supplied is an IPv6 address as a string; e.g., "fe80::7809:baff:fec6:7744".

The entries in InterfaceName represent actual OS level interfaces that should be moved by LibNetwork into the sandbox; the SrcName is the name of the OS level interface that the remote process created, and the DstPrefix is a prefix for the name the OS level interface should have after it has been moved into the sandbox (LibNetwork will append an index to make sure the actual name does not collide with others).

The entries in "StaticRoutes" represent routes that should be added to an interface once it has been moved into the sandbox. Since there may be zero or more routes for an interface, unlike the interface name they can be supplied in any order.

Routes are either given a RouteType of 0 and a value for NextHop; or, a RouteType of 1 and no value for NextHop, meaning a connected route.

If no gateway and no default static route is set by the driver in the Join response, LibNetwork will add an additional interface to the sandbox connecting to a default gateway network (a bridge network named docker_gwbridge) and program the default gateway into the sandbox accordingly, pointing to the interface address of the bridge docker_gwbridge.

Leave

If the proxy is asked to remove an endpoint from a sandbox, the remote process shall receive a POST to the URL /NetworkDriver.Leave of the form

{
	"NetworkID": string,
	"EndpointID": string
}

where NetworkID and EndpointID have meanings as above. The success response is empty:

{}

DiscoverNew Notification

LibNetwork listens to inbuilt docker discovery notifications and passes it along to the interested drivers.

When the proxy receives a DiscoverNew notification, the remote process shall receive a POST to the URL /NetworkDriver.DiscoverNew of the form

{
	"DiscoveryType": int,
	"DiscoveryData": {
		...
	}
}

DiscoveryType represents the discovery type. Each Discovery Type is represented by a number. DiscoveryData carries discovery data the structure of which is determined by the DiscoveryType

The response indicating success is empty:

{}

• Node Discovery

Node Discovery is represented by a DiscoveryType value of 1 and the corresponding DiscoveryData will carry Node discovery data.

{
	"DiscoveryType": int,
	"DiscoveryData": {
                "Address" : string
                "self" : bool
	}
}

DiscoverDelete Notification

When the proxy receives a DiscoverDelete notification, the remote process shall receive a POST to the URL /NetworkDriver.DiscoverDelete of the form

{
	"DiscoveryType": int,
	"DiscoveryData": {
		...
	}
}

DiscoveryType represents the discovery type. Each Discovery Type is represented by a number. DiscoveryData carries discovery data the structure of which is determined by the DiscoveryType

The response indicating success is empty:

{}

• Node Discovery

Similar to the DiscoverNew call, Node Discovery is represented by a DiscoveryType value of 1 and the corresponding DiscoveryData will carry Node discovery data to be deleted.

{
	"DiscoveryType": int,
	"DiscoveryData": {
                "Address" : string
                "self" : bool
	}
}