Adds support for sysctl options in docker services.
* Adds API plumbing for creating services with sysctl options set.
* Adds swagger.yaml documentation for new API field.
* Updates the API version history document.
* Changes executor package to make use of the Sysctls field on objects
* Includes integration test to verify that new behavior works.
Essentially, everything needed to support the equivalent of docker run's
`--sysctl` option except the CLI.
Includes a vendoring of swarmkit for proto changes to support the new
behavior.
Signed-off-by: Drew Erny <drew.erny@docker.com>
This should eliminate a bunch of new (go-1.11 related) validation
errors telling that the code is not formatted with `gofmt -s`.
No functional change, just whitespace (i.e.
`git show --ignore-space-change` shows nothing).
Patch generated with:
> git ls-files | grep -v ^vendor/ | grep .go$ | xargs gofmt -s -w
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
When requesting information about the daemon's configuration through the `/info`
endpoint, missing features (or non-recommended settings) may have to be presented
to the user.
Detecting these situations, and printing warnings currently is handled by the
cli, which results in some complications:
- duplicated effort: each client has to re-implement detection and warnings.
- it's not possible to generate warnings for reasons outside of the information
returned in the `/info` response.
- cli-side detection has to be updated for new conditions. This means that an
older cli connecting to a new daemon may not print all warnings (due to
it not detecting the new conditions)
- some warnings (in particular, warnings about storage-drivers) depend on
driver-status (`DriverStatus`) information. The format of the information
returned in this field is not part of the API specification and can change
over time, resulting in cli-side detection no longer being functional.
This patch adds a new `Warnings` field to the `/info` response. This field is
to return warnings to be presented by the user.
Existing warnings that are currently handled by the CLI are copied to the daemon
as part of this patch; This change is backward-compatible with existing
clients; old client can continue to use the client-side warnings, whereas new
clients can skip client-side detection, and print warnings that are returned by
the daemon.
Example response with this patch applied;
```bash
curl --unix-socket /var/run/docker.sock http://localhost/info | jq .Warnings
```
```json
[
"WARNING: bridge-nf-call-iptables is disabled",
"WARNING: bridge-nf-call-ip6tables is disabled"
]
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This feature allows user to specify list of subnets for global
default address pool. User can configure subnet list using
'swarm init' command. Daemon passes the information to swarmkit.
We validate the information in swarmkit, then store it in cluster
object. when IPAM init is called, we pass subnet list to IPAM driver.
Signed-off-by: selansen <elango.siva@docker.com>
* Expose license status in Info
This wires up a new field in the Info payload that exposes the license.
For moby this is hardcoded to always report a community edition.
Downstream enterprise dockerd will have additional licensing logic wired
into this function to report details about the current license status.
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Code review comments
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Add windows autogen support
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
Should fix
```
api/types/volume/volume_create.go
Line 10: warning: comment on exported type VolumeCreateBody should be of the form "VolumeCreateBody ..." (with optional leading article) (golint)
api/types/volume/volume_list.go
Line 12: warning: comment on exported type VolumeListOKBody should be of the form "VolumeListOKBody ..." (with optional leading article) (golint)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This partially reverts https://github.com/moby/moby/pull/37350
Although specs.Platform is desirable in the API, there is more work
to be done on helper functions, namely containerd's platforms.Parse
that assumes the default platform of the Go runtime.
That prevents a client to use the recommended Parse function to
retrieve a specs.Platform object.
With this change, no parsing is expected from the client.
Signed-off-by: Tibor Vass <tibor@docker.com>
This stuff doesn't belong here and is causing imports of libnetwork into
the router, which is not what we want.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This adds MaskedPaths and ReadOnlyPaths options to HostConfig for containers so
that a user can override the default values.
When the value sent through the API is nil the default is used.
Otherwise the default is overridden.
Adds integration tests for MaskedPaths and ReadonlyPaths.
Signed-off-by: Jess Frazelle <acidburn@microsoft.com>
Adds functionality to parse and return network attachment spec
information. Network attachment tasks are phony tasks created in
swarmkit to deal with unmanaged containers attached to swarmkit. Before
this change, attempting `docker inspect` on the task id of a network
attachment task would result in an empty task object. After this change,
a full task object is returned
Fixes#26548 the correct way.
Signed-off-by: Drew Erny <drew.erny@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Arrays expect a type to be set for items in the array.
This patch adds the "string" type, adds a short description,
and some example values.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`GetTimestamp()` "assumed" values it could not parse
to be a valid unix timestamp, and would use invalid
values ("hello world") as-is (even testing that
it did so).
This patch validates unix timestamp to be a valid
numeric value, and makes other values invalid.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Since Go 1.7, context is a standard package. Since Go 1.9, everything
that is provided by "x/net/context" is a couple of type aliases to
types in "context".
Many vendored packages still use x/net/context, so vendor entry remains
for now.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
We are using interface in the api routers to not explicitely depend on
the daemon struct (`daemon.Daemon`), but somehow, we do depend on the
`daemon` package for the cluster functionalities.
This removes this dependency by defining the correct interfaces.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Docker daemon has a 16K buffer for log messages. If a message length
exceeds 16K, it should be split by the logger and merged at the
endpoint.
This change adds `PartialLogMetaData` struct for enhanced partial support
- LastPartial (bool) : indicates if this is the last of all partials.
- ID (string) : unique 32 bit ID. ID is same across all partials.
- Ordinal (int starts at 1) : indicates the position of msg in the series of partials.
Also, the timestamps across partials in the same.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
Makes sure if the user specifies an older API version that we don't pass
through templating options for versions that templating was not
supported.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The goal of this refactor is to make it easier to integrate buildkit
and containerd snapshotters.
Commit is used from two places (api and build), each calls it
with distinct arguments. Refactored to pull out the common commit
logic and provide different interfaces for each consumer.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This fix tries to address the issue raised in 36139 where
ExitCode and PID does not show up in Task.Status.ContainerStatus
The issue was caused by `json:",omitempty"` in PID and ExitCode
which interprate 0 as null.
This is confusion as ExitCode 0 does have a meaning.
This fix removes `json:",omitempty"` in ExitCode and PID,
but changes ContainerStatus to pointer so that ContainerStatus
does not show up at all if no content. If ContainerStatus
does have a content, then ExitCode and PID will show up (even if
they are 0).
This fix fixes 36139.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 36142 where
there are discrepancies between Swarm API and swagger.yaml.
This fix adds two recently added state `REMOVE` and `ORPHANED` to TaskState.
This fix fixes 36142.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 33661 where
network alias does not work when connect to a network the second time.
This fix address the issue.
This fix fixes 33661.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
The re-coalesces the daemon stores which were split as part of the
original LCOW implementation.
This is part of the work discussed in https://github.com/moby/moby/issues/34617,
in particular see the document linked to in that issue.
The `POST /volumes/create` expects a request body to be provided.
If no body was provided, a 500 status was returned. A 500 status
is incorrect, because the request is invalid (it's not a server
error).
Before this change:
$ curl --unix-socket /var/run/docker.sock -v -X POST http://localhost/volumes/create
* Trying /var/run/docker.sock...
* Connected to localhost (/Users/sebastiaan/Library/Containers/com.dock) port 80 (#0)
> POST /volumes/create HTTP/1.1
> Host: localhost
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.30
< Content-Length: 18
< Content-Type: application/json
< Date: Wed, 19 Jul 2017 11:29:26 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/17.06.0-ce (linux)
<
{"message":"EOF"}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
After this change:
$ curl --unix-socket /var/run/docker.sock -v -X POST http://localhost/volumes/create
* Trying /var/run/docker.sock...
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> POST /volumes/create HTTP/1.1
> Host: localhost
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Api-Version: 1.36
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Tue, 09 Jan 2018 15:00:13 GMT
< Content-Length: 42
<
{"message":"no body provided in request"}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix is a follow up to 30397, with `FindUniqueNetwork`
changed to `FindNetwork` based on the review feedback.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Some improvements to the test;
- Combine tests to reduce duplicated code
- Add test-cases for empty version in request using the default version
- Add test for valid versions in request actually setting the version
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of having to create a bunch of custom error types that are doing
nothing but wrapping another error in sub-packages, use a common helper
to create errors of the requested type.
e.g. instead of re-implementing this over and over:
```go
type notFoundError struct {
cause error
}
func(e notFoundError) Error() string {
return e.cause.Error()
}
func(e notFoundError) NotFound() {}
func(e notFoundError) Cause() error {
return e.cause
}
```
Packages can instead just do:
```
errdefs.NotFound(err)
```
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Actually the specification was expecting a 'additionalProperties' for the Volumes data, where in fact it's expecting
a map of string pointing to empty object.
Signed-off-by: Joel Wurtz <joel.wurtz@gmail.com>
This fix is part of the effort to address 30242 where
issue arise because of the fact that multiple networks
may share the same name (within or across local/swarm scopes).
The focus of this fix is to allow creation of service
when a network in local scope has the same name as the
service network.
An integration test has been added.
This fix fixes 30242.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
The building machinery was being handed an uninitialized container
Config. This changes it to use the target container's Config.
Resolves#30538
Signed-off-by: Anthony Sottile <asottile@umich.edu>
This change adds a Platform struct with a Name field and a general
Components field to the Version API type. This will allow API
consumers to show version information for the whole platform and
it will allow API providers to set the versions for the various
components of the platform.
All changes here are backwards compatible.
Signed-off-by: Tibor Vass <tibor@docker.com>
When the client closes websocket connections that sends container
output through websocket, an error message is displayed:
"Error attaching websocket: %!s(<nil>)"
This message is misleading. Thus, this change suggests to check
if error is nil and print the correct message accordingly.
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
This route expects `application/json`. Sending a content type header of `application/octet-stream` results in an error.
Signed-off-by: Asad Saeeduddin <masaeedu@gmail.com>
Commit 3ba1dda191 bumped
the API version, but forgot to actually bump the version
in code.
This patch fixes the version to match those changes :-)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Swagger file contained a version matrix to
find which API version is used by which version
of Docker.
Given that Docker is a downstream of the Moby project,
we should not be maintaining such a matrix in this
repository.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Currently, if a container removal has failed for some reason,
any client waiting for removal (e.g. `docker run --rm`) is
stuck, waiting for removal to succeed while it has failed already.
For more details and the reproducer, please check
https://github.com/moby/moby/issues/34945
This commit addresses that by allowing `ContainerWait()` with
`container.WaitCondition == "removed"` argument to return an
error in case of removal failure. The `ContainerWaitOKBody`
stucture returned to a client is amended with a pointer to `struct Error`,
containing an error message string, and the `Client.ContainerWait()`
is modified to return the error, if any, to the client.
Note that this feature is only available for API version >= 1.34.
In order for the old clients to be unstuck, we just close the connection
without writing anything -- this causes client's error.
Now, docker-cli would need a separate commit to bump the API to 1.34
and to show an error returned, if any.
[v2: recreate the waitRemove channel after closing]
[v3: document; keep legacy behavior for older clients]
[v4: convert Error from string to pointer to a struct]
[v5: don't emulate old behavior, send empty response in error case]
[v6: rename legacy* vars to include version suffix]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Describe more how host port allocation is done when
container is stopped/started in "PublishAllPorts".
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
If a 400 error is returned due to an API version mismatch, no
version and server-identification headers were returned by the API.
All information in these headers is "static", so there is no
reason to omit the information in case of an error being
returned.
This patch updates the version middleware to always
return the headers.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit e98e4a7111 implemented API version
negotiation using the `/_ping` endpoint. In that change, URL validation for the
maximum supported API version was removed from the API server (validation for
the _minimum_ version was kept in place).
With this feature, clients that support version negotiation would negotiate the
maximum version supported by the daemon, and downgrade to an older API version
if the client's default API version is not supported.
However, clients that do _not_ support version negotiation can call API versions
that are higher than the maximum supported version. Due to the missing version
check, this is silently ignored, and the daemon's default API version is used.
This is a problem, because the actual API version in use is non-deterministic;
for example, calling `/v9999.9999/version` on a daemon that runs API v1.34 will
use API v1.34, but calling the same URL on an older daemon may use API version
v1.24.
This patch reverts the removal of the API check for maximum supported versions.
The documentation has been updated accordingly
Before this patch is applied, the daemon returns a 200 (success):
$ curl -v --unix-socket /var/run/docker.sock http://localhost/v9999.9999/version
* Trying /var/run/docker.sock...
* Connected to localhost (/Users/sebastiaan/Library/Containers/com.dock) port 80 (#0)
> GET /v9999.9999/version HTTP/1.1
> Host: localhost
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Api-Version: 1.32
< Content-Length: 240
< Content-Type: application/json
< Date: Tue, 10 Oct 2017 09:11:29 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/17.09.0-ce (linux)
<
{"Version":"17.09.0-ce","ApiVersion":"1.32","MinAPIVersion":"1.12","GitCommit":"afdb6d4","GoVersion":"go1.8.3","Os":"linux","Arch":"amd64","KernelVersion":"4.9.49-moby","Experimental":true,"BuildTime":"2017-09-26T22:45:38.000000000+00:00"}
* Connection #0 to host localhost left intact
After this patch is applied, a 400 (Bad Request) is returned:
$ curl -v --unix-socket /var/run/docker.sock http://localhost/v9999.9999/version
* Trying /var/run/docker.sock...
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> GET /v9999.9999/info HTTP/1.1
> Host: localhost
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Date: Tue, 10 Oct 2017 08:08:34 GMT
< Content-Length: 89
<
{"message":"client version 9999.9999 is too new. Maximim supported API version is 1.34"}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: John Howard <jhoward@microsoft.com>
This PR has the API changes described in https://github.com/moby/moby/issues/34617.
Specifically, it adds an HTTP header "X-Requested-Platform" which is a JSON-encoded
OCI Image-spec `Platform` structure.
In addition, it renames (almost all) uses of a string variable platform (and associated)
methods/functions to os. This makes it much clearer to disambiguate with the swarm
"platform" which is really os/arch. This is a stepping stone to getting the daemon towards
fully multi-platform/arch-aware, and makes it clear when "operating system" is being
referred to rather than "platform" which is misleadingly used - sometimes in the swarm
meaning, but more often as just the operating system.
`filters.ToParam()` and `filters.FromParam()` were deprecated in favor of
`filters.ToJSON()` and `filters.FromJSON()` in 065118390a,
but still used in various locations.
This patch replaces uses of `filters.ToParam()` and `filters.FromParam()` with
`filters.ToJSON()` and `filters.FromJSON()`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `filters.Include()` method was deprecated in favor of `filters.Contains()`
in 065118390a, but still used in various
locations.
This patch replaces uses of `filters.Include()` with `filters.Contains()`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This enables docker cp and ADD/COPY docker build support for LCOW.
Originally, the graphdriver.Get() interface returned a local path
to the container root filesystem. This does not work for LCOW, so
the Get() method now returns an interface that LCOW implements to
support copying to and from the container.
Signed-off-by: Akash Gupta <akagup@microsoft.com>
The `--enable-api-cors` flag was deprecated in f3dd2db4ff,
and marked for removal in docker 17.09 through 85f92ef359.
This patch removes the deprecated flag.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `/info` endpoint was badly documented, missing various
fields and incorrectly describing others.
This patch defines a type for the endpoint, based on the
API types in the source.
Also removing the response example in favor of
per-field examples, as this prevents an incorrectly
formatted response from masking omissions in the
actual type.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `Node.ManagerStatus` property was only present in
the example, but not in the definition.
This patch adds definitions for `ManagerStatus`
and `Reachability`, similar to what is used in the
code;
f02a5b50c4/api/types/swarm/node.go (L84-L101)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- `TLSInfo` is part of `Node.Description`, but was documented as a
direct child of `Node`
- `Node.TLSInfo` incorrectly was using the `SwarmSpec` type,
instead of `TLSInfo`
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- `ObjectVersion.Index` is an `uint64` 0fd90c4d5d/api/types/swarm/common.go (L5-L15)
- `ClusterInfo` is nullable in the `/info` output (see ff4f700f74/api/types/swarm/swarm.go (L203))
- `CAConfig.ForceRotate` was missing a type, therefore treated as an `object` in Swagger: ff4f700f74/api/types/swarm/swarm.go (L121)
- `Raft.SnapshotInterval`, `Raft.KeepOldSnapshots`, and `Raft.LogEntriesForSlowFollowers` are an `uint64` not an `int64`
- Various fields in `swarm.Info` are nullable; added `x-nullable`
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use strongly typed errors to set HTTP status codes.
Error interfaces are defined in the api/errors package and errors
returned from controllers are checked against these interfaces.
Errors can be wraeped in a pkg/errors.Causer, as long as somewhere in the
line of causes one of the interfaces is implemented. The special error
interfaces take precedence over Causer, meaning if both Causer and one
of the new error interfaces are implemented, the Causer is not
traversed.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This patch updates the definition of `NetworkConfig` to match the code, and
renames to the definition to `NetworkSettings` (also to match the type in
the code).
Add definitions for:
- `Address`
- `PortMap`
- `PortBinding`
- `EndpointIPAMConfig`
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Since the commit d88fe447df ("Add support for sharing /dev/shm/ and
/dev/mqueue between containers") container's /dev/shm is mounted on the
host first, then bind-mounted inside the container. This is done that
way in order to be able to share this container's IPC namespace
(and the /dev/shm mount point) with another container.
Unfortunately, this functionality breaks container checkpoint/restore
(even if IPC is not shared). Since /dev/shm is an external mount, its
contents is not saved by `criu checkpoint`, and so upon restore any
application that tries to access data under /dev/shm is severily
disappointed (which usually results in a fatal crash).
This commit solves the issue by introducing new IPC modes for containers
(in addition to 'host' and 'container:ID'). The new modes are:
- 'shareable': enables sharing this container's IPC with others
(this used to be the implicit default);
- 'private': disables sharing this container's IPC.
In 'private' mode, container's /dev/shm is truly mounted inside the
container, without any bind-mounting from the host, which solves the
issue.
While at it, let's also implement 'none' mode. The motivation, as
eloquently put by Justin Cormack, is:
> I wondered a while back about having a none shm mode, as currently it is
> not possible to have a totally unwriteable container as there is always
> a /dev/shm writeable mount. It is a bit of a niche case (and clearly
> should never be allowed to be daemon default) but it would be trivial to
> add now so maybe we should...
...so here's yet yet another mode:
- 'none': no /dev/shm mount inside the container (though it still
has its own private IPC namespace).
Now, to ultimately solve the abovementioned checkpoint/restore issue, we'd
need to make 'private' the default mode, but unfortunately it breaks the
backward compatibility. So, let's make the default container IPC mode
per-daemon configurable (with the built-in default set to 'shareable'
for now). The default can be changed either via a daemon CLI option
(--default-shm-mode) or a daemon.json configuration file parameter
of the same name.
Note one can only set either 'shareable' or 'private' IPC modes as a
daemon default (i.e. in this context 'host', 'container', or 'none'
do not make much sense).
Some other changes this patch introduces are:
1. A mount for /dev/shm is added to default OCI Linux spec.
2. IpcMode.Valid() is simplified to remove duplicated code that parsed
'container:ID' form. Note the old version used to check that ID does
not contain a semicolon -- this is no longer the case (tests are
modified accordingly). The motivation is we should either do a
proper check for container ID validity, or don't check it at all
(since it is checked in other places anyway). I chose the latter.
3. IpcMode.Container() is modified to not return container ID if the
mode value does not start with "container:", unifying the check to
be the same as in IpcMode.IsContainer().
3. IPC mode unit tests (runconfig/hostconfig_test.go) are modified
to add checks for newly added values.
[v2: addressed review at https://github.com/moby/moby/pull/34087#pullrequestreview-51345997]
[v3: addressed review at https://github.com/moby/moby/pull/34087#pullrequestreview-53902833]
[v4: addressed the case of upgrading from older daemon, in this case
container.HostConfig.IpcMode is unset and this is valid]
[v5: document old and new IpcMode values in api/swagger.yaml]
[v6: add the 'none' mode, changelog entry to docs/api/version-history.md]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Current insider builds of Windows have support for mounting individual
named pipe servers from the host to the guest. This allows, for example,
exposing the docker engine's named pipe to a container.
This change allows the user to request such a mount via the normal bind
mount syntax in the CLI:
docker run -v \\.\pipe\docker_engine:\\.\pipe\docker_engine <args>
Signed-off-by: John Starks <jostarks@microsoft.com>
- When a network is created with the null ipam driver, docker api server
thread will deference a nil pointer on `docker network ls` and on
`docker network inspect <nw>`. This because buildIpamResource()
assumes a gateway address is always present, which is not correct.
Signed-off-by: Alessandro Boch <aboch@tetrationanalytics.com>
