This changes the long-standing bug of copy operations not preserving the
UID/GID information after the files arrive to the container.
Signed-off-by: Erik Hollensbe <github@hollensbe.org>
This adds a new parameter insertDefaults to /services/{id}. When this is
set, an empty field (such as UpdateConfig) will be populated with
default values in the API response. Make "service inspect" use this, so
that empty fields do not result in missing information when inspecting a
service.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Service logs API is now stable. Service logs now support all features,
except retrieving details provided to the log driver.
Signed-off-by: Drew Erny <drew.erny@docker.com>
Logging plugins use the same HTTP interface as other plugins for basic
command operations meanwhile actual logging operations are handled (on
Unix) via a fifo.
The plugin interface looks like so:
```go
type loggingPlugin interface {
StartLogging(fifoPath string, loggingContext Context) error
StopLogging(fifoPath)
```
This means a plugin must implement `LoggingDriver.StartLogging` and
`LoggingDriver.StopLogging` endpoints and be able to consume the passed
in fifo.
Logs are sent via stream encoder to the fifo encoded with protobuf.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
- Defined "normalized" type for Credential Spec and SELinux
- Added --credential-spec to docker service create & update
- SELinux is API only at the time
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
Refactor container logs system to make communicating log messages
internally much simpler. Move responsibility for marshalling log
messages into the REST server. Support TTY logs. Pave the way for fixing
the ambiguous bytestream format. Pave the way for fixing details.
Signed-off-by: Drew Erny <drew.erny@docker.com>
This parameter controls the order of operations when rolling out an
update task. Either the old task is stopped before starting the new one,
or the new task is started first, and the running tasks will briefly
overlap.
This commit adds Rollout to the API, and --update-order / --rollback-order
flags to the CLI.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This fix tries to address the issue raised in 29999 where it was not
possible to mask these items (like important non-removable stuff)
from `docker system prune`.
This fix adds `label` and `label!` field for `--filter` in `system prune`,
so that it is possible to selectively prune items like:
```
$ docker container prune --filter label=foo
$ docker container prune --filter label!=bar
```
Additional unit tests and integration tests have been added.
This fix fixes 29999.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Refactored the API to more easily accept new endpoints. Added REST,
client, and CLI endpoints for getting logs from a specific task. All
that is needed after this commit to enable arbitrary service log
selectors is a REST endpoint and handler.
Task logs can be retrieved by putting in a task ID at the CLI instead of
a service ID.
Signed-off-by: Drew Erny <drew.erny@docker.com>
Currently these fields are included in the response JSON with zero
values. It's better not to include them if the information is
unavailable (for example, on a worker node).
This turns Cluster into a pointer so that it can be left out.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This fix tries to address the request in 31324 by adding
`--filter scope=swarm|local` for `docker network ls`.
As `docker network ls` has a `SCOPE` column by default,
it is natural to add the support of `--filter scope=swarm|local`.
This fix adds the `scope=swarm|local` support for
`docker network ls --filter`.
Related docs has been updated.
Additional unit test cases have been added.
This fix fixes 31324.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Embedding DockerVersion in plugin config when the plugin is created,
enables users to do a docker plugin inspect and know which version
the plugin was built on. This is helpful in cases where users are
running a new plugin on older docker releases and confused at
unexpected behavior.
By embedding DockerVersion in the config, we claim that there's no
guarantee that if the plugin config's DockerVersion is greater that
the version of the docker engine the plugin is executed against, the
plugin will work as expected.
For example, lets say:
- in 17.03, a plugin was released as johndoe/foo:v1
- in 17.05, the plugin uses the new ipchost config setting and author
publishes johndoe/foo:v2
In this case, johndoe/foo:v2 was built on 17.05 using ipchost, but is
running on docker-engine version 17.03. Since 17.05 > 17.03, there's
no guarantee that the plugin will work as expected. Ofcourse, if the
plugin did not use newly added config settings (ipchost in this case)
in 17.05, it would work fine in 17.03.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
Tested using global-net-plugin-ipc which sets PidHost in config.json.
Plugins might need access to host pid namespace. Add support for that.
Tested using aragunathan/global-net-plugin-ipc which sets "pidhost" in
config.json. Observed using `readlink /proc/self/ns/pid` that plugin and
host have the same ns.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
Plugins might need access to host ipc namespace. A good usecase is
a volume plugin running iscsi multipath commands that need access to
host kernel locks.
Tested with a custom plugin (aragunathan/global-net-plugin-full) that's
built with `"ipchost" : true` in config.json. Observed using
`readlink /proc/self/ns/ipc` that plugin and host have the same ns.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
This type is only used by CLI code. It duplicates SecretReference in the
types/swarm package. Change the CLI code to use that type instead.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This more (in spirit) mimics the handler usage in net/http/pprof.
It also makes sure that any new profiles that are added are
automatically supported (e.g. `mutex` profiles in go1.8).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fix tries to address the request in 31325 by adding
`--filter mode=global|replicated` to `docker service ls`.
As `docker service ls` has a `MODE` column by default, it is natural
to support `--filter mode=global|replicated` for `docker service ls`.
There are multiple ways to address the issue. One way is to pass
the filter of mode to SwarmKit, another way is to process the filter
of mode in the daemon.
This fix process the filter in the daemon.
Related docs has been updated.
An integration test has been added.
This fix fixes 31325.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
In https://github.com/torvalds/linux/commit/5ca3726 (released in v4.7-rc1) the
content of the `cpuacct.usage_percpu` file in sysfs was changed to include both
online and offline cpus. This broke the arithmetic in the stats helpers used by
`docker stats`, since it was using the length of the PerCPUUsage array as a
proxy for the number of online CPUs.
Add current number of online CPUs to types.StatsJSON and use it in the
calculation.
Keep a fallback to `len(v.CPUStats.CPUUsage.PercpuUsage)` so this code
continues to work when talking to an older daemon. An old client talking to a
new daemon will ignore the new field and behave as before.
Fixes#28941.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Running the rm command on a paused/restarting container
will give an error message saying the container is running
which is incorrect.
To fix that, the error message will have the correct
container state and a procedure to remove it accordingly.
Notice: docker-py was bumped to:
4a08d04aef0595322e1b5ac7c52f28a931da85a5
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
Server-side rollback can take advantage of the rollback-specific update
parameters, instead of being treated as a normal update that happens to
go back to a previous version of the spec.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This adds 'consistency' mode flags to the mount command line argument.
Initially, the valid 'consistency' flags are 'consistent', 'cached',
'delegated', and 'default'.
Signed-off-by: David Sheets <dsheets@docker.com>
Signed-off-by: Jeremy Yallop <yallop@docker.com>
This adds support for placement preferences in Swarm services.
- Convert PlacementPreferences between GRPC API and HTTP API
- Add --placement-pref, --placement-pref-add and --placement-pref-rm to CLI
- Add support for placement preferences in service inspect --pretty
- Add integration test
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This fix is an attempt to improve the documentation of
`Version` usage for Swarm API in Swagger docs.
As is shown in the docs updates, `Version` field in Swarm
API is a way to ensure safe concurrency and determinism
so that multiple updates to the same swarm related objects
(Node, Service, etc) will not unintentially overwrite each other.
This fix fixes 29448.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 25696 where
it was not possible to specify `--stop-signal` for `docker service create`
and `docker service update`, in order to use special signal to stop
the container.
This fix adds `--stop-signal` and update the `StopSignal` in `Config`
through `service create` and `service update`.
Related docs has been updated.
Integration test has been added.
This fix fixes 25696.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue in 31179 where
secrets fields is missing for `docker service create`
or `docker service update`.
This fix adds Secrets for `ContainerSpec` in Swagger docs.
This fix fixes 31179.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
PR #31197 made me think I needed to add more text explaing why
BuildArgs needs to be a *string instead of string.
Signed-off-by: Doug Davis <dug@us.ibm.com>
This fix adds the missing fields of `Hostname`, `OpenStdin`,
`Hosts`, `Groups`, and `Healthcheck` to `ContainerSpec`.
This fix also moves the `Label` in `ContainerSpec` to align with
the definition order.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Previously, `docker logs` (and by extension, `docker service logs`) had
an error in the way they returned errors that occurred after a stream
had already been started. Specifically, the errors were added verbatim
to the HTTP response, which caused StdCopy to fail with an esoteric
error. This change updates StdCopy to accept errors from the source
stream and then return those errors when copying to the destination
stream.
Signed-off-by: Drew Erny <drew.erny@docker.com>
This fix adds `Attachable` property for network object
in the documentation of Swagger API docs
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix is partially based on comment
https://github.com/docker/docker/issues/30242#issuecomment-273517205
Currently, `docker network inspect` relies on `FindNetwork()` which
does not take into consideration that multiple networks with the same
name might exist.
This fix propose to return `docker network inspect` in a similiar
fashion like other commands:
1. Lookup full ID
2. Lookup full name
3. Lookup partial ID
If multiple networks exist, an error will be returned.
NOTE: this fix is not a complete fix for the issue raised in
https://github.com/docker/docker/issues/30242#issuecomment-273517205
where SwarmKit is unable to update when multiple networks with the same
name exit.
To fix that issue requires multiple places when `FindNetwork()` is called.
Because of the impact of changing `FindNetwork()`, this fix focus on
the issue in `docker network inspect`.
A separate PR will be created to address
https://github.com/docker/docker/issues/30242#issuecomment-273517205
An integration test has been added.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Remove forked reference package. Use normalized named values
everywhere and familiar functions to convert back to familiar
strings for UX and storage compatibility.
Enforce that the source repository in the distribution metadata
is always a normalized string, ignore invalid values which are not.
Update distribution tests to use normalized values.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This allows a plugin to be upgraded without requiring to
uninstall/reinstall a plugin.
Since plugin resources (e.g. volumes) are tied to a plugin ID, this is
important to ensure resources aren't lost.
The plugin must be disabled while upgrading (errors out if enabled).
This does not add any convenience flags for automatically
disabling/re-enabling the plugin during before/after upgrade.
Since an upgrade may change requested permissions, the user is required
to accept permissions just like `docker plugin install`.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Pulling all the endpoints is a very resource heavy operation especially
for Global-scoped networks with a backing KVStore. Such heavy operations
can be fetched for individual network inspect. These are unneccessary
for a simple network list operation.
Signed-off-by: Madhu Venugopal <madhu@docker.com>
After running the test suite with the race detector enabled I found
these gems that need to be fixed.
This is just round one, sadly lost my test results after I built the
binary to test this... (whoops)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This allows the user to set a logging mode to "blocking" (default), or
"non-blocking", which uses the ring buffer as a proxy to the real log
driver.
This allows a container to never be blocked on stdio at the cost of
dropping log messages.
Introduces 2 new log-opts that works for all drivers, `log-mode` and
`log-size`. `log-mode` takes a value of "blocking", or "non-blocking"
I chose not to implement this as a bool since it is difficult to
determine if the mode was set to false vs just not set... especially
difficult when merging the default daemon config with the container config.
`log-size` takes a size string, e.g. `2MB`, which sets the max size
of the ring buffer. When the max size is reached, it will start
dropping log messages.
```
BenchmarkRingLoggerThroughputNoReceiver-8 2000000000 36.2 ns/op 856.35 MB/s 0 B/op 0 allocs/op
BenchmarkRingLoggerThroughputWithReceiverDelay0-8 300000000 156 ns/op 198.48 MB/s 32 B/op 0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay1-8 2000000000 36.1 ns/op 857.80 MB/s 0 B/op 0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay10-8 1000000000 36.2 ns/op 856.53 MB/s 0 B/op 0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay50-8 2000000000 34.7 ns/op 894.65 MB/s 0 B/op 0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay100-8 2000000000 35.1 ns/op 883.91 MB/s 0 B/op 0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay300-8 1000000000 35.9 ns/op 863.90 MB/s 0 B/op 0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay500-8 2000000000 35.8 ns/op 866.88 MB/s 0 B/op 0 allocs/op
```
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fix tries to address the issue raised in 28176 where
text frame was used in websocket attach endpoint. In case
the data send out contains non utf8 data, the connection
will be closed in certain browsers, e.g., Safari.
This fix address the issue by change `PayloadType` to `BinaryFrame`.
This fix is tested manually with Safari. The docker daemon is inside a Linux Virtual Machine.
Create a container with:
```
docker run -itd --name websocket busybox sh -c "while true; do echo -e 'he\\xc3\\x28o'; sleep 5; done"
```
Use the following url (172.16.66.128:2375 is the tcp address of the daemon):
```
file:///websocket.html?url=ws://172.16.66.128:2375/v1.25/containers/websocket/attach/ws?logs=1&stderr=1&stdout=1&stream=1&stdin=1
```
and the following html:
```
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Websocket</title>
<script type="text/javascript">
function DockerWebSocket() {
if ("WebSocket" in window) {
console.log("WebSocket is supported by Browser...")
// Remove '?url=' prefix
url = window.location.search.replace(/^(\?url=)/,"");
console.log("URL ["+url+"]...");
var ws = new WebSocket(url);
ws.onopen = function() {
console.log("Connection is opened...");
};
ws.onclose = function() {
console.log("Connection is closed...");
};
ws.onmessage = function (e) {
if (typeof e.data === "string") {
alert("WebSocket received text message ["+e.data+"]!")
} else {
console.log("Message is received...")
var blobReader = new FileReader();
blobReader.onload = function(event) {
console.log(JSON.stringify(blobReader.result))
};
blobReader.readAsText(e.data)
console.log("Message complete...")
}
};
} else {
alert("WebSocket is not supported by Browser!");
}
}
</script>
</head>
<body>
<div>
<a href="javascript:DockerWebSocket()">Run DockerWebSocket</a>
</div>
</body>
</html>
```
This fix fixes 28176.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 29972 where
it was not possible to specify `--read-only` for `docker service create`
and `docker service update`, in order to have the container's root file
system to be read only.
This fix adds `--read-only` and update the `ReadonlyRootfs` in `HostConfig`
through `service create` and `service update`.
Related docs has been updated.
Integration test has been added.
This fix fixes 29972.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix adds `--filter enabled=true` to `docker plugin ls`,
as was specified in 28624.
The related API and docs has been updated.
An integration test has been added.
This fix fixes 28624.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This introduce a new `--device-cgroup-rule` flag that allow a user to
add one or more entry to the container cgroup device `devices.allow`
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
This fix tries to address the issue raised in 30242 where the `Scope`
field always changed to `swarm` in the ouput of `docker network ls/inspect`
when duplicate networks name exist.
The reason for the issue was that `buildNetworkResource()` use network name
(which may not be unique) to check for the scope.
This fix fixes the issue by always use network ID in `buildNetworkResource()`.
A test has been added. The test fails before the fix and passes after the fix.
This fix fixes 30242.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Docker 1.13 moves the `--rm` flag to the daemon,
through an AutoRemove option in HostConfig.
When using API 1.24 and under, AutoRemove should not be
used, even if the daemon is version 1.13 or above and
"supports" this feature.
This patch fixes a situation where an 1.13 client,
talking to an 1.13 daemon, but using the 1.24 API
version, still set the AutoRemove property.
As a result, both the client _and_ the daemon
were attempting to remove the container, resulting
in an error:
ERRO[0000] error removing container: Error response from daemon:
removal of container ce0976ad22495c7cbe9487752ea32721a282164862db036b2f3377bd07461c3a
is already in progress
In addition, the validation of conflicting options
is moved from `docker run` to `opts.parse()`, so
that conflicting options are also detected when
running `docker create` and `docker start` separately.
To resolve the issue, the `AutoRemove` option is now
always set to `false` both by the client and the
daemon, if API version 1.24 or under is used.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This patch fixed below 4 types of code line
1. Remove unnecessary variable assignment
2. Use variables declaration instead of explicit initial zero value
3. Change variable name to underbar when variable not used
4. Add erro check and return for ignored error
Signed-off-by: Daehyeok Mun <daehyeok@gmail.com>
This fix tries to address the issue raised in 29946 where
listing networks from API will return `null` if the result of
network filter is empty.
The reason for the issue was that inside the `filterNetworks()`,
the return value was initialized as `nil`:
```
var typeNet []types.NetworkResource
```
This is inconsistent with other places where return value was
initialized with `[]`
```
displayNet := []types.NetworkResource{}
```
This fix addresses the issue by changing `typeNet` to `[]` as well.
This fix fixes 29946.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix adds a new flag `--availability` to `swarm join`.
Related documentation has been updated.
An integration test has been added.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 24596 where it was not
possible to join as manager only (`--availability=drain`).
This fix adds a new flag `--availability` to `swarm join`.
Related documentation has been updated.
An integration test has been added.
NOTE: Additional pull request for swarmkit and engine-api will
be created separately.
This fix fixes 24596.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
At the "Build image from Dockerfile" section in the API docs
the Content-Type header is missing.
In addition, some parts in the code are still setting the
Content-Type header to application/tar while it was changed
to application/x-tar since 16th September 2015.
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
The `digest` data type, used throughout docker for image verification
and identity, has been broken out into `opencontainers/go-digest`. This
PR updates the dependencies and moves uses over to the new type.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This fix is a follow up for comment
https://github.com/docker/docker/pull/28535#issuecomment-263215225
This fix provides `--filter until=<timestamp>` for `docker container/image prune`.
This fix adds `--filter until=<timestamp>` to `docker container/image prune`
so that it is possible to specify a timestamp and prune those containers/images
that are earlier than the timestamp.
Related docs has been updated
Several integration tests have been added to cover changes.
This fix fixes#28497.
This fix is related to #28535.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Signed-off-by: Ke Li <kel@splunk.com>
Add missing changes
Signed-off-by: Ke Li <kel@splunk.com>
User errors.New to create error
Signed-off-by: Ke Li <kel@splunk.com>
Move plugins to shared distribution stack with images.
Create immutable plugin config that matches schema2 requirements.
Ensure data being pushed is same as pulled/created.
Store distribution artifacts in a blobstore.
Run init layer setup for every plugin start.
Fix breakouts from unsafe file accesses.
Add support for `docker plugin install --alias`
Uses normalized references for default names to avoid collisions when using default hosts/tags.
Some refactoring of the plugin manager to support the change, like removing the singleton manager and adding manager config struct.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
When plugins have a positive refcount, they were not allowed to be
removed. However, plugins could still be disabled when volumes
referenced it and containers using them were running.
This change fixes that by enforcing plugin refcount during disable.
A "force" disable option is also added to ignore reference refcounting.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Previously, it doesn't allow creating such a network:
e.g.
$ docker network inspect -f '{{.Id}}' ingress
84xh9knigj6zyt00u31e26nj3
$ docker network create 84
Error response from daemon: network with name 84 already exists
Fix#27866
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Much easier than the previous method of copying over to the docs
repository and generating the docs.
And, as @cpuguy83 pointed out, that actually didn't work
because the PR that adds Swagger docs isn't merged yet. Oopsy.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
In #29071, we made the tags the correct name for generating types,
at the expense of the menu in the documentation looking good.
ReDoc now has support for tag display names
( https://github.com/Rebilly/ReDoc/pull/152 ), so we can assign
a more human-friendly name to the menu items.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
Legacy plugins expect host-relative paths (such as for Volume.Mount).
However, a containerized plugin cannot respond with a host-relative
path. Therefore, this commit modifies new volume plugins' paths in Mount
and List to prepend the container's rootfs path.
This introduces a new PropagatedMount field in the Plugin Config.
When it is set for volume plugins, RootfsPropagation is set to rshared
and the path specified by PropagatedMount is bind-mounted with rshared
prior to launching the container. This is so that the daemon code can
access the paths returned by the plugin from the host mount namespace.
Signed-off-by: Tibor Vass <tibor@docker.com>
This fix fixes a couple of incorrect version strings in swagger.yaml
`v1.25` should be `v1.26`. This applies to 1.14 version only.
Also checked 1.13rc and it is fine.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This reverts 26103. 26103 was trying to make it so that if someone did:
docker build --build-arg FOO .
and FOO wasn't set as an env var then it would pick-up FOO from the
Dockerfile's ARG cmd. However, it went too far and removed the ability
to specify a build arg w/o any value. Meaning it required the --build-arg
param to always be in the form "name=value", and not just "name".
This PR does the right fix - it allows just "name" and it'll grab the value
from the env vars if set. If "name" isn't set in the env then it still needs
to send "name" to the server so that a warning can be printed about an
unused --build-arg. And this is why buildArgs in the options is now a
*string instead of just a string - 'nil' == mentioned but no value.
Closes#29084
Signed-off-by: Doug Davis <dug@us.ibm.com>
This fix convert DanglingOnly in ImagesPruneConfig to Filters,
so that it is possible to maintain API compatibility in the future.
Several integration tests have been added to cover changes.
This fix is related to 28497.
A follow up to this PR will be done once this PR is merged.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
