Windows: Retry workaround for RS1/RS2 compute system enumeration
(cherry picked from commit 7761c69e23)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- otherwise the attachment task will stay in store and
consume IP addresses and there is no way to remove it.
Signed-off-by: Alessandro Boch <aboch@docker.com>
While it is important to not create controllers for an invalid task,
certain properties should only be checked immediately before use. Early
host validation of mounts prevents resolution of the task Executor when
the mounts are not relevant to execution flow. In this case, we have a
check for the existence of a bind mount path in a creation function that
prevents a task controller from being resolved. Such early validation
prevents one from interacting directly with a controller and result in
unnecessary error reporting.
In accordance with the above, we move the validation of the existence of
host bind mount paths to the `Controller.Start` phase. We also call
these "checks", as they are valid mounts but reference non-existent
paths.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
(cherry picked from commit 92899ffac8)
Signed-off-by: Victor Vieux <victorvieux@gmail.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
Working directory processing was handled differently for Hyper-V and Windows-Server containers, as annotated in the builder documentation (updated in this PR). For Hyper-V containers, the working directory set by WORKDIR was not created. This PR makes Hyper-V containers work the same as Windows Server containers (and the same as Linux).
Example (only applies to Hyper-V containers, so not reproducible under CI environment)
Dockerfile:
FROM microsoft/nanoserver
WORKDIR c:\installer
ENV GOROOT=c:\installer
ADD go.exe .
RUN go --help
Running on Windows Server 2016, using docker master without this change, but with daemon set to --exec-opt isolation=hyperv as it would be for Client operating systems.
PS E:\go\src\github.com\docker\docker> dockerd -g c:\control --exec-opt isolation=hyperv
time="2017-02-01T15:48:09.657286100-08:00" level=info msg="Windows default isolation mode: hyperv"
time="2017-02-01T15:48:09.662720900-08:00" level=info msg="[graphdriver] using prior storage driver: windowsfilter"
time="2017-02-01T15:48:10.011588000-08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2017-02-01T15:48:10.016655800-08:00" level=info msg="Loading containers: start."
time="2017-02-01T15:48:10.460820000-08:00" level=info msg="Loading containers: done."
time="2017-02-01T15:48:10.509859600-08:00" level=info msg="Daemon has completed initialization"
time="2017-02-01T15:48:10.509859600-08:00" level=info msg="Docker daemon" commit=3c64061 graphdriver=windowsfilter version=1.14.0-dev
First with no explicit isolation:
PS E:\docker\build\unifyworkdir> docker build --no-cache .
Sending build context to Docker daemon 10.1 MB
Step 1/5 : FROM microsoft/nanoserver
---> 89b8556cb9ca
Step 2/5 : WORKDIR c:\installer
---> 7e0f41d08204
Removing intermediate container 236c7802042a
Step 3/5 : ENV GOROOT c:\installer
---> Running in 8ea5237183c1
---> 394b70435261
Removing intermediate container 8ea5237183c1
Step 4/5 : ADD go.exe .
---> e47401a1745c
Removing intermediate container 88dcc28e74b1
Step 5/5 : RUN go --help
---> Running in efe90e1b6b8b
container efe90e1b6b8b76586abc5c1dc0e2797b75adc26517c48733d90651e767c8463b encountered an error during CreateProcess: failure in a Windows system call: The directory name is invalid. (0x10b) extra info: {"ApplicationName":"","CommandLine":"cmd /S /C go --help","User":"","WorkingDirectory":"C:\\installer","Environment":{"GOROOT":"c:\\installer"},"EmulateConsole":false,"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}
PS E:\docker\build\unifyworkdir>
Then forcing process isolation:
PS E:\docker\build\unifyworkdir> docker build --isolation=process --no-cache .
Sending build context to Docker daemon 10.1 MB
Step 1/5 : FROM microsoft/nanoserver
---> 89b8556cb9ca
Step 2/5 : WORKDIR c:\installer
---> 350c955980c8
Removing intermediate container 8339c1e9250c
Step 3/5 : ENV GOROOT c:\installer
---> Running in bde511c5e3e0
---> b8820063b5b6
Removing intermediate container bde511c5e3e0
Step 4/5 : ADD go.exe .
---> e4ac32f8902b
Removing intermediate container d586e8492eda
Step 5/5 : RUN go --help
---> Running in 9e1aa235af5f
Cannot mkdir: C:\installer is not a directory
PS E:\docker\build\unifyworkdir>
Now compare the same results after this PR. Again, first with no explicit isolation (defaulting to Hyper-V containers as that's what the daemon it set to) - note it now succeeds 😄
PS E:\docker\build\unifyworkdir> docker build --no-cache .
Sending build context to Docker daemon 10.1 MB
Step 1/5 : FROM microsoft/nanoserver
---> 89b8556cb9ca
Step 2/5 : WORKDIR c:\installer
---> 4f319f301c69
Removing intermediate container 61b9c0b1ff6f
Step 3/5 : ENV GOROOT c:\installer
---> Running in c464a1d612d8
---> 96a26ab9a7b5
Removing intermediate container c464a1d612d8
Step 4/5 : ADD go.exe .
---> 0290d61faf57
Removing intermediate container dc5a085fffe3
Step 5/5 : RUN go --help
---> Running in 60bd56042ff8
Go is a tool for managing Go source code.
Usage:
go command [arguments]
The commands are:
build compile packages and dependencies
clean remove object files
doc show documentation for package or symbol
env print Go environment information
fix run go tool fix on packages
fmt run gofmt on package sources
generate generate Go files by processing source
get download and install packages and dependencies
install compile and install packages and dependencies
list list packages
run compile and run Go program
test test packages
tool run specified go tool
version print Go version
vet run go tool vet on packages
Use "go help [command]" for more information about a command.
Additional help topics:
c calling between Go and C
buildmode description of build modes
filetype file types
gopath GOPATH environment variable
environment environment variables
importpath import path syntax
packages description of package lists
testflag description of testing flags
testfunc description of testing functions
Use "go help [topic]" for more information about that topic.
The command 'cmd /S /C go --help' returned a non-zero code: 2
And the same with forcing process isolation. Also works 😄
PS E:\docker\build\unifyworkdir> docker build --isolation=process --no-cache .
Sending build context to Docker daemon 10.1 MB
Step 1/5 : FROM microsoft/nanoserver
---> 89b8556cb9ca
Step 2/5 : WORKDIR c:\installer
---> f423b9cc3e78
Removing intermediate container 41330c88893d
Step 3/5 : ENV GOROOT c:\installer
---> Running in 0b99a2d7bf19
---> e051144bf8ec
Removing intermediate container 0b99a2d7bf19
Step 4/5 : ADD go.exe .
---> 7072e32b7c37
Removing intermediate container a7a97aa37fd1
Step 5/5 : RUN go --help
---> Running in 7097438a54e5
Go is a tool for managing Go source code.
Usage:
go command [arguments]
The commands are:
build compile packages and dependencies
clean remove object files
doc show documentation for package or symbol
env print Go environment information
fix run go tool fix on packages
fmt run gofmt on package sources
generate generate Go files by processing source
get download and install packages and dependencies
install compile and install packages and dependencies
list list packages
run compile and run Go program
test test packages
tool run specified go tool
version print Go version
vet run go tool vet on packages
Use "go help [command]" for more information about a command.
Additional help topics:
c calling between Go and C
buildmode description of build modes
filetype file types
gopath GOPATH environment variable
environment environment variables
importpath import path syntax
packages description of package lists
testflag description of testing flags
testfunc description of testing functions
Use "go help [topic]" for more information about that topic.
The command 'cmd /S /C go --help' returned a non-zero code: 2
PS E:\docker\build\unifyworkdir>
(cherry picked from commit f42033ba94)
Signed-off-by: Victor Vieux <victorvieux@gmail.com>
- When the node leaves the cluster, if any user run
container(s) is connected to the swarm network,
then daemon needs to detach the container(s) and
remove the network.
Signed-off-by: Alessandro Boch <aboch@docker.com>
(cherry picked from commit 3cedca5d53)
Signed-off-by: Victor Vieux <vieux@docker.com>
commit 56f77d5ade
added support for cpu-rt-period and cpu-rt-runtime,
but always initialized the cgroup path, even if not
used.
As a result, containers failed to start on a
read-only filesystem.
This patch only creates the cgroup path if
one of these options is set.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f285d5b3e8)
Signed-off-by: Victor Vieux <vieux@docker.com>
Upon each container create I'm seeing these warning **every** time in the
daemon output:
```
WARN[0002] Your kernel does not support swap memory limit
WARN[0002] Your kernel does not support cgroup rt period
WARN[0002] Your kernel does not support cgroup rt runtime
```
Showing them for each container.create() fills up the logs and encourages
people to ignore the output being generated - which means its less likely
they'll see real issues when they happen. In short, I don't think we
need to show these warnings more than once, so let's only show these
warnings at daemon start-up time.
Signed-off-by: Doug Davis <dug@us.ibm.com>
(cherry picked from commit ff42a2eb41)
Signed-off-by: Victor Vieux <vieux@docker.com>
When Size was reverted to be equal to VirtualSize, the df command
formatter was not correctly updated to account for the change.
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
(cherry picked from commit dcc1b4baf6)
Signed-off-by: Victor Vieux <vieux@docker.com>
This is necessary for swarmkit to support cluster wide plugins, such as
globally scoped network plugins.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
The prune PR changed the meaning of the file to mean "space on disk
only unique to this image", this PR revert this change.
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
(cherry picked from commit be20dc15af)
Signed-off-by: Victor Vieux <vieux@docker.com>
Call daemon.Mount will increase the refcount of mounted path,
for those previous running containers, `Mount` call will make
the refcount to 2. see
https://github.com/docker/docker/blob/v1.13.0-rc4/daemon/graphdriver/counter.go#L38
```
if !m.check {
m.check = true
if c.checker.IsMounted(path) {
m.count++
}
}
m.count++
```
graphdrive could restore on reboot after #22541, call
daemon.Mount to resore the graphdriver is not necessary.
And call `daemon.Mount` on restorting will mount all the containers
mounted layer even if it was stop.
This fix call Mount and then Unmount to get `BaseFs`
Signed-off-by: Lei Jitang <leijitang@huawei.com>
This fix is a follow up to #29365. In #29365 a bug was fixed for
`docker exec -u user` after live restore by remounting.
However, #29365 will prevent containers from restored if mount failed.
In this fix, containers will be restored even if mount in that step failed.
Some functionalities might be missing (like `docker exec -u user`) but
at least it is possible to do certain operations like stop/restart/delete.
This fix is related to #29365.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
(cherry picked from commit 3003ae1d8b)
returned in "docker info". Currently info endpoint isnt using the
GetAllByCap, but relies on existing networks to get the plugin names.
This causes a basic issue when it comes to global network plugins which
swarm-mode relies on, wherein swarmkit will not be able to schedule the
network on the worker nodes due to the filtering logic.
In order to break this chicken & egg issue, we must start to use the
GetAllManagedPluginsByCap. We are unable to use GetAllByCap due to
various issues with Plugin-V1's lazy loading approach causing issues
especially during daemon restarts (which uses SystemInfo)
Signed-off-by: Madhu Venugopal <madhu@docker.com>
Signed-off-by: Lei Jitang <leijitang@huawei.com>
(cherry picked from commit 737b5b1781)
Signed-off-by: Victor Vieux <vieux@docker.com>
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Move plugins to shared distribution stack with images.
Create immutable plugin config that matches schema2 requirements.
Ensure data being pushed is same as pulled/created.
Store distribution artifacts in a blobstore.
Run init layer setup for every plugin start.
Fix breakouts from unsafe file accesses.
Add support for `docker plugin install --alias`
Uses normalized references for default names to avoid collisions when using default hosts/tags.
Some refactoring of the plugin manager to support the change, like removing the singleton manager and adding manager config struct.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
(cherry picked from commit 3d86b0c79b)
Move configurations into a single file.
Abstract download manager in pull config.
Add supports for schema2 only and schema2 type checking.
Add interface for providing push layers.
Abstract image store to generically handle configurations.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
(cherry picked from commit 3c7676a057)
Ensures all known volumes (known b/c they are persisted to disk) have
their volume drivers refcounted properly.
In testing this, I found an issue with `--live-restore` (required since
currently the provided volume plugin doesn't keep state on restart)
where restorted plugins did not have a plugin client loaded causing a
panic when trying to use the plugin.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 6ef1060cd0)
Signed-off-by: Victor Vieux <vieux@docker.com>
Previously, it doesn't allow creating such a network:
e.g.
$ docker network inspect -f '{{.Id}}' ingress
84xh9knigj6zyt00u31e26nj3
$ docker network create 84
Error response from daemon: network with name 84 already exists
Fix#27866
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
(cherry picked from commit edfbc3b876)
Signed-off-by: Victor Vieux <vieux@docker.com>
Also enables `PropagatedMount` for graphdrivers.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 500210475f)
Signed-off-by: Victor Vieux <vieux@docker.com>
This fix tries to address the issue raised in 29342 where
`docker exec -u` after docker daemon restart returns an error:
```
unable to find user test: no matching entries in passwd file
```
The reason was that `container.BaseFS` is not present after restart.
This fix adds the `daemon.Mount` during the restore to bring up the
`container.BaseFS`.
An integration test has been added to cover the changes.
This fix fixes 29342.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
(cherry picked from commit 7feb2a17e4)
Signed-off-by: Victor Vieux <vieux@docker.com>
Implement notary-based digest lookup in the client when
DOCKER_CONTENT_TRUST=1.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
(cherry picked from commit d4d6f8c0d0)
Adds 2 new methods to v2 plugin `Acquire` and `Release` which allow
refcounting directly at the plugin level instead of just the store.
Since a graphdriver is initialized exactly once, and is really managed
by a separate object, it didn't really seem right to call
`getter.Get()` to refcount graphdriver plugins.
On shutdown it was particularly weird where we'd either need to keep a
driver reference in daemon, or keep a reference to the pluggin getter in
the layer store, and even then still store extra details on if the
graphdriver is a plugin or not.
Instead the plugin proxy itself will handle calling the neccessary
refcounting methods directly on the plugin object.
Also adds a new interface in `plugingetter` to account for these new
functions which are not going to be implemented by v1 plugins.
Changes terms `plugingetter.CREATE` and `plugingetter.REMOVE` to
`ACQUIRE` and `RELEASE` respectively, which seems to be better
adjectives for what we're doing.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit f29bbd16f5)
Signed-off-by: Victor Vieux <vieux@docker.com>
In certain cases (unattended upgrades), system services can disable
loaded AppArmor profiles. However, since /etc being read-only is a
supported setup we cannot just write a copy of the profile to
/etc/apparmor.d.
Instead, dynamically load the docker-default AppArmor profile if a
container is started with that profile set. This code will short-cut if
the profile is already loaded.
Fixes: 2f7596aaef ("apparmor: do not save profile to /etc/apparmor.d")
Signed-off-by: Aleksa Sarai <asarai@suse.de>
(cherry picked from commit 567ef8e785)
Signed-off-by: Victor Vieux <vieux@docker.com>
This fix tries to fix the issue raised in 29044 where
the IPAM options is missing in swarm network mode
after the service is deployed. Before the service
is deployed, the IPAM options is available.
The reason for the issue is that, before service is
deployed, `network inspect` is querying the swarm and
obtained the correct information.
However, after service is deployed, swarm executor
does not pass the IPAM options to the backend (daemon).
Also after service is deployed, `network inspect` is
actually querying the local daemon for information.
At this time the network information with missing IPAM
options is returned.
This fix fixes the issue by updating the swarm network
allocator and swarm executor.
A separate PR for swarmkit will be opened.
An integration test has been added to cover the change.
This fix fixes 29044.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
(cherry picked from commit 4d958e99c1)
Signed-off-by: Victor Vieux <vieux@docker.com>
