Commit graph

244 commits

Author SHA1 Message Date
Michael Crosby
45e328b0ac Remove libcontainerd status type
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-04 15:17:13 -04:00
John Howard
85ad4b16c1 Windows: Experimental: Allow containerd for runtime
Signed-off-by: John Howard <jhoward@microsoft.com>

This is the first step in refactoring moby (dockerd) to use containerd on Windows.
Similar to the current model in Linux, this adds the option to enable it for runtime.
It does not switch the graphdriver to containerd snapshotters.

 - Refactors libcontainerd to a series of subpackages so that either a
  "local" containerd (1) or a "remote" (2) containerd can be loaded as opposed
  to conditional compile as "local" for Windows and "remote" for Linux.

 - Updates libcontainerd such that Windows has an option to allow the use of a
   "remote" containerd. Here, it communicates over a named pipe using GRPC.
   This is currently guarded behind the experimental flag, an environment variable,
   and the providing of a pipename to connect to containerd.

 - Infrastructure pieces such as under pkg/system to have helper functions for
   determining whether containerd is being used.

(1) "local" containerd is what the daemon on Windows has used since inception.
It's not really containerd at all - it's simply local invocation of HCS APIs
directly in-process from the daemon through the Microsoft/hcsshim library.

(2) "remote" containerd is what docker on Linux uses for it's runtime. It means
that there is a separate containerd service running, and docker communicates over
GRPC to it.

To try this out, you will need to start with something like the following:

Window 1:
	containerd --log-level debug

Window 2:
	$env:DOCKER_WINDOWS_CONTAINERD=1
	dockerd --experimental -D --containerd \\.\pipe\containerd-containerd

You will need the following binary from github.com/containerd/containerd in your path:
 - containerd.exe

You will need the following binaries from github.com/Microsoft/hcsshim in your path:
 - runhcs.exe
 - containerd-shim-runhcs-v1.exe

For LCOW, it will require and initrd.img and kernel in `C:\Program Files\Linux Containers`.
This is no different to the current requirements. However, you may need updated binaries,
particularly initrd.img built from Microsoft/opengcs as (at the time of writing), Linuxkit
binaries are somewhat out of date.

Note that containerd and hcsshim for HCS v2 APIs do not yet support all the required
functionality needed for docker. This will come in time - this is a baby (although large)
step to migrating Docker on Windows to containerd.

Note that the HCS v2 APIs are only called on RS5+ builds. RS1..RS4 will still use
HCS v1 APIs as the v2 APIs were not fully developed enough on these builds to be usable.
This abstraction is done in HCSShim. (Referring specifically to runtime)

Note the LCOW graphdriver still uses HCS v1 APIs regardless.

Note also that this does not migrate docker to use containerd snapshotters
rather than graphdrivers. This needs to be done in conjunction with Linux also
doing the same switch.
2019-03-12 18:41:55 -07:00
Sebastiaan van Stijn
3449b12cc7
Use assert.NilError() instead of assert.Assert()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-01-21 13:16:02 +01:00
Kir Kolyshkin
6533136961 pkg/mount: wrap mount/umount errors
The errors returned from Mount and Unmount functions are raw
syscall.Errno errors (like EPERM or EINVAL), which provides
no context about what has happened and why.

Similar to os.PathError type, introduce mount.Error type
with some context. The error messages will now look like this:

> mount /tmp/mount-tests/source:/tmp/mount-tests/target, flags: 0x1001: operation not permitted

or

> mount tmpfs:/tmp/mount-test-source-516297835: operation not permitted

Before this patch, it was just

> operation not permitted

[v2: add Cause()]
[v3: rename MountError to Error, document Cause()]
[v4: fixes; audited all users]
[v5: make Error type private; changes after @cpuguy83 reviews]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-12-10 20:07:02 -08:00
John Stephens
b3e9f7b13b
Merge pull request #35521 from salah-khan/35507
Add --chown flag support for ADD/COPY commands for Windows
2018-08-17 11:31:16 -07:00
Salahuddin Khan
763d839261 Add ADD/COPY --chown flag support to Windows
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.

NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.

The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>

On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.

Signed-off-by: Salahuddin Khan <salah@docker.com>
2018-08-13 21:59:11 -07:00
Derek McGowan
dd2e19ebd5
libcontainerd: split client and supervisor
Adds a supervisor package for starting and monitoring containerd.
Separates grpc connection allowing access from daemon.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-06 10:23:04 -07:00
Vincent Demeester
3845728524
Update tests to use gotest.tools 👼
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-06-13 09:04:30 +02:00
Tonis Tiigi
44f3dd7653 vendor: update containerd to 63522d9
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 19:19:06 -07:00
Vincent Demeester
712dd62a34
Merge pull request #37234 from kolyshkin/plugin-panic
Fix daemon panic on restart when a plugin is running
2018-06-08 09:45:25 +02:00
Brian Goff
dbeb432965 Fix panic on daemon restart with running plugin
Scenario:

Daemon is ungracefully shutdown and leaves plugins running (no
live-restore).
Daemon comes back up.
The next time a container tries to use that plugin it will cause a
daemon panic because the plugin client is not set.

This fixes that by ensuring that the plugin does get shutdown.
Note, I do not think there would be any harm in just re-attaching to the
running plugin instead of shutting it down, however historically we shut
down plugins and containers when live-restore is not enabled.

[kir@: consolidate code to deleteTaskAndContainer, a few minor nits]

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-06-07 17:27:02 -07:00
Joe Ferguson
76e99e1a8b Close readclosers returned by DecompressStream
Signed-off-by: Joe Ferguson <joe@infosiftr.com>
2018-06-07 11:16:13 -07:00
Brian Goff
f51a96c016 Move plugin client creation to the extension point
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-05-25 15:18:53 -04:00
Sebastiaan van Stijn
f23c00d870
Various code-cleanup
remove unnescessary import aliases, brackets, and so on.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-23 17:50:54 +02:00
Sebastiaan van Stijn
3646562a5c
Merge pull request #37088 from ohbarye/fix-typos-duplicated-the
Fix typos: remove duplicated "the"
2018-05-19 20:57:47 +02:00
Sebastiaan van Stijn
4f8c870d62
Fix some linting issues
These showed locally when running `make validate`. CI doesn't seem to have the
same (possibly it's disabled in the configuration)

    builder/fscache/fscache.go:618::error: github.com/docker/docker/vendor/github.com/tonistiigi/fsutil.StatInfo composite literal uses unkeyed fields (vet)
    client/swarm_unlock_test.go:44::error: github.com/docker/docker/api/types/swarm.UnlockRequest composite literal uses unkeyed fields (vet)
    client/swarm_unlock_test.go:20::error: github.com/docker/docker/api/types/swarm.UnlockRequest composite literal uses unkeyed fields (vet)
    cmd/dockerd/daemon_unix.go:113::error: github.com/docker/docker/cmd/dockerd/hack.MalformedHostHeaderOverride composite literal uses unkeyed fields (vet)
    cmd/dockerd/daemon_unix.go:110::error: github.com/docker/docker/cmd/dockerd/hack.MalformedHostHeaderOverride composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay/overlay.go:171::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay/overlay.go:413::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay2/overlay.go:203::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay2/overlay.go:584::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/zfs/zfs.go:109::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/zfs/zfs.go:388::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/volumes_windows.go:27::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    integration/service/network_test.go:31::error: github.com/docker/docker/api/types/network.NetworkingConfig composite literal uses unkeyed fields (vet)
    api/server/server.go:129:10⚠️ should not use basic type string as key in context.WithValue (golint)
    integration/service/network_test.go:54::error: github.com/docker/docker/api/types/network.NetworkingConfig composite literal uses unkeyed fields (vet)
    libcontainerd/client_daemon_linux.go:61::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    libcontainerd/client_daemon_linux.go:74::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    pkg/archive/archive_windows.go:76::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    plugin/manager_linux.go:56::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-17 19:28:27 +02:00
ohbarye
0f95b23d98 Fix typos: remove duplicated "the"
Signed-off-by: Masato Ohba <over.rye@gmail.com>
2018-05-17 21:49:51 +09:00
Kir Kolyshkin
7d62e40f7e Switch from x/net/context -> context
Since Go 1.7, context is a standard package. Since Go 1.9, everything
that is provided by "x/net/context" is a couple of type aliases to
types in "context".

Many vendored packages still use x/net/context, so vendor entry remains
for now.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-23 13:52:44 -07:00
Vincent Demeester
a7999aaa53
Skip some tests requires root uid when run as user
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-23 10:14:39 +02:00
Vincent Demeester
859e43e64c
Merge pull request #36715 from cpuguy83/plugin_exec_fixes
Make sure plugin container is removed on failure
2018-03-30 13:17:31 +02:00
Brian Goff
ec90839ca3 Don't sort plugin mounts slice
This was added as part of a53930a04f with
the intent to sort the mounts in the plugin config, but this was sorting
*all* the mounts from the default OCI spec which is problematic.

In reality we don't need to sort this because we are only adding a
self-binded mount to flag it as rshared.

We may want to look at sorting the plugin mounts before they are added
to the OCI spec in the future, but for now I think the existing behavior
is fine since the plugin author has control of the order (except for the
propagated mount).

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-03-28 09:10:43 -04:00
Brian Goff
f81172b903 Make sure plugin container is removed on failure
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-03-28 09:07:24 -04:00
Anusha Ragunathan
89a882e2f1 Add missing error return for plugin creation.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
2018-03-23 12:07:43 -07:00
John Stephens
3e1505e3e6
Merge pull request #36327 from Microsoft/jjh/block-pulling-uplevel
Windows: Block pulling uplevel images
2018-03-05 15:12:52 -08:00
John Howard
83908836d3 Windows: Block pulling uplevel images
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-02-26 12:33:54 -08:00
Sebastiaan van Stijn
20028325da
Merge pull request #35829 from cpuguy83/no_private_mount_for_plugins
Perform plugin mounts in the runtime
2018-02-21 12:28:13 +01:00
Brian Goff
e6f784e3df Make sure plugin mounts are cleaned up
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 15:54:52 -05:00
Brian Goff
37d7b7ddc3 Revert "Make plugins dir private."
This reverts commit 0c2821d6f2.

Due to other changes this is no longer needed and resolves some other
issues with plugins.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 15:48:27 -05:00
Brian Goff
0e5eaf8ee3 Ensure plugin returns correctly scoped paths
Before this change, volume management was relying on the fact that
everything the plugin mounts is visible on the host within the plugin's
rootfs. In practice this caused some issues with mount leaks, so we
changed the behavior such that mounts are not visible on the plugin's
rootfs, but available outside of it, which breaks volume management.

To fix the issue, allow the plugin to scope the path correctly rather
than assuming that everything is visible in `p.Rootfs`.
In practice this is just scoping the `PropagatedMount` paths to the
correct host path.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 15:48:27 -05:00
Brian Goff
a53930a04f Plugins perform propagated mount in runtime spec
Setting up the mounts on the host increases chances of mount leakage and
makes for more cleanup after the plugin has stopped.
With this change all mounts for the plugin are performed by the
container runtime and automatically cleaned up when the container exits.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 15:48:27 -05:00
Brian Goff
426e610e43 Use runtime spec modifier for metrics plugin hook
Currently the metrics plugin uses a really hackish host mount with
propagated mounts to get the metrics socket into a plugin after the
plugin is alreay running.
This approach ends up leaking mounts which requires setting the plugin
manager root to private, which causes some other issues.

With this change, plugin subsystems can register a set of modifiers to
apply to the plugin's runtime spec before the plugin is ever started.
This will help to generalize some of the customization work that needs
to happen for various plugin subsystems (and future ones).

Specifically it lets the metrics plugin subsystem append a mount to the
runtime spec to mount the metrics socket in the plugin's mount namespace
rather than the host's and prevetns any leaking due to this mount.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 15:48:26 -05:00
Daniel Nephin
4f0d95fa6e Add canonical import comment
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-02-05 16:51:57 -05:00
Vincent Demeester
6d4d3c52ae
Merge pull request #34372 from cpuguy83/more_error_handling_for_pluginrm
Ignore exist/not-exist errors on plugin remove
2018-01-25 20:45:17 -08:00
Brian Goff
93027b1ff2 Ignore exist/not-exist errors on plugin remove
During a plugin remove, docker performs an `os.Rename` to move the
plugin data dir to a new location before removing to acheive an atomic
removal.

`os.Rename` can return either a `NotExist` error if the source path
doesn't exist, or an `Exist` error if the target path already exists.
Both these cases can happen when there is an error on the final
`os.Remove` call, which is common on older kernels (`device or resource
busy`).

When calling rename, we can safely ignore these error types and proceed
to try and remove the plugin.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-01-25 09:23:54 -08:00
John Howard
afd305c4b5 LCOW: Refactor to multiple layer-stores based on feedback
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-01-18 08:31:05 -08:00
John Howard
ce8e529e18 LCOW: Re-coalesce stores
Signed-off-by: John Howard <jhoward@microsoft.com>

The re-coalesces the daemon stores which were split as part of the
original LCOW implementation.

This is part of the work discussed in https://github.com/moby/moby/issues/34617,
in particular see the document linked to in that issue.
2018-01-18 08:29:19 -08:00
Brian Goff
d453fe35b9 Move api/errdefs to errdefs
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-01-11 21:21:43 -05:00
Brian Goff
87a12421a9 Add helpers to create errdef errors
Instead of having to create a bunch of custom error types that are doing
nothing but wrapping another error in sub-packages, use a common helper
to create errors of the requested type.

e.g. instead of re-implementing this over and over:

```go
type notFoundError struct {
  cause error
}

func(e notFoundError) Error() string {
  return e.cause.Error()
}

func(e notFoundError) NotFound() {}

func(e notFoundError) Cause() error {
  return e.cause
}
```

Packages can instead just do:

```
  errdefs.NotFound(err)
```

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-01-11 21:21:43 -05:00
Daniel Nephin
3fec7c0858 Remove libcontainerd.IOPipe
replaced with cio.DirectIO

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-01-09 12:00:28 -05:00
Victor Vieux
745278d242
Merge pull request #35812 from stevvooe/follow-conventions
daemon, plugin: follow containerd namespace conventions
2017-12-19 15:55:39 -08:00
Sebastiaan van Stijn
6ed1163c98
Remove redundant build-tags
Files that are suffixed with `_linux.go` or `_windows.go` are
already only built on Linux / Windows, so these build-tags
were redundant.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-12-18 17:41:53 +01:00
Sebastiaan van Stijn
1589cc0a85
Remove Solaris files
Solaris is no longer being worked on, so these files
are now just dead code.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-12-18 17:22:25 +01:00
Stephen J Day
521e7eba86
daemon, plugin: follow containerd namespace conventions
Follow the conventions for namespace naming set out by other projects,
such as linuxkit and cri-containerd. Typically, they are some sort of
host name, with a subdomain describing functionality of the namespace.
In the case of linuxkit, services are launched in `services.linuxkit`.
In cri-containerd, pods are launched in `k8s.io`, making it clear that
these are from kubernetes.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-12-15 17:20:42 -08:00
Brian Goff
5017b5bef5 Add test to check for plugin mounts on remove
Ensures that when a plugin is removed that it doesn't interfere with
other plugins mounts and also ensures its own mounts are cleaned up.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2017-12-04 22:01:14 -05:00
Michael Crosby
aa3ce07c41 Update daemon code for containerd API changes
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-11-30 09:55:03 -05:00
Anusha Ragunathan
6572e27df7 Fix potential panic during plugin set.
Plugin config can have Mounts without a 'Source' field. In such cases,
performing a 'plugin set' on the mount source will panic the daemon. Its
the same case for device paths as well. This detects the case and
returns error.

Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
2017-11-28 13:06:24 -08:00
Victor Vieux
5745a8531e
Merge pull request #35265 from cpuguy83/32609_defreference_voldriver_on_error
Fixup some issues with plugin refcounting
2017-11-07 09:47:07 -08:00
Boaz Shuster
d75f1d8487 Nitpick plugin/manager.go: use loop to create directories
Instead of duplicating the same if condition per plugin manager directory,
use one if condition and a for-loop.

Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
2017-11-01 11:29:37 +02:00
Brian Goff
3816b51438 Fixup some issues with plugin refcounting
In some circumstances we were not properly releasing plugin references,
leading to failures in removing a plugin with no way to recover other
than restarting the daemon.

1. If volume create fails (in the driver)
2. If a driver validation fails (should be rare)
3. If trying to get a plugin that does not match the passed in capability

Ideally the test for 1 and 2 would just be a unit test, however the
plugin interfaces are too complicated as `plugingetter` relies on
github.com/pkg/plugin/Client (a concrete type), which will require
spinning up services from within the unit test... it just wouldn't be a
unit test at this point.
I attempted to refactor this a bit, but since both libnetwork and
swarmkit are reliant on `plugingetter` as well, this would not work.
This really requires a re-write of the lower-level plugin management to
decouple these pieces.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2017-10-21 15:17:57 -04:00
Kenfe-Mickael Laventure
ddae20c032
Update libcontainerd to use containerd 1.0
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-10-20 07:11:37 -07:00