Commit graph

324 commits

Author SHA1 Message Date
Yusuf Tarık Günaydın
86bd2e9864 Implemented memory and CPU limits for LCOW.
Signed-off-by: Yusuf Tarık Günaydın <yusuf_tarik@hotmail.com>
2019-02-02 13:02:23 +03:00
Simão Reis
3134161be3 Fix nil pointer derefence on failure to connect to containerd
Signed-off-by: Simão Reis <smnrsti@gmail.com>
2019-01-30 12:41:54 -01:00
Tonis Tiigi
332f134890 libcontainerd: prevent exec delete locking
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-12-17 12:22:37 +02:00
Justin Terry (VM)
b2d99865ea Add --device support for Windows
Implements the --device forwarding for Windows daemons. This maps the physical
device into the container at runtime.

Ex:

docker run --device="class/<clsid>" <image> <cmd>

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-11-21 15:31:17 -08:00
Sebastiaan van Stijn
dd7799afd4
update containerd client and dependencies to v1.2.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-11-05 18:46:26 +01:00
Wei Fu
c7890f25a9 bugfix: wait for stdin creation before CloseIO
The stdin fifo of exec process is created in containerd side after
client calls Start. If the client calls CloseIO before Start call, the
stdin of exec process is still opened and wait for close.

For this case, client closes stdinCloseSync channel after Start.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-10-10 19:59:01 +08:00
Tibor Vass
34eede0296 Remove 'docker-' prefix for containerd and runc binaries
This allows to run the daemon in environments that have upstream containerd installed.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-24 21:49:03 +00:00
Sebastiaan van Stijn
06b9588c2d
Merge pull request #37759 from dmcgowan/fix-libcontainerd-startup-error
Add fail fast path when containerd fails on startup
2018-09-14 15:15:38 +02:00
Derek McGowan
ce0b0b72bc
Add fail fast path when containerd fails on startup
Prevents looping of startup errors such as containerd
not being found on the path.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-13 17:34:52 -07:00
Kir Kolyshkin
9b0097a699 Format code with gofmt -s from go-1.11beta1
This should eliminate a bunch of new (go-1.11 related) validation
errors telling that the code is not formatted with `gofmt -s`.

No functional change, just whitespace (i.e.
`git show --ignore-space-change` shows nothing).

Patch generated with:

> git ls-files | grep -v ^vendor/ | grep .go$ | xargs gofmt -s -w

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-09-06 15:24:16 -07:00
Derek McGowan
c3e3293843
Fix supervisor healthcheck throttling
Fix default case causing the throttling to not be used.
Ensure that nil client condition is handled.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-04 11:00:28 -07:00
John Howard
5accd82634 Add containerd.WithTimeout(60*time.Second) to match old calls
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-08-23 12:03:43 -07:00
Sebastiaan van Stijn
7d4fa69e33
Merge pull request #37664 from dmcgowan/vendor-containerd-1.2
Update containerd vendor to 1.2 beta
2018-08-18 22:22:41 +01:00
Derek McGowan
98e23f2a8e
Update containerd vendor to 1.2 beta
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-17 13:08:22 -07:00
John Stephens
b3e9f7b13b
Merge pull request #35521 from salah-khan/35507
Add --chown flag support for ADD/COPY commands for Windows
2018-08-17 11:31:16 -07:00
Sebastiaan van Stijn
4d62192646
Merge pull request #37149 from dmcgowan/split-libcontainerd
libcontainerd: split client and daemon supervision
2018-08-16 19:00:10 +02:00
Salahuddin Khan
763d839261 Add ADD/COPY --chown flag support to Windows
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.

NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.

The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>

On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.

Signed-off-by: Salahuddin Khan <salah@docker.com>
2018-08-13 21:59:11 -07:00
Derek McGowan
dd2e19ebd5
libcontainerd: split client and supervisor
Adds a supervisor package for starting and monitoring containerd.
Separates grpc connection allowing access from daemon.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-06 10:23:04 -07:00
John Howard
5cfededc7c Don't invoke HCS shutdown if terminate called
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-08-02 16:29:37 -07:00
Sebastiaan van Stijn
13a8031e1c
Merge pull request #37514 from jterry75/linux_metadata
Adds LinuxMetadata support by default on Windows
2018-07-27 17:51:10 +01:00
Sebastiaan van Stijn
c3a0207714
Merge pull request #37360 from bjbroder/checkpoint-exit
Fix checkpoint's exiting semantics.
2018-07-26 14:25:49 +02:00
Brielle Broder
db621eb7ee
Fix checkpoint's exiting semantics.
Previously, dockerd would always ask containerd to pass --leave-running
to runc/runsc, ignoring the exit boolean value. Hence, even `docker
checkpoint create --leave-running=false ...` would not stop the
container.

Signed-off-by: Brielle Broder <bbroder@google.com>
2018-07-25 09:21:33 +02:00
Derek McGowan
8fb5f4d5c9
Add configuration option for containerd cri
Disable cri plugin by default in containerd and
allows an option to enable the plugin. This only
has an effect on containerd when supervised by
dockerd. When containerd is managed outside of
dockerd, the configuration is not effected.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-24 11:34:47 -07:00
Justin Terry (VM)
e93077bcdf Adds LinuxMetadata support by default on Windows
1. Sets the LinuxMetadata flag by default on Windows LCOW v1
MappedDirectories.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-07-23 09:59:35 -07:00
Sebastiaan van Stijn
aaa1392279
Pass log-level to containerd
dockerd allows the `--log-level` to be specified, but this log-level
was not forwarded to the containerd process.

This patch sets containerd's log-level to the same as dockerd if a
custom level is provided.

Now that `--log-level` is also passed to containerd, the default "info"
is removed, so that containerd's default (or the level configured in containerd.toml)
is still used if no log-level is set.

Before this change:

containerd would always be started without a log-level set (only the level that's configured in `containerd.toml`);

```
root      1014  2.5  2.1 496484 43468 pts/0    Sl+  12:23   0:00 dockerd
root      1023  1.2  1.1 681768 23832 ?        Ssl  12:23   0:00  \_ docker-containerd --config /var/run/docker/containerd/containerd.toml
```

After this change:

when running `dockerd` without options (same as current);

```
root      1014  2.5  2.1 496484 43468 pts/0    Sl+  12:23   0:00 dockerd
root      1023  1.2  1.1 681768 23832 ?        Ssl  12:23   0:00  \_ docker-containerd --config /var/run/docker/containerd/containerd.toml
```

when running `dockerd --debug`:

```
root       600  0.8  2.1 512876 43180 pts/0    Sl+  12:20   0:00 dockerd --debug
root       608  0.6  1.1 624428 23672 ?        Ssl  12:20   0:00  \_ docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level debug
```

when running `dockerd --log-level=panic`

```
root       747  0.6  2.1 496548 43996 pts/0    Sl+  12:21   0:00 dockerd --log-level=panic
root       755  0.7  1.1 550696 24100 ?        Ssl  12:21   0:00  \_ docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level panic
```

combining `--debug` and `--log-level` (`--debug` takes precedence):

```
root       880  2.7  2.1 634692 43336 pts/0    Sl+  12:23   0:00 dockerd --debug --log-level=panic
root       888  1.0  1.1 616232 23652 ?        Ssl  12:23   0:00  \_ docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level debug
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-07-09 15:38:08 +02:00
Vincent Demeester
3845728524
Update tests to use gotest.tools 👼
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-06-13 09:04:30 +02:00
Tonis Tiigi
44f3dd7653 vendor: update containerd to 63522d9
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 19:19:06 -07:00
Derek McGowan
a0009345f5
Update containerd daemon
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-04 15:37:03 -07:00
Derek McGowan
52ed3e0896
Update containerd to 1.1
Updates swarmkit, grpc, and all related vendors

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-04 15:37:03 -07:00
Sebastiaan van Stijn
f23c00d870
Various code-cleanup
remove unnescessary import aliases, brackets, and so on.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-23 17:50:54 +02:00
Sebastiaan van Stijn
4f8c870d62
Fix some linting issues
These showed locally when running `make validate`. CI doesn't seem to have the
same (possibly it's disabled in the configuration)

    builder/fscache/fscache.go:618::error: github.com/docker/docker/vendor/github.com/tonistiigi/fsutil.StatInfo composite literal uses unkeyed fields (vet)
    client/swarm_unlock_test.go:44::error: github.com/docker/docker/api/types/swarm.UnlockRequest composite literal uses unkeyed fields (vet)
    client/swarm_unlock_test.go:20::error: github.com/docker/docker/api/types/swarm.UnlockRequest composite literal uses unkeyed fields (vet)
    cmd/dockerd/daemon_unix.go:113::error: github.com/docker/docker/cmd/dockerd/hack.MalformedHostHeaderOverride composite literal uses unkeyed fields (vet)
    cmd/dockerd/daemon_unix.go:110::error: github.com/docker/docker/cmd/dockerd/hack.MalformedHostHeaderOverride composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay/overlay.go:171::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay/overlay.go:413::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay2/overlay.go:203::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay2/overlay.go:584::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/zfs/zfs.go:109::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/zfs/zfs.go:388::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/volumes_windows.go:27::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    integration/service/network_test.go:31::error: github.com/docker/docker/api/types/network.NetworkingConfig composite literal uses unkeyed fields (vet)
    api/server/server.go:129:10⚠️ should not use basic type string as key in context.WithValue (golint)
    integration/service/network_test.go:54::error: github.com/docker/docker/api/types/network.NetworkingConfig composite literal uses unkeyed fields (vet)
    libcontainerd/client_daemon_linux.go:61::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    libcontainerd/client_daemon_linux.go:74::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    pkg/archive/archive_windows.go:76::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    plugin/manager_linux.go:56::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-17 19:28:27 +02:00
Vincent Demeester
8bb5a28eed
Merge pull request #36684 from cpuguy83/bump_containerd_client
Bump containerd client
2018-04-19 10:38:43 +02:00
Brian Goff
806700e410 Relax global client lock on containerd restore.
This unblocks the client to take other restore requests and makes sure
that a long/stuck request can't block the client forever.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-17 12:07:06 -04:00
Vincent Demeester
859e43e64c
Merge pull request #36715 from cpuguy83/plugin_exec_fixes
Make sure plugin container is removed on failure
2018-03-30 13:17:31 +02:00
Brian Goff
2c682d5209 Use containerd client Reconnect() API.
This fixes an issue where the containerd client is cached in a container
object in libcontainerd and becomes stale after containerd is restarted.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-03-28 09:43:08 -04:00
Brian Goff
f81172b903 Make sure plugin container is removed on failure
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-03-28 09:07:24 -04:00
Shukui Yang
a50d9c0765 Add if judgment before receiving operations on daemonWaitCh
receive operations on a nil channel will always block, this lead containerd not started.

Signed-off-by: Shukui Yang <yangshukui@huawei.com>
2018-03-21 05:53:55 -04:00
Daniel Nephin
6be0f70983 Automated migration using
gty-migrate-from-testify --ignore-build-tags

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-03-16 11:03:43 -04:00
Brian Goff
04a0d6b863 Change containerd monitor ticker to sleep
With the ticker this could end up just doing back-to-back checks, which
isn't really what we want here.
Instead use a sleep to ensure we actually sleep for the desired
interval.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-03-13 13:20:19 -04:00
John Howard
d4f37c0885 Windows: Remove servicing mode
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-02-27 08:48:31 -08:00
Yong Tang
66e6beeb24
Merge pull request #35967 from Microsoft/jjh/32838-pass-container-shutdown-error-back
Windows: Pass back system errors on container exit
2018-02-22 19:12:10 -08:00
John Howard
8c52560ea4 Windows: Pass back system errors on container exit
Signed-off-by: John Howard <jhoward@microsoft.com>

While debugging #32838, it was found (https://github.com/moby/moby/issues/32838#issuecomment-356005845) that the utility VM in some circumstances was crashing. Unfortunately, this was silently thrown away, and as far as the build step (also applies to docker run) was concerned, the exit code was zero and the error was thrown away. Windows containers operate differently to containers on Linux, and there can be legitimate system errors during container shutdown after the init process exits. This PR handles this and passes the error all the way back to the client, and correctly causes a build step running a container which hits a system error to fail, rather than blindly trying to keep going, assuming all is good, and get a subsequent failure on a commit.

With this change, assuming an error occurs, here's an example of a failure which previous was reported as a commit error:

```
The command 'powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; Install-WindowsFeature -Name Web-App-Dev ;   Install-WindowsFeature -Name ADLDS;   Install-WindowsFeature -Name Web-Mgmt-Compat;   Install-WindowsFeature -Name Web-Mgmt-Service;   Install-WindowsFeature -Name Web-Metabase;   Install-WindowsFeature -Name Web-Lgcy-Scripting;   Install-WindowsFeature -Name Web-WMI;   Install-WindowsFeature -Name Web-WHC;   Install-WindowsFeature -Name Web-Scripting-Tools;   Install-WindowsFeature -Name Web-Net-Ext45;   Install-WindowsFeature -Name Web-ASP;   Install-WindowsFeature -Name Web-ISAPI-Ext;   Install-WindowsFeature -Name Web-ISAPI-Filter;   Install-WindowsFeature -Name Web-Default-Doc;   Install-WindowsFeature -Name Web-Dir-Browsing;   Install-WindowsFeature -Name Web-Http-Errors;   Install-WindowsFeature -Name Web-Static-Content;   Install-WindowsFeature -Name Web-Http-Redirect;   Install-WindowsFeature -Name Web-DAV-Publishing;   Install-WindowsFeature -Name Web-Health;   Install-WindowsFeature -Name Web-Http-Logging;   Install-WindowsFeature -Name Web-Custom-Logging;   Install-WindowsFeature -Name Web-Log-Libraries;   Install-WindowsFeature -Name Web-Request-Monitor;   Install-WindowsFeature -Name Web-Http-Tracing;   Install-WindowsFeature -Name Web-Stat-Compression;   Install-WindowsFeature -Name Web-Dyn-Compression;   Install-WindowsFeature -Name Web-Security;   Install-WindowsFeature -Name Web-Windows-Auth;   Install-WindowsFeature -Name Web-Basic-Auth;   Install-WindowsFeature -Name Web-Url-Auth;   Install-WindowsFeature -Name Web-WebSockets;   Install-WindowsFeature -Name Web-AppInit;   Install-WindowsFeature -Name NET-WCF-HTTP-Activation45;   Install-WindowsFeature -Name NET-WCF-Pipe-Activation45;   Install-WindowsFeature -Name NET-WCF-TCP-Activation45;' returned a non-zero code: 4294967295: container shutdown failed: container ba9c65054d42d4830fb25ef55e4ab3287550345aa1a2bb265df4e5bfcd79c78a encountered an error during WaitTimeout: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106)
```

Without this change, it would be incorrectly reported such as in this comment: https://github.com/moby/moby/issues/32838#issuecomment-309621097

```
Step 3/8 : ADD buildtools C:/buildtools
re-exec error: exit status 1: output: time="2017-06-20T11:37:38+10:00" level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\\\?\\C:\\ProgramData\\docker\\windowsfilter\\b41d28c95f98368b73fc192cb9205700e21
6691495c1f9ac79b9b04ec4923ea2 flavour=1 folder=C:\\Windows\\TEMP\\hcs232661915"
hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\?\C:\ProgramData\docker\windowsfilter\b41d28c95f98368b73fc192cb9205700e216691495c1f9ac79b9b04ec4923ea2 flavour=1 folder=C:\Windows\TEMP\hcs232661915
```
2018-02-22 08:53:43 -08:00
John Stephens
bf1345d0b6
Merge pull request #36268 from Microsoft/jjh/rs3-bump
Windows: Bump to final RS3 build number
2018-02-12 14:49:33 -08:00
John Howard
c04504383a Windows: Bump to final RS3 build number
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-02-09 11:39:57 -08:00
Brian Goff
c0d56ab717 Fix container cleanup on daemon restart
When the daemon restores containers on daemon restart, it syncs up with
containerd to determine the existing state. For stopped containers it
then removes the container metadata from containerd.

In some cases this is not handled properly and causes an error when
someone attempts to start that container again.
In particular, this case is just a bad error check.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-09 14:36:36 -05:00
Yong Tang
384ff69f2f
Merge pull request #36173 from cpuguy83/fix_containerd_crash_spin
Refresh containerd remotes on containerd restarted
2018-02-08 06:19:29 -08:00
Brian Goff
400126f869 Refresh containerd remotes on containerd restarted
Before this patch, when containerd is restarted (due to a crash, or
kill, whatever), the daemon would keep trying to process the event
stream against the old socket handles. This would lead to a CPU spin due
to the error handling when the client can't connect to containerd.

This change makes sure the containerd remote client is updated for all
registered libcontainerd clients.

This is not neccessarily the ideal fix which would likely require a
major refactor, but at least gets things to a working state with a
minimal patch.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 11:53:00 -05:00
Daniel Nephin
4f0d95fa6e Add canonical import comment
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-02-05 16:51:57 -05:00
Brian Goff
d453fe35b9 Move api/errdefs to errdefs
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-01-11 21:21:43 -05:00
Brian Goff
87a12421a9 Add helpers to create errdef errors
Instead of having to create a bunch of custom error types that are doing
nothing but wrapping another error in sub-packages, use a common helper
to create errors of the requested type.

e.g. instead of re-implementing this over and over:

```go
type notFoundError struct {
  cause error
}

func(e notFoundError) Error() string {
  return e.cause.Error()
}

func(e notFoundError) NotFound() {}

func(e notFoundError) Cause() error {
  return e.cause
}
```

Packages can instead just do:

```
  errdefs.NotFound(err)
```

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-01-11 21:21:43 -05:00