Commit graph

39479 commits

Author SHA1 Message Date
Sebastiaan van Stijn
9b2f55bc1c
update containerd binary to v1.5.0
Welcome to the v1.5.0 release of containerd!

The sixth major release of containerd includes many stability improvements
and code organization changes to make contribution easier and make future
features cleaner to develop. This includes bringing CRI development into the
main containerd repository and switching to Go modules. This release also
brings support for the Node Resource Interface (NRI).

Highlights
--------------------------------------------------------------------------------

*Project Organization*

- Merge containerd/cri codebase into containerd/containerd
- Move to Go modules
- Remove selinux build tag
- Add json log format output option for daemon log

*Snapshots*

- Add configurable overlayfs path
- Separate overlay implementation from plugin
- Native snapshotter configuration and plugin separation
- Devmapper snapshotter configuration and plugin separation
- AUFS snapshotter configuration and plugin separation
- ZFS snapshotter configuration and plugin separation
- Pass custom snapshot labels when creating snapshot
- Add platform check for snapshotter support when unpacking
- Handle loopback mounts
- Support userxattr mount option for overlay in user namespace
- ZFS snapshotter implementation of usage

*Distribution*

- Improve registry response errors
- Improve image pull performance over HTTP 1.1
- Registry configuration package
- Add support for layers compressed with zstd
- Allow arm64 to fallback to arm (v8, v7, v6, v5)

*Runtime*

- Add annotations to containerd task update API
- Add logging binary support when terminal is true
- Runtime support on FreeBSD

*Windows*

- Implement windowsDiff.Compare to allow outputting OCI images
- Optimize WCOW snapshotter to commit writable layers as read-only parent layers
- Optimize LCOW snapshotter use of scratch layers

*CRI*

- Add NRI injection points cri#1552
- Add support for registry host directory configuration
- Update privileged containers to use current capabilities instead of known capabilities
- Add pod annotations to CNI call
- Enable ocicrypt by default
- Support PID NamespaceMode_TARGET

Impactful Client Updates
--------------------------------------------------------------------------------

This release has changes which may affect projects which import containerd.

*Switch to Go modules*

containerd and all containerd sub-repositories are now using Go modules. This
should help make importing easier for handling transitive dependencies. As of
this release, containerd still does not guarantee client library compatibility
for 1.x versions, although best effort is made to minimize impact from changes
to exported Go packages.

*CRI plugin moved to main repository*

With the CRI plugin moving into the main repository, imports under github.com/containerd/cri/
can now be found github.com/containerd/containerd/pkg/cri/.
There are no changes required for end users of CRI.

*Library changes*

oci

The WithAllCapabilities has been removed and replaced with WithAllCurrentCapabilities
and WithAllKnownCapabilities. WithAllKnownCapabilities has similar
functionality to the previous WithAllCapabilities with added support for newer
capabilities. WithAllCurrentCapabilities can be used to give privileged
containers the same set of permissions as the calling process, preventing errors
when privileged containers attempt to get more permissions than given to the
caller.

*Configuration changes*

New registry.config_path for CRI plugin

registry.config_path specifies a directory to look for registry hosts
configuration. When resolving an image name during pull operations, the CRI
plugin will look in the <registry.config_path>/<image hostname>/ directory
for host configuration. An optional hosts.toml file in that directory may be
used to configure which hosts will be used for the pull operation as well
host-specific configurations. Updates under that directory do not require
restarting the containerd daemon.

Enable registry.config_path in the containerd configuration file.

    [plugins."io.containerd.grpc.v1.cri".registry]
       config_path = "/etc/containerd/certs.d"
    Configure registry hosts, such as /etc/containerd/certs.d/docker.io/hosts.toml
    for any image under the docker.io namespace (any image on Docker Hub).

    server = "https://registry-1.docker.io"

    [host."https://public-mirror.example.com"]
      capabilities = ["pull"]
    [host."https://docker-mirror.internal"]
      capabilities = ["pull", "resolve"]
      ca = "docker-mirror.crt"

If no hosts.toml configuration exists in the host directory, it will fallback
to check certificate files based on Docker's certificate file
pattern (".crt" files for CA certificates and ".cert"/".key" files for client
certificates).

*Deprecation of registry.mirrors and registry.configs in CRI plugin*

Mirroring and TLS can now be configured using the new registry.config_path
option. Existing configurations may be migrated to new host directory
configuration. These fields are only deprecated with no planned removal,
however, these configurations cannot be used while registry.config_path is
defined.

*Version 1 schema is deprecated*

Version 2 of the containerd configuration toml is recommended format and the
default. Starting this version, a deprecation warning will be logged when
version 1 is used.

To check version, see the version value in the containerd toml configuration.

    version=2

FreeBSD Runtime Support (Experimental)
--------------------------------------------------------------------------------

This release includes changes that allow containerd to run on FreeBSD with a
compatible runtime, such as runj. This
support should be considered experimental and currently there are no official
binary releases for FreeBSD. The runtimes used by containerd are maintained
separately and have their own stability guarantees. The containerd project
strives to be compatible with any runtime which aims to implement containerd's
shim API and OCI runtime specification.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-05-04 15:45:01 +02:00
Tianon Gravi
5a82cee6d4
Merge pull request #42077 from WhyNotHugo/patch-1
rootless: Set service type to `notify`
2021-05-03 10:36:03 -07:00
Tianon Gravi
27a066ce53
Merge pull request #42343 from zzeebbii/fix-test-cleanup
Added missing test cleanup for temporary directory
2021-05-03 10:33:12 -07:00
Muhammad Zohaib Aslam
56c88c94dd Added missing test cleanup for temporary directory
A temporary directory was created but not removed at the end of the test.
The missing remove directory call is added now.

Signed-off-by: Muhammad Zohaib Aslam <zohaibse011@gmail.com>
2021-05-01 15:39:50 +03:00
Sebastiaan van Stijn
59751bb233
Merge pull request #42331 from AkihiroSuda/dind-fix-cgroup2-evac
hack/dind: fix cgroup v2 evacuation with `docker run --init`
2021-04-30 10:08:36 +02:00
Sebastiaan van Stijn
dd3275c5f9
Merge pull request #42182 from thaJeztah/fix_exec_start_err_handling
Fix error-handling in `daemon.ContainerExecStart()` and `daemon.getExecConfig()`
2021-04-29 21:33:19 +02:00
Brian Goff
ac99c869e5
Merge pull request #42181 from thaJeztah/fix_exec_apparmor_test
Fix panic in TestExecSetPlatformOpt, TestExecSetPlatformOptPrivileged
2021-04-29 12:25:27 -07:00
Akihiro Suda
bd61fdc65d
Merge pull request #42327 from awmirantis/bump-winio-4.19
Update Microsoft/go-winio to version 0.4.19
2021-04-30 00:00:43 +09:00
Akihiro Suda
42b1175eda
hack/dind: fix cgroup v2 evacuation with docker run --init
Evacuate all the processes in `/sys/fs/cgroup/cgroup.procs`, not just PID 1.

Before:
```console
$ docker run --rm --privileged --init $(docker build -q .) cat /sys/fs/cgroup/cgroup.subtree_control
sed: couldn't flush stdout: Device or resource busy
```

After:
```console
$ docker run --rm --privileged --init $(docker build -q .) cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma
```

Fix docker-library/docker issue 308

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-04-28 13:30:10 +09:00
John Howard
bac9062496
Merge pull request #42326 from thaJeztah/jhoward_alumni
Add John Howard to our alumni
2021-04-26 19:22:46 +01:00
Adam Williams
42a485f7fa Update Microsoft/go-winio to version 0.4.19
Signed-off-by: Adam Williams <awilliams@mirantis.com>
2021-04-26 10:02:58 -07:00
Sebastiaan van Stijn
46b27a9f9a
Merge pull request #42309 from lowenna/jhowardmsft-lowenna-maintainer
Remove myself (lowenna aka jhowardmsft) from maintainers
2021-04-26 17:51:26 +02:00
Sebastiaan van Stijn
b6a05b7371
Add John Howard to our alumni
Thank you John!!

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-26 14:57:37 +02:00
Tianon Gravi
e0170da0dc
Merge pull request #42315 from AkihiroSuda/whichless
dockerd-rootless.sh: use `command -v` instead of `which`
2021-04-23 11:32:57 -07:00
Akihiro Suda
e928692c69
dockerd-rootless.sh: use command -v instead of which
`which` binary is often missing

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-04-23 14:47:18 +09:00
Sebastiaan van Stijn
6d1eceb509
Fix panic in TestExecSetPlatformOpt, TestExecSetPlatformOptPrivileged
These tests would panic;

- in WithRLimits(), because HostConfig was not set;
  470ae8422f/daemon/oci_linux.go (L46-L47)
- in daemon.mergeUlimits(), because daemon.configStore was not set;
  470ae8422f/daemon/oci_linux.go (L1069)

This panic was not discovered because the current version of runc/libcontainer that we vendor
would not always return false for `apparmor.IsEnabled()` when running docker-in-docker or if
`apparmor_parser` is not found. Starting with v1.0.0-rc93 of libcontainer, this is no longer
the case (changed in bfb4ea1b1b)

This patch;

- changes the tests to initialize Daemon.configStore and Container.HostConfig
- Combines TestExecSetPlatformOpt and TestExecSetPlatformOptPrivileged into a new test
  (TestExecSetPlatformOptAppArmor)
- Runs the test both if AppArmor is enabled and if not (in which case it tests
  that the container's AppArmor profile is left empty).
- Adds a FIXME comment for a possible bug in execSetPlatformOpts, which currently
  prefers custom profiles over "privileged".

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-23 00:39:39 +02:00
Brian Goff
369d0dc092
Merge pull request #42254 from thaJeztah/bump_containerd_protobuf 2021-04-22 14:16:04 -07:00
Sebastiaan van Stijn
7f0fb3eccf
Merge pull request #42164 from olljanat/tests/win-prepare-for-containerd
Prepare tests for Windows containerd support
2021-04-22 21:09:52 +02:00
Sebastiaan van Stijn
82ba9b47ca
Merge pull request #42310 from WhyNotHugo/patch-2
Remove needless check
2021-04-22 21:08:28 +02:00
Sebastiaan van Stijn
20bd03b7a8
Merge pull request #42272 from brettmilford/master
Fixes subvol delete on a non-btrfs volume
2021-04-22 20:49:22 +02:00
Olli Janatuinen
bffa730860 Prepare tests for Windows containerd support
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2021-04-22 10:50:00 +03:00
Sebastiaan van Stijn
1c7585a7da
vendor: github.com/docker/swarmkit 5a5494a9a7b408b790533a5e4e1cb43ca1c32aad
full diff: 17d8d4e4d8...5a5494a9a7

Updated version of SwarmKit with protos generated with gogo/protobuf v1.3.2
due to CVE-2021-3121 reported on gogo/protobuf version 1.3.1, CVE has been
fixed for version 1.3.2

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3121

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-20 23:46:35 +02:00
Sebastiaan van Stijn
2bef937507
vendor: github.com/coreos/etcd v3.3.25
full diff: https://github.com/coreos/etcd/compare/v3.3.12...v3.3.25

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-20 23:46:33 +02:00
Sebastiaan van Stijn
6202322b60
vendor: github.com/containerd/containerd 19ee068f93c91f7b9b2a858457f1af2cabc7bc06
full diff: 55eda46b22...19ee068f93

brings in updated protobufs, generated with gogo/protobuf v1.3.2

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-20 23:46:31 +02:00
Sebastiaan van Stijn
8b385fde57
vendor: github.com/gogo/protobuf v1.3.2
full diff: https://github.com/gogo/protobuf/compare/v1.3.1...v1.3.2

bump version 1.3.2 for gogo/protobuf due to CVE-2021-3121 reported on
gogo/protobuf version 1.3.1, CVE has been fixed for version 1.3.2

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3121

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-20 23:46:27 +02:00
Tianon Gravi
72fef53cec
Merge pull request #42270 from cpuguy83/bump_hcsshim
Bump hcsshim to get some fixes.
2021-04-20 14:42:29 -07:00
Hugo Barrera
3980d0462d Remove needless check
Starting `dockerd-rootless.sh` checks that `$HOME` is writeable, but does not
require it to be so.
Make the check more precise, and check that it actually exists and is a
directory.

Signed-off-by: Hugo Osvaldo Barrera <hugo@barrera.io>
2021-04-20 21:30:26 +02:00
John Howard
825e9855b7 Remove lowenna (aka jhowardmsft) from maintainters
Signed-off-by: John Howard <github@lowenna.com>
2021-04-20 19:26:51 +01:00
Tianon Gravi
bcdb3c7368
Merge pull request #42293 from AkihiroSuda/rootlesskit-0.14.2
bump up rootlesskit to v0.14.2 (Fix `Timed out proxy starting the userland proxy.` error with `DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns`)
2021-04-19 11:29:02 -07:00
Akihiro Suda
3cd025bc67
bump up rootlesskit to v0.14.2
Fix `Timed out proxy starting the userland proxy.` error with `DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns`.
(https://github.com/rootless-containers/rootlesskit/issues/250)

Full changes: https://github.com/rootless-containers/rootlesskit/compare/v0.14.1...v0.14.2

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-04-19 16:41:25 +09:00
Brian Goff
8eb947c5b1
Merge pull request #42296 from thaJeztah/update_yamllint
Dockerfile: update yamllint to v1.26.1 to fix build
2021-04-15 15:13:34 -07:00
Sebastiaan van Stijn
c35cefb489
Dockerfile: update yamllint to v1.26.1 to fix build
Installation of yamllint started failing, on non-amd64 builds, which could
be if the version we were using wasn't specific enough about a dependency
to install.

    copying Cython/Utility/CppSupport.cpp -> build/lib.linux-aarch64-3.7/Cython/Utility
    running build_ext
    building 'Cython.Plex.Scanners' extension
    creating build/temp.linux-aarch64-3.7
    creating build/temp.linux-aarch64-3.7/tmp
    creating build/temp.linux-aarch64-3.7/tmp/pip-install-jasgbmp7
    creating build/temp.linux-aarch64-3.7/tmp/pip-install-jasgbmp7/Cython
    creating build/temp.linux-aarch64-3.7/tmp/pip-install-jasgbmp7/Cython/Cython
    creating build/temp.linux-aarch64-3.7/tmp/pip-install-jasgbmp7/Cython/Cython/Plex
    aarch64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.7m -c /tmp/pip-install-jasgbmp7/Cython/Cython/Plex/Scanners.c -o build/temp.linux-aarch64-3.7/tmp/pip-install-jasgbmp7/Cython/Cython/Plex/Scanners.o
         /tmp/pip-install-jasgbmp7/Cython/Cython/Plex/Scanners.c:21:10: fatal error: Python.h: No such file or directory
          #include "Python.h"
                   ^~~~~~~~~~
         compilation terminated.
         error: command 'aarch64-linux-gnu-gcc' failed with exit status 1

         ----------------------------------------
     Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jasgbmp7/Cython/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-if5qclwe/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-_dtiuyfw --compile" failed with error code 1 in /tmp/pip-install-jasgbmp7/Cython/

      ----------------------------------------
    Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_dtiuyfw --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Cython" failed with error code 1 in None
    #22 ERROR: executor failed running [/bin/sh -c pip3 install yamllint==1.16.0]: exit code: 1

Trying if updating to the latest version fixes this.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-15 21:30:10 +02:00
Brian Goff
225e046d9d Error string match: do not match command path
Whether or not the command path is in the error message is a an
implementation detail.
For example, on Windows the only reason this ever matched was because it
dumped the entire container config into the error message, but this had
nothing to do with the actual error.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2021-04-14 23:03:18 +00:00
Brian Goff
6110ba3d7c
Merge pull request #41586 from sparrc/stop-refactor
Fix hung docker stop if stop signal and daemon.Kill both fail
2021-04-14 13:33:14 -07:00
Brian Goff
99d3c07c1b
Merge pull request #42273 from mrostecki/btrfs-fix-quota
btrfs: Do not disable quota on cleanup
2021-04-14 13:27:49 -07:00
Cam
8e362b75cb
docker daemon container stop refactor
this refactors the Stop command to fix a few issues and behaviors that
dont seem completely correct:

1. first it fixes a situation where stop could hang forever (#41579)
2. fixes a behavior where if sending the
stop signal failed, then the code directly sends a -9 signal. If that
fails, it returns without waiting for the process to exit or going
through the full docker kill codepath.
3. fixes a behavior where if sending the stop signal failed, then the
code sends a -9 signal. If that succeeds, then we still go through the
same stop waiting process, and may even go through the docker kill path
again, even though we've already sent a -9.
4. fixes a behavior where the code would wait the full 30 seconds after
sending a stop signal, even if we already know the stop signal failed.

fixes #41579

Signed-off-by: Cam <gh@sparr.email>
2021-04-13 09:53:00 -07:00
Michal Rostecki
1ec689c4c2 btrfs: Do not disable quota on cleanup
Before this change, cleanup of the btrfs driver (occuring on each daemon
shutdown) resulted in disabling quotas. It was done with an assumption
that quotas can be enabled or disabled on a subvolume level, which is
not true - enabling or disabling quota is always done on a filesystem
level.

That was leading to disabling quota on btrfs filesystems on each daemon
shutdown.

This change fixes that behavior and removes misleading `subvol` prefix
from functions and methods which set up quota (on a filesystem level).

Fixes: #34593
Fixes: 401c8d1767 ("Add disk quota support for btrfs")
Signed-off-by: Michal Rostecki <mrostecki@opensuse.org>
2021-04-13 16:23:39 +01:00
Tibor Vass
68bec0fcf7
Merge pull request #42276 from thaJeztah/apparmor_detect_fix
Use containerd's apparmor package to detect if apparmor can be used
2021-04-09 16:09:54 -07:00
Sebastiaan van Stijn
1760f8bb7c
Merge pull request #39846 from thaJeztah/jenkinsfile_add_windows_1903
Jenkinsfile: add stage for Windows 2022 (SAC)
2021-04-09 10:13:00 +02:00
Brett Milford
a0d1a1f78c Update contrib/nuke-graph-directory.sh
Signed-off-by: Brett Milford <brettmilford@gmail.com>
2021-04-09 10:05:35 +10:00
Sebastiaan van Stijn
8f4b3b0ad4
pkg/archive temporarily skip some tests on Windowsd V19H1 (1903) and up
These tests fail, possibly due to changes in the kernel. Temporarily skipping
these tests, so that we at least have some coverage on these windows versions
in this repo, and we can look into this specific issue separately.;

    === FAIL: github.com/docker/docker/pkg/archive TestChangesDirsEmpty (0.21s)
        changes_test.go:261: Reported changes for identical dirs: [{\dirSymlink C}]

    === FAIL: github.com/docker/docker/pkg/archive TestChangesDirsMutated (0.14s)
        changes_test.go:391: unexpected change "C \\dirSymlink" "\\dirnew"

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-09 01:07:27 +02:00
Sebastiaan van Stijn
be95eae6d2
Merge pull request #41999 from diakovliev/fix_update_sync
Fix for lack of synchronization in daemon/update.go
2021-04-09 00:27:43 +02:00
Sebastiaan van Stijn
b9f127276f
Merge pull request #42268 from feliperuhland/fix-capadd-changelog
Fix typo on the v1.41 changelog
2021-04-08 21:57:56 +02:00
Sebastiaan van Stijn
b5f0096584
Jenkinsfile: temporarily use insider build for Windows 2022 (SAC)
Images for Windows 2022 (SAC) are not yet available, so using insider builds
in the meantime; mcr.microsoft.com/windows/servercore/insider:10.0.20295.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-08 21:25:35 +02:00
Sebastiaan van Stijn
e13d41db1b
Jenkinsfile: update to Windows 2022
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-08 21:25:30 +02:00
Sebastiaan van Stijn
0100a9ebc1
Jenkinsfile: windows-1903: send junit.xml results
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-08 21:24:08 +02:00
Sebastiaan van Stijn
fa194ec86c
Jenkinsfile: add stage for Windows 1903 (SAC)
This adds a stage to test against the current SAC (Semi Annual Channel),
which allows us to catch possible regressions on upcoming LTS versions.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-08 21:22:20 +02:00
Sebastiaan van Stijn
1cb7ee4cd9
Merge pull request #42246 from thaJeztah/replace_toml
Replace BurntSushi/toml with pelletier/go-toml
2021-04-08 21:13:11 +02:00
Sebastiaan van Stijn
2834f842ee
Use containerd's apparmor package to detect if apparmor can be used
The runc/libcontainer apparmor package on master no longer checks if apparmor_parser
is enabled, or if we are running docker-in-docker.

While those checks are not relevant to runc (as it doesn't load the profile), these
checks _are_ relevant to us (and containerd). So switching to use the containerd
apparmor package, which does include the needed checks.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-08 20:22:08 +02:00
Sebastiaan van Stijn
42ef2c5036
vendor: github.com/containerd/containerd 55eda46b22f985cde99b599e469ff9c13994bf68
This patch picks the first commit in containerd that exports the AppArmor package
functions to keep the vendor diff small (there are some updates to that package
after this, but those will be included in other patches).

full diff: fbf1a72de7...55eda46b22

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-08 20:22:06 +02:00