Commit graph

188 commits

Author SHA1 Message Date
Sebastiaan van Stijn
cff4f20c44
migrate to github.com/containerd/log v0.1.0
The github.com/containerd/containerd/log package was moved to a separate
module, which will also be used by upcoming (patch) releases of containerd.

This patch moves our own uses of the package to use the new module.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-11 17:52:23 +02:00
Sebastiaan van Stijn
3614749b55
Merge pull request #45966 from neersighted/buildkit_0.12
Update to BuildKit 0.12
2023-09-22 02:13:15 +02:00
Bjorn Neergaard
2246297ae6
builder-next: adopt new wrapped types
With BuildKit 0.12, some existing types are now required to be wrapped
by new types:

* containerd's LeaseManager and ContentStore have to be a
  (namespace-aware) BuildKit type since f044e0a946
* BuildKit's solver.CacheManager is used instead of
  bboltstorage.CacheKeyStorage since 2b30693409
* The MaxAge config field is a bkconfig.Duration since e06c96274f

Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-09-21 14:18:48 -06:00
Bjorn Neergaard
0e80073e01
daemon: strongly type containerd log.OutputFormat
This type was introduced in
0a79e67e4f

Make use of it throughout our log-format handling code, and convert back
to a string before we pass it to the containerd client.

Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-09-21 05:40:17 -06:00
Sebastiaan van Stijn
bd523abd44
remove more direct uses of logrus
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-15 20:12:27 +02:00
Sebastiaan van Stijn
1ffbbe798a
vendor: github.com/containerd/containerd v1.6.24
full diff: https://github.com/containerd/containerd/compare/v1.6.22...v1.6.24

v1.6.24 release notes:

full diff: https://github.com/containerd/containerd/compare/v1.6.23...v1.6.24

The twenty-fourth patch release for containerd 1.6 contains various fixes
and updates.

Notable Updates

- CRI: fix leaked shim caused by high IO pressure
- Update to go1.20.8
- Update runc to v1.1.9
- Backport: add configurable mount options to overlay snapshotter
- log: cleanups and improvements to decouple more from logrus

v1.6.23 release notes:

full diff: https://github.com/containerd/containerd/compare/v1.6.22...v1.6.23

The twenty-third patch release for containerd 1.6 contains various fixes
and updates.

Notable Updates

- Add stable ABI support in windows platform matcher + update hcsshim tag
- cri: Don't use rel path for image volumes
- Upgrade GitHub actions packages in release workflow
- update to go1.19.12
- backport: ro option for userxattr mount check + cherry-pick: Fix ro mount option being passed

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-15 17:34:58 +02:00
Sebastiaan van Stijn
ce01d779df
daemon: format code with gofumpt
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-24 17:59:22 +02:00
Evan Lezar
3b71197fb8 Set default CDI spec dirs after parsing args
Signed-off-by: Evan Lezar <elezar@nvidia.com>
2023-08-03 12:21:44 +02:00
Cory Snider
0c2699da27
Merge pull request #45737 from pkwarren/pkw/issue-44940-dockerd-json-logs
Update dockerd to support JSON logging format
2023-07-13 19:00:31 -04:00
Sebastiaan van Stijn
c90229ed9a
api/types: move system info types to api/types/system
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-07 13:01:36 +02:00
Sebastiaan van Stijn
b8220f5d0d
daemon/config: move MTU to BridgeConfig
This option is only used for the default bridge network; let's move the
field to that struct to make it clearer what it's used for.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-05 14:43:35 +02:00
Sebastiaan van Stijn
11abd0742e
dockerd: "--mtu": update description, hide on Windows and warn if set
The --mtu option is only used for the default "bridge" network on Linux.
On Windows, the flag is available, but ignored. As this option has been
available for a long time, and was always silently ignored, deprecating
or removing it would be a breaking change (and perhaps it's possible to
support it in future).

This patch:

- hides the option on Windows binaries
- logs a warning if the option is set to any non-zero value other than
  the default on a Windows binary

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-05 14:43:27 +02:00
Sebastiaan van Stijn
13472c6cf1
daemon/config: remove unused CommonConfig.Context
This field was added in f0e6e135a8, and
from that change I suspect it was intended to store the default SELinux
mount-labels to be set on containers.

However, it was never used, so let's remove it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-30 16:44:53 +02:00
Sebastiaan van Stijn
8d923c60b9
daemon/config: format code with gofumpt
Formatting the code with https://github.com/mvdan/gofumpt

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-29 00:31:11 +02:00
Philip K. Warren
a08abec9f8 Update dockerd to support JSON logging format
Update docker to support a '--log-format' option, which accepts either
'text' (default) or 'json'. Propagate the log format to containerd as
well, to ensure that everything will be logged consistently.

Signed-off-by: Philip K. Warren <pkwarren@gmail.com>
2023-06-28 12:46:28 -05:00
Brian Goff
74da6a6363 Switch all logging to use containerd log pkg
This unifies our logging and allows us to propagate logging and trace
contexts together.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-06-24 00:23:44 +00:00
Cory Snider
0f6eeecac0 daemon: consolidate runtimes config validation
The daemon has made a habit of mutating the DefaultRuntime and Runtimes
values in the Config struct to merge defaults. This would be fine if it
was a part of the regular configuration loading and merging process,
as is done with other config options. The trouble is it does so in
surprising places, such as in functions with 'verify' or 'validate' in
their name. It has been necessary in order to validate that the user has
not defined a custom runtime named "runc" which would shadow the
built-in runtime of the same name. Other daemon code depends on the
runtime named "runc" always being defined in the config, but merging it
with the user config at the same time as the other defaults are merged
would trip the validation. The root of the issue is that the daemon has
used the same config values for both validating the daemon runtime
configuration as supplied by the user and for keeping track of which
runtimes have been set up by the daemon. Now that a completely separate
value is used for the latter purpose, surprising contortions are no
longer required to make the validation work as intended.

Consolidate the validation of the runtimes config and merging of the
built-in runtimes into the daemon.setupRuntimes() function. Set the
result of merging the built-in runtimes config and default default
runtime on the returned runtimes struct, without back-propagating it
onto the config.Config argument.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-06-01 14:45:25 -04:00
Cory Snider
d222bf097c daemon: reload runtimes w/o breaking containers
The existing runtimes reload logic went to great lengths to replace the
directory containing runtime wrapper scripts as atomically as possible
within the limitations of the Linux filesystem ABI. Trouble is,
atomically swapping the wrapper scripts directory solves the wrong
problem! The runtime configuration is "locked in" when a container is
started, including the path to the runC binary. If a container is
started with a runtime which requires a daemon-managed wrapper script
and then the daemon is reloaded with a config which no longer requires
the wrapper script (i.e. some args -> no args, or the runtime is dropped
from the config), that container would become unmanageable. Any attempts
to stop, exec or otherwise perform lifecycle management operations on
the container are likely to fail due to the wrapper script no longer
existing at its original path.

Atomically swapping the wrapper scripts is also incompatible with the
read-copy-update paradigm for reloading configuration. A handler in the
daemon could retain a reference to the pre-reload configuration for an
indeterminate amount of time after the daemon configuration has been
reloaded and updated. It is possible for the daemon to attempt to start
a container using a deleted wrapper script if a request to run a
container races a reload.

Solve the problem of deleting referenced wrapper scripts by ensuring
that all wrapper scripts are *immutable* for the lifetime of the daemon
process. Any given runtime wrapper script must always exist with the
same contents, no matter how many times the daemon config is reloaded,
or what changes are made to the config. This is accomplished by using
everyone's favourite design pattern: content-addressable storage. Each
wrapper script file name is suffixed with the SHA-256 digest of its
contents to (probabilistically) guarantee immutability without needing
any concurrency control. Stale runtime wrapper scripts are only cleaned
up on the next daemon restart.

Split the derived runtimes configuration from the user-supplied
configuration to have a place to store derived state without mutating
the user-supplied configuration or exposing daemon internals in API
struct types. Hold the derived state and the user-supplied configuration
in a single struct value so that they can be updated as an atomic unit.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-06-01 14:45:25 -04:00
Cory Snider
0b592467d9 daemon: read-copy-update the daemon config
Ensure data-race-free access to the daemon configuration without
locking by mutating a deep copy of the config and atomically storing
a pointer to the copy into the daemon-wide configStore value. Any
operations which need to read from the daemon config must capture the
configStore value only once and pass it around to guarantee a consistent
view of the config.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-06-01 14:45:24 -04:00
Evan Lezar
7ec9561a77 Add support for CDI devices to docker daemon under linux
These changes add basic CDI integration to the docker daemon.

A cdi driver is added to handle cdi device requests. This
is gated by an experimental feature flag and is only supported on linux

This change also adds a CDISpecDirs (cdi-spec-dirs) option to the config.
This allows the default values of `/etc/cdi`, /var/run/cdi` to be overridden
which is useful for testing.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2023-05-16 17:07:57 +02:00
Sebastiaan van Stijn
fb96b94ed0
daemon: remove handling for deprecated "oom-score-adjust", and produce error
This option was deprecated in 5a922dc162, which
is part of the v24.0.0 release, so we can remove it from master.

This patch;

- adds a check to ValidatePlatformConfig, and produces a fatal error
  if oom-score-adjust is set
- removes the deprecated libcontainerd/supervisor.WithOOMScore
- removes the warning from docker info

With this patch:

    dockerd --oom-score-adjust=-500 --validate
    Flag --oom-score-adjust has been deprecated, and will be removed in the next release.
    unable to configure the Docker daemon with file /etc/docker/daemon.json: merged configuration validation from file and command line flags failed: DEPRECATED: The "oom-score-adjust" config parameter and the dockerd "--oom-score-adjust" options have been removed.

And when using `daemon.json`:

    dockerd --validate
    unable to configure the Docker daemon with file /etc/docker/daemon.json: merged configuration validation from file and command line flags failed: DEPRECATED: The "oom-score-adjust" config parameter and the dockerd "--oom-score-adjust" options have been removed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-06 16:36:17 +02:00
Sebastiaan van Stijn
5a922dc162
daemon: deprecate --oom-score-adjust for the daemon
The `oom-score-adjust` option was added in a894aec8d8,
to prevent the daemon from being OOM-killed before other processes. This
option was mostly added as a "convenience", as running the daemon as a
systemd unit was not yet common.

Having the daemon set its own limits is not best-practice, and something
better handled by the process-manager starting the daemon.

Commit cf7a5be0f2 fixed this option to allow
disabling it, and 2b8e68ef06 removed the default
score adjust.

This patch deprecates the option altogether, recommending users to set these
limits through the process manager used, such as the "OOMScoreAdjust" option
in systemd units.

With this patch:

    dockerd --oom-score-adjust=-500 --validate
    Flag --oom-score-adjust has been deprecated, and will be removed in the next release.
    configuration OK

    echo '{"oom-score-adjust":-500}' > /etc/docker/daemon.json
    dockerd
    INFO[2023-04-12T21:34:51.133389627Z] Starting up
    INFO[2023-04-12T21:34:51.135607544Z] containerd not running, starting managed containerd
    WARN[2023-04-12T21:34:51.135629086Z] DEPRECATED: The "oom-score-adjust" config parameter and the dockerd "--oom-score-adjust" option will be removed in the next release.

    docker info
    Client:
      Context:    default
      Debug Mode: false
    ...
    DEPRECATED: The "oom-score-adjust" config parameter and the dockerd "--oom-score-adjust" option will be removed in the next release

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-13 00:02:39 +02:00
Cory Snider
750218c5e5 d/config: fix default-network-opts in daemon.json
Prevent the daemon from erroring out if daemon.json contains default
network options for network drivers aside from bridge. Configuring
defaults for the bridge driver previously worked by coincidence because
the unrelated CLI flag '--bridge' exists.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-04-03 15:59:53 -04:00
Tianon Gravi
6caaa8cadc Prefer loading docker-init from an appropriate "libexec" directory
The `docker-init` binary is not intended to be a user-facing command, and as such it is more appropriate for it to be found in `/usr/libexec` (or similar) than in `PATH` (see the FHS, especially https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html and https://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA).

This adjusts the logic for using that configuration option to take this into account and appropriately search for `docker-init` (or the user's configured alternative) in these directories before falling back to the existing `PATH` lookup behavior.

This behavior _used_ to exist for the old `dockerinit` binary (of a similar name and used in a similar way but for an alternative purpose), but that behavior was removed in 4357ed4a73 when that older `dockerinit` was also removed.

Most of this reasoning _also_ applies to `docker-proxy` (and various `containerd-xxx` binaries such as the shims), but this change does not affect those.  It would be relatively straightforward to adapt `LookupInitPath` to be a more generic function such as `libexecLookupPath` or similar if we wanted to explore that.

See 14482589df/cli-plugins/manager/manager_unix.go for the related path list in the CLI which loads CLI plugins from a similar set of paths (with a similar rationale - plugin binaries are not typically intended to be run directly by users but rather invoked _via_ the CLI binary).

Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2023-03-24 14:25:12 -07:00
Akihiro Suda
e807ae4f2e
vendor: github.com/containerd/cgroups/v3 v3.0.1
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-03-08 20:15:17 +09:00
Sebastiaan van Stijn
2323f9deda
Merge pull request #43197 from dajudge/default-bridge-mtu
Introduce config option for default generic network options of newly created networks
2023-03-02 20:21:47 +01:00
Cory Snider
a9e7360775 daemon/config: remove AuthzMiddleware field
The authorization.Middleware contains a sync.Mutex field, making it
non-copyable. Remove one of the barriers to allowing deep copies of
config.Config values.

Inject the middleware into Daemon as a constructor argument instead.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-03-01 09:43:39 -05:00
Alex Stockinger
91c2b12205 Make default options for newly created networks configurable
Signed-off-by: Alex Stockinger <alex@atomicjar.com>
Co-authored-by: Sergei Egorov <bsideup@gmail.com>
Co-authored-by: Cory Snider <corhere@gmail.com>
2023-03-01 07:58:26 +01:00
Tonis Tiigi
666334bd48
vendor: update buildkit to v0.11.2
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-02-01 23:29:10 -08:00
Bjorn Neergaard
8dbc5df952
daemon/config: inform the user when the input JSON contains invalid UTF-8
This helps ensure that users are not surprised by unexpected tokens in
the JSON parser, or fallout later in the daemon.

Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
2023-01-10 17:27:21 -07:00
Bjorn Neergaard
d42495033e
daemon/config: support alternate (common) unicode encodings using a BOM
This is a pragmatic but impure choice, in order to better support the
default tools available on Windows Server, and reduce user confusion due
to otherwise inscrutable-to-the-uninitiated errors like the following:

> invalid character 'þ' looking for beginning of value
> invalid character 'ÿ' looking for beginning of value

While meaningful to those who are familiar with and are equipped to
diagnose encoding issues, these characters will be hidden when the file
is edited with a BOM-aware text editor, and further confuse the user.

Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
2023-01-10 15:49:15 -07:00
Bjorn Neergaard
1dcf7d5b03
daemon/config: clean up tests to use common helper
Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
2023-01-10 13:54:17 -07:00
Bjorn Neergaard
62227e1bba
Merge pull request #44733 from jg-public/fix-rootless-specspaths--T43111
Use user data path for plugin discovery in rootless mode
2023-01-09 17:32:11 -07:00
Bjorn Neergaard
cba3edbc37
daemon/config: group JSON preprocessing steps
Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
2023-01-09 13:17:32 -07:00
Bjorn Neergaard
bb19265ba8
daemon/config: ignore UTF-8 BOM in config JSON
[RFC 8259] allows for JSON implementations to optionally ignore a BOM
when it helps with interoperability; do so in Moby as Notepad (the only
text editor available out of the box in many versions of Windows Server)
insists on writing UTF-8 with a BOM.

  [RFC 8259]: https://tools.ietf.org/html/rfc8259#section-8.1

Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
2023-01-09 13:16:57 -07:00
Jan Garcia
6ab12ec8f4 rootless: move ./rootless to ./pkg/rootless
Signed-off-by: Jan Garcia <github-public@n-garcia.com>
2023-01-09 16:26:06 +01:00
Sebastiaan van Stijn
e19dff9285
daemon/config: remove deprecated RootDeprecated field
Since b58de39ca7, this option was now only used
to produce a fatal error when starting the daemon. That change is in the 23.0
release, so we can remove it from the master branch.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-15 10:44:32 +01:00
Sebastiaan van Stijn
32b299f63e
daemon/config: rename CommonTLSOptions to TLSOptions
This type was added in 677a6b3506, and named
"common", because at the time, the "docker" and "dockerd" (daemon) code
were still in the same repository, and shared this type. Renaming it, now
that's no longer the case.

As there are no external consumers of this type, I'm not adding an alias.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-15 10:44:24 +01:00
Sebastiaan van Stijn
5cdd6ab7cd
daemon/config: remove TrustKeyPath, and local utilities
Turned out that the loadOrCreateTrustKey() utility was doing exactly the
same as libtrust.LoadOrCreateTrustKey(), so making it a thin wrapped. I kept
the tests to verify the behavior, but we could remove them as we only need this
for our integration tests.

The storage location for the generated key was changed (again as we only need
this for some integration tests), so we can remove the TrustKeyPath from the
config.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 20:29:19 +01:00
Sebastiaan van Stijn
b529870558
daemon/config: use strings.Cut(), fix panic in BuilderGCFilter
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-16 15:15:05 +01:00
Cory Snider
ad4073edc1 daemon: fix docs for config-default constants
Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-28 15:52:57 -04:00
Sebastiaan van Stijn
56e64270f3
daemon: use strconv instead of fmt.Sprintf()
Also cleaning up some errors

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-08 17:41:39 +02:00
Sebastiaan van Stijn
d082bbcc05
Merge pull request #43978 from cpuguy83/default_runtime_shim
Allow containerd shim refs in default-runtime
2022-08-18 22:59:57 +02:00
Sebastiaan van Stijn
61e5cf083a
Merge pull request #43971 from thaJeztah/daemon_fix_hosts_validation_step1c3
daemon: use initialized config struct in more tests
2022-08-18 21:30:07 +02:00
Brian Goff
e6ee27a541 Allow containerd shim refs in default-runtime
Since runtimes can now just be containerd shims, we need to check if the
reference is possibly a containerd shim.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2022-08-18 18:41:03 +00:00
Sebastiaan van Stijn
b58de39ca7
daemon: complete the "--graph" / "-g" deprecation
The `-g` / `--graph` options were soft deprecated in favor of `--data-root` in
261ef1fa27 (v17.05.0) and at the time considered
to not be removed. However, with the move towards containerd snapshotters, having
these options around adds additional complexity to handle fallbacks for deprecated
(and hidden) flags, so completing the deprecation.

With this patch:

    dockerd --graph=/var/lib/docker --validate
    Flag --graph has been deprecated, Use --data-root instead
    unable to configure the Docker daemon with file /etc/docker/daemon.json: merged configuration validation from file and command line flags failed: the "graph" config file option is deprecated; use "data-root" instead

    mkdir -p /etc/docker
    echo '{"graph":"/var/lib/docker"}' > /etc/docker/daemon.json

    dockerd --validate
    unable to configure the Docker daemon with file /etc/docker/daemon.json: merged configuration validation from file and command line flags failed: the "graph" config file option is deprecated; use "data-root" instead

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-18 14:57:16 +02:00
Sebastiaan van Stijn
327c8fc52f
daemon/config: use initialized config in more tests
Makes sure that tests use a config struct that's more representative
to how it's used in the code.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-17 14:31:25 +02:00
Sebastiaan van Stijn
b28e66cf4f
daemon/config: New(): initialize config with platform-specific defaults
This centralizes more defaults, to be part of the config struct that's
created, instead of interweaving the defaults with other code in various
places.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-17 08:54:32 +02:00
Sebastiaan van Stijn
8925f735a1
daemon/config: fix GoDoc for StockRuntimeName
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-11 18:16:50 +02:00
Sebastiaan van Stijn
68bf777ece
daemon: remove deprecated fields and flags for cluster-advertise
Removes:

- CommonConfig.ClusterStore
- CommonConfig.ClusterOpts
- CommonConfig.ClusterAdvertise
- The deprecated --cluster-advertise, --cluster-store, and
  --cluster-store-opt flags

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-07-12 15:39:20 +02:00