Previously, the API server configuration would be initialized and
validated when starting the API. Because of this, invalid configuration
(e.g. missing or invalid TLS certificates) would not be detected
when using `dockerd --validate`.
This patch moves creation of the validation earlier, so that it's
validated as part of `dockerd --validate`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously, hosts were de-duplicated and normalized when starting
the API server (in `loadListeners()`), which meant that errors could
occur in that step (but not detected when using `dockerd --validate`),
as well as the list of hosts in the config not matching what would
actually be used (i.e., if duplicates were present).
This patch extracts the de-duplicating to a separate function, and
executes it as part of loading the daemon configuration, so that we
can fail early.
Moving this code also showed that some of this validation depended
on `newAPIServerConfig()` modifying the configuration (adding an
empty host if none was set) in order to have the parsing set a
default. This code was moved elsewhere, but a TODO comment added
as this logic is somewhat sketchy.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- un-export `daemonOptions.InstallFlags()`; `daemonOptions` itself isn't exported,
not exported, and `InstallFlags()` isn't matching any interface and only used
internally.
- un-export `daemonOptions.SetDefaultOptions()` and remove the `flags` argument
as we were passing `daemonOptions.flags` as argument on a method attached to
`daemonOptions`, which was somewhat backwards. While at it, also removing an
intermediate variable that wasn't needed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Log-level validation was previously performed when configuring the daemon-logs;
this moves the validation to config.Validate() so that we can catch invalid
settings when running dockerd --validate.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Logging boolean was unconditionally set to true and ignored in all locations,
except for enabling the debugging middleware, which was also gated by the active
logrus logging level.
While it could make sense to have a Loglevel option configured on the API server,
we don't have this currently, and to make that actually useful, that config would
need to be tollerated by all locations that produce logs (which isn't the case
either).
Looking at the history of this option; a boolean to disable logging was originally
added in commit c423a790d6, which hard-coded it to
"disabled" in a test, and "enabled" for the API server outside of tests (before
that commit, logging was always enabled).
02ddaad5d9 and 5c42b2b512
changed the hard-coded values to be configurable through a `Logging` env-var (env-
vars were used _internally_ at the time to pass on options), which later became
a configuration struct in a0bf80fe03.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Perform the validation when the daemon starts instead of performing these
validations for each individual container, so that we can fail early.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The daemon can print the proxy configuration as part of error-messages,
and when reloading the daemon configuration (SIGHUP). Make sure that
the configuration is sanitized before printing.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This allows configuring the daemon's proxy server through the daemon.json con-
figuration file or command-line flags configuration file, in addition to the
existing option (through environment variables).
Configuring environment variables on Windows to configure a service is more
complicated than on Linux, and adding alternatives for this to the daemon con-
figuration makes the configuration more transparent and easier to use.
The configuration as set through command-line flags or through the daemon.json
configuration file takes precedence over env-vars in the daemon's environment,
which allows the daemon to use a different proxy. If both command-line flags
and a daemon.json configuration option is set, an error is produced when starting
the daemon.
Note that this configuration is not "live reloadable" due to Golang's use of
`sync.Once()` for proxy configuration, which means that changing the proxy
configuration requires a restart of the daemon (reload / SIGHUP will not update
the configuration.
With this patch:
cat /etc/docker/daemon.json
{
"http-proxy": "http://proxytest.example.com:80",
"https-proxy": "https://proxytest.example.com:443"
}
docker pull busybox
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": proxyconnect tcp: dial tcp: lookup proxytest.example.com on 127.0.0.11:53: no such host
docker build .
Sending build context to Docker daemon 89.28MB
Step 1/3 : FROM golang:1.16-alpine AS base
Get "https://registry-1.docker.io/v2/": proxyconnect tcp: dial tcp: lookup proxytest.example.com on 127.0.0.11:53: no such host
Integration tests were added to test the behavior:
- verify that the configuration through all means are used (env-var,
command-line flags, damon.json), and used in the expected order of
preference.
- verify that conflicting options produce an error.
Signed-off-by: Anca Iordache <anca.iordache@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
It's the only location where this is used, and it's quite specific
to dockerd (not really a reusable function for external use), so
moving it into that package.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The "quiet" argument was only used in a single place (at daemon startup), and
every other use had to pass "false" to prevent this function from logging
warnings.
Now that SysInfo contains the warnings that occurred when collecting the
system information, we can make leave it up to the caller to use those
warnings (and log them if wanted).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fixes#36911
If config file is invalid we'll exit anyhow, so this just prevents
the daemon from starting if the configuration is fine.
Mainly useful for making config changes and restarting the daemon
iff the config is valid.
Signed-off-by: Rich Horwood <rjhorwood@apple.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Anca Iordache <anca.iordache@docker.com>
The LCOW implementation in dockerd has been deprecated in favor of re-implementation
in containerd (in progress). Microsoft started removing the LCOW V1 code from the
build dependencies we use in Microsoft/opengcs (soon to be part of Microsoft/hcshhim),
which means that we need to start removing this code.
This first step removes the lcow graphdriver, the LCOW initialization code, and
some LCOW-related utilities.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signal systemd when we start shutting down to complement the "READY" notify
that was originally implemented in 97088ebef7
From [sd_notify(3)](https://www.freedesktop.org/software/systemd/man/sd_notify.html#STOPPING=1)
> STOPPING=1
> Tells the service manager that the service is beginning its shutdown. This is useful
> to allow the service manager to track the service's internal state, and present it to
> the user.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
People keep doing this and getting pwned because they accidentally left
it exposed to the internet.
The warning about doing this has been there forever.
This introduces a sleep after warning.
To disable the extra sleep users must explicitly specify `--tls=false`
or `--tlsverify=false`
Warning also specifies this sleep will be removed in the next release
where the flag will be required if running unauthenticated.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This patch adds a new "prune" event type to indicate that pruning of a resource
type completed.
This event-type can be used on systems that want to perform actions after
resources have been cleaned up. For example, Docker Desktop performs an fstrim
after resources are deleted (https://github.com/linuxkit/linuxkit/tree/v0.7/pkg/trim-after-delete).
While the current (remove, destroy) events can provide information on _most_
resources, there is currently no event triggered after the BuildKit build-cache
is cleaned.
Prune events have a `reclaimed` attribute, indicating the amount of space that
was reclaimed (in bytes). The attribute can be used, for example, to use as a
threshold for performing fstrim actions. Reclaimed space for `network` events
will always be 0, but the field is added to be consistent with prune events for
other resources.
To test this patch:
Create some resources:
for i in foo bar baz; do \
docker network create network_$i \
&& docker volume create volume_$i \
&& docker run -d --name container_$i -v volume_$i:/volume busybox sh -c 'truncate -s 5M somefile; truncate -s 5M /volume/file' \
&& docker tag busybox:latest image_$i; \
done;
docker pull alpine
docker pull nginx:alpine
echo -e "FROM busybox\nRUN truncate -s 50M bigfile" | DOCKER_BUILDKIT=1 docker build -
Start listening for "prune" events in another shell:
docker events --filter event=prune
Prune containers, networks, volumes, and build-cache:
docker system prune -af --volumes
See the events that are returned:
docker events --filter event=prune
2020-07-25T12:12:09.268491000Z container prune (reclaimed=15728640)
2020-07-25T12:12:09.447890400Z network prune (reclaimed=0)
2020-07-25T12:12:09.452323000Z volume prune (reclaimed=15728640)
2020-07-25T12:12:09.517236200Z image prune (reclaimed=21568540)
2020-07-25T12:12:09.566662600Z builder prune (reclaimed=52428841)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Close#40484
Note that the support for cgroup v2 isn't ready for production yet,
regardless to rootful or rootless.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The namespaces com.docker.*, io.docker.*, org.dockerproject.*
have been documented to be reserved for Docker's internal use.
Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com>
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
This feature was used by docker build --stream and it was kept experimental.
Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.
Signed-off-by: Tibor Vass <tibor@docker.com>
Fix TODO: move into startMetricsServer()
Fix errors.Wrap return nil when passed err is nil
Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com>
Signed-off-by: HuanHuan Ye <logindaveye@gmail.com>
also renamed the non-windows variant of this file to be
consistent with other files in this package
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This hack was added to fix a compatibility with clients
that were built using Go 1.5 and older (added in 3d6f5984f5)
This hack causes some problems with current clients; with Go 1.5 and older
no longer being supported for some time, and being several years old, it
should now be ok to remove this hack altogether.
People using tools that are built with those versions of Go wouldn't have
updated those for years, and are probably out of date anyway; that's not
something we can continue taking into account.
This will affect docker clients (the docker cli) for docker 1.12 and older.
Those versions have reached EOL a long time ago (and have known unpatched
vulnerabilities), so should no longer be used anyway, but We should add
a nebtuib in the release notes, just in case someone, somewhere, still
has such old tools.
For those affected, using a more recent client (and if needed, setting
the DOCKER_API_VERSION environment variable to the needed API version)
should provide a way out.
This reverts the changes originally made in; #22000 and #22888,
which were to address #20865.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before this change:
dockerd -H unix:///run/docker.sock -H unix:///run/docker.sock -H unix:///run/docker.sock
...
INFO[2019-07-13T00:02:36.195090937Z] Daemon has completed initialization
INFO[2019-07-13T00:02:36.215940441Z] API listen on /run/docker.sock
INFO[2019-07-13T00:02:36.215933172Z] API listen on /run/docker.sock
INFO[2019-07-13T00:02:36.215990566Z] API listen on /run/docker.sock
After this change:
dockerd -H unix:///run/docker.sock -H unix:///run/docker.sock -H unix:///run/docker.sock
...
INFO[2019-07-13T00:01:37.533579874Z] Daemon has completed initialization
INFO[2019-07-13T00:01:37.567045771Z] API listen on /run/docker.sock
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This reverts commit 98fc09128b in order to
keep registry v2 schema1 handling and libtrust-key-based engine ID.
Because registry v2 schema1 was not officially deprecated and
registries are still relying on it, this patch puts its logic back.
However, registry v1 relics are not added back since v1 logic has been
removed a while ago.
This also fixes an engine upgrade issue in a swarm cluster. It was relying
on the Engine ID to be the same upon upgrade, but the mentioned commit
modified the logic to use UUID and from a different file.
Since the libtrust key is always needed to support v2 schema1 pushes,
that the old engine ID is based on the libtrust key, and that the engine ID
needs to be conserved across upgrades, adding a UUID-based engine ID logic
seems to add more complexity than it solves the problems.
Hence reverting the engine ID changes as well.
Signed-off-by: Tibor Vass <tibor@docker.com>
The `--rootless` flag had a couple of issues:
* #38702: euid=0, $USER="root" but no access to cgroup ("rootful" Docker in rootless Docker)
* #39009: euid=0 but $USER="docker" (rootful boot2docker)
To fix#38702, XDG dirs are ignored as in rootful Docker, unless the
dockerd is directly running under RootlessKit namespaces.
RootlessKit detection is implemented by checking whether `$ROOTLESSKIT_STATE_DIR` is set.
To fix#39009, the non-robust `$USER` check is now completely removed.
The entire logic can be illustrated as follows:
```
withRootlessKit := getenv("ROOTLESSKIT_STATE_DIR")
rootlessMode := withRootlessKit || cliFlag("--rootless")
honorXDG := withRootlessKit
useRootlessKitDockerProxy := withRootlessKit
removeCgroupSpec := rootlessMode
adjustOOMScoreAdj := rootlessMode
```
Close#39024Fix#38702#39009
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
`defer r.WaitTimeout(10s)` was in a wrong place and had caused the
daemon to hang for 10 seconds.
Fix#39025
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
