Commit graph

278 commits

Author SHA1 Message Date
Sebastiaan van Stijn
2f003f7774
Merge pull request #40930 from thaJeztah/bump_cobra
vendor: spf13/cobra v1.0.0, spf13/pflag v1.0.5
2020-05-14 21:28:51 +02:00
Tibor Vass
298ba5b131
Merge pull request #40427 from thaJeztah/prometheus_remove_experimental
Do not require "experimental" for metrics API
2020-05-08 11:10:53 -07:00
Sebastiaan van Stijn
dc0a400738
vendor: spf13/cobra v1.0.0
full diff: https://github.com/spf13/cobra/compare/v0.0.3...v1.0.0

Notable Changes

- Fish completion (including support for Go custom completion)
- API (urgent): Rename BashCompDirectives to ShellCompDirectives
- Remove/replace SetOutput on Command - deprecated
- Custom completions coded in Go (instead of Bash)
- Partial Revert of 922
- Correct documentation for InOrStdin
- Apply formatting to templates
- Revert change so help is printed on stdout again
- Update md2man to v2.0.0
- update viper to v1.4.0
- Update cmd/root.go example in README.md

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-05-08 10:44:36 +02:00
Sebastiaan van Stijn
c3b3aedfa4
Merge pull request #40662 from AkihiroSuda/cgroup2-dockerinfo
cgroup2: implement `docker info`
2020-04-29 22:57:00 +02:00
Akihiro Suda
90f19fc375 cmd/dockerd: remove outdated "experimental" annotation
Follow up to #40759

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-04-29 11:06:11 +09:00
Sebastiaan van Stijn
8312004f41
remove uses of deprecated pkg/term
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-21 16:29:27 +02:00
Sebastiaan van Stijn
f337a8d21d
Do not require "experimental" for metrics API
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-20 22:19:00 +02:00
Brian Goff
130b0bc603
Merge pull request #40823 from thaJeztah/remove_distrib_uuid
daemon: remove distribution/uuid package
2020-04-16 20:59:58 -07:00
Akihiro Suda
f350b53241 cgroup2: implement docker info
ref: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-04-17 07:20:01 +09:00
Sebastiaan van Stijn
2376ebd3f2
fix --cluster-store-opt deprecation annotation
Commit 1470697b67 marked this flag
as deprecated, but inadvertedtly used the plural (which is used
for `daemon.json`, but not for the flag name.

This fixes the inccorect name of the flag.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-16 14:42:19 +02:00
Sebastiaan van Stijn
7400375526
daemon: remove distribution/uuid package
This appeared to be unused because we no longer generate
a uuid using this package.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-16 09:16:38 +02:00
Tonis Tiigi
0cdf6ba9c8 vendor: update buildkit to ae7ff174
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-04-14 08:26:07 -07:00
Akihiro Suda
5ca47f5179 rootless: graduate from experimental
Close #40484

Note that the support for cgroup v2 isn't ready for production yet,
regardless to rootful or rootless.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-04-07 00:59:15 +09:00
Akihiro Suda
9a82a9a8ea vendor containerd, BuildKit, protobuf, grpc, and golang.org/x
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-03-03 10:25:20 +09:00
Brian Goff
bc1c0c7a8a
Merge pull request #40510 from aiordache/moby_cluster_flags_deprecate
Deprecate '--cluster-xx' options and add warning
2020-02-27 11:25:31 -08:00
Anca Iordache
f7c4be86c6 fixed deprecation message
Signed-off-by: Anca Iordache <anca.iordache@docker.com>
2020-02-19 11:22:20 +01:00
Anca Iordache
1470697b67 Deprecate '--cluster-xx' options and add warning
Co-authored-by: Yves Brissaud <yves.brissaud@gmail.com>

Signed-off-by: Anca Iordache <anca.iordache@docker.com>
2020-02-12 18:33:23 +01:00
Jintao Zhang
35d6c1870f enforce reserve internal labels.
The namespaces com.docker.*, io.docker.*, org.dockerproject.*
have been documented to be reserved for Docker's internal use.

Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com>
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
2020-02-12 12:03:35 +08:00
Sebastiaan van Stijn
9f0b3f5609
bump gotest.tools v3.0.1 for compatibility with Go 1.14
full diff: https://github.com/gotestyourself/gotest.tools/compare/v2.3.0...v3.0.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-11 00:06:42 +01:00
Sebastiaan van Stijn
ca20bc4214
Merge pull request #40007 from arkodg/add-host-docker-internal
Support host.docker.internal in dockerd on Linux
2020-01-27 13:42:26 +01:00
Arko Dasgupta
92e809a680 Support host.docker.internal in dockerd on Linux
Docker Desktop (on MAC and Windows hosts) allows containers
running inside a Linux VM to connect to the host using
the host.docker.internal DNS name, which is implemented by
VPNkit (DNS proxy on the host)

This PR allows containers to connect to Linux hosts
by appending a special string "host-gateway" to --add-host
e.g. "--add-host=host.docker.internal:host-gateway" which adds
host.docker.internal DNS entry in /etc/hosts and maps it to host-gateway-ip

This PR also add a daemon flag call host-gateway-ip which defaults to
the default bridge IP
Docker Desktop will need to set this field to the Host Proxy IP
so DNS requests for host.docker.internal can be routed to VPNkit

Addresses: https://github.com/docker/for-linux/issues/264

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2020-01-22 13:30:00 -08:00
Akihiro Suda
19baeaca26 cgroup2: enable cgroup namespace by default
For cgroup v1, we were unable to change the default because of
compatibility issue.

For cgroup v2, we should change the default right now because switching
to cgroup v2 is already breaking change.

See also containers/libpod#4363 containers/libpod#4374

Privileged containers also use cgroupns=private by default.
https://github.com/containers/libpod/pull/4374#issuecomment-549776387

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-01-01 02:58:40 +09:00
Sebastiaan van Stijn
6b91ceff74
Use hcsshim osversion package for Windows versions
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-22 02:53:00 +02:00
Tibor Vass
6ca3ec88ae builder: remove legacy build's session handling
This feature was used by docker build --stream and it was kept experimental.

Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-10-02 20:29:15 +00:00
Lukas Heeren
ce61a1ed98
Adding ability to change max download attempts
Moby works perfectly when you are in a situation when one has a good and stable
internet connection. Operating in area's where internet connectivity is likely
to be lost in undetermined intervals, like a satellite connection or 4G/LTE in
rural area's, can become a problem when pulling a new image. When connection is
lost while image layers are being pulled, Moby will try to reconnect up to 5 times.
If this fails, the incompletely downloaded layers are lost will need to be completely
downloaded again during the next pull request. This means that we are using more
data than we might have to.

Pulling a layer multiple times from the start can become costly over a satellite
or 4G/LTE connection. As these techniques (especially 4G) quite common in IoT and
Moby is used to run Azure IoT Edge devices, I would like to add a settable maximum
download attempts. The maximum download attempts is currently set at 5
(distribution/xfer/download.go). I would like to change this constant to a variable
that the user can set. The default will still be 5, so nothing will change from
the current version unless specified when starting the daemon with the added flag
or in the config file.

I added a default value of 5 for DefaultMaxDownloadAttempts and a settable
max-download-attempts in the daemon config file. It is also added to the config
of dockerd so it can be set with a flag when starting the daemon. This value gets
stored in the imageService of the daemon when it is initiated and can be passed
to the NewLayerDownloadManager as a parameter. It will be stored in the
LayerDownloadManager when initiated. This enables us to set the max amount of
retries in makeDownoadFunc equal to the max download attempts.

I also added some tests that are based on maxConcurrentDownloads/maxConcurrentUploads.

You can pull this version and test in a development container. Either create a config
`file /etc/docker/daemon.json` with `{"max-download-attempts"=3}``, or use
`dockerd --max-download-attempts=3 -D &` to start up the dockerd. Start downloading
a container and disconnect from the internet whilst downloading. The result would
be that it stops pulling after three attempts.

Signed-off-by: Lukas Heeren <lukas-heeren@hotmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-19 13:51:40 +02:00
Sebastiaan van Stijn
90c372bb2c
cmd/dockerd: explicitly ignore unchecked errors (errcheck)
```
cmd/dockerd/config.go:50:18: Error return value of `flags.MarkHidden` is not checked (errcheck)
cmd/dockerd/config.go:55:22: Error return value of `flags.MarkDeprecated` is not checked (errcheck)
cmd/dockerd/config.go:78:18: Error return value of `flags.MarkHidden` is not checked (errcheck)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:57:53 +02:00
Sebastiaan van Stijn
07ff4f1de8
goimports: fix imports
Format the source according to latest goimports.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:56:54 +02:00
HuanHuan Ye
88c554f950 DaemonCli: Move check into startMetricsServer
Fix TODO: move into startMetricsServer()
Fix errors.Wrap return nil when passed err is nil

Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com>
Signed-off-by: HuanHuan Ye <logindaveye@gmail.com>
2019-09-12 15:18:05 +08:00
Sebastiaan van Stijn
e554ab5589
Allow system.MkDirAll() to be used as drop-in for os.MkDirAll()
also renamed the non-windows variant of this file to be
consistent with other files in this package

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-08 15:05:49 +02:00
Sebastiaan van Stijn
f6b1f01de3
Remove hack MalformedHostHeaderOverride
This hack was added to fix a compatibility with clients
that were built using Go 1.5 and older (added in 3d6f5984f5)

This hack causes some problems with current clients; with Go 1.5 and older
no longer being supported for some time, and being several years old, it
should now be ok to remove this hack altogether.

People using tools that are built with those versions of Go wouldn't have
updated those for years, and are probably out of date anyway; that's not
something we can continue taking into account.

This will affect docker clients (the docker cli) for docker 1.12 and older.
Those versions have reached EOL a long time ago (and have known unpatched
vulnerabilities), so should no longer be used anyway, but We should add
a nebtuib in the release notes, just in case someone, somewhere, still
has such old tools.

For those affected, using a more recent client (and if needed, setting
the DOCKER_API_VERSION environment variable to the needed API version)
should provide a way out.

This reverts the changes originally made in; #22000 and #22888,
which were to address #20865.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-18 21:25:04 +02:00
Sebastiaan van Stijn
c7bbb1c5a1
Merge pull request #39329 from tiborvass/buildkit-honor-daemon-dnsconfig
build: buildkit now honors daemon's DNS config
2019-07-16 16:19:20 +02:00
Sebastiaan van Stijn
d470252e87
daemon: don't listen on the same address multiple times
Before this change:

    dockerd -H unix:///run/docker.sock -H unix:///run/docker.sock -H unix:///run/docker.sock
    ...
    INFO[2019-07-13T00:02:36.195090937Z] Daemon has completed initialization
    INFO[2019-07-13T00:02:36.215940441Z] API listen on /run/docker.sock
    INFO[2019-07-13T00:02:36.215933172Z] API listen on /run/docker.sock
    INFO[2019-07-13T00:02:36.215990566Z] API listen on /run/docker.sock

After this change:

    dockerd -H unix:///run/docker.sock -H unix:///run/docker.sock -H unix:///run/docker.sock
    ...
    INFO[2019-07-13T00:01:37.533579874Z] Daemon has completed initialization
    INFO[2019-07-13T00:01:37.567045771Z] API listen on /run/docker.sock

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-13 13:21:08 +02:00
Brian Goff
24ad2f486d Add (hidden) flags to set containerd namespaces
This allows our tests, which all share a containerd instance, to be a
bit more isolated by setting the containerd namespaces to the generated
daemon ID's rather than the default namespaces.

This came about because I found in some cases we had test daemons
failing to start (really very slow to start) because it was (seemingly)
processing events from other tests.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-07-11 17:27:48 -07:00
Wei Fu
dbb5da7fcd dockerd: update reload signal comment
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2019-07-11 09:28:49 +08:00
Tibor Vass
a1cdd4bfcc build: buildkit now honors daemon's DNS config
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-07-10 00:26:03 +00:00
Sebastiaan van Stijn
6f9cdd6c73
Improve description for cpu-rt-period,runtime daemon flags
These options configure the parent cgroup, not the default for containers,
nor the daemon itself, so adding that information to the flag description
to make this slightly more clear.

relates to 56f77d5ade (#23430) which implemented
these flags.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-21 17:15:21 +02:00
Tibor Vass
53dad9f027 Remove v1 manifest code
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-18 01:40:25 +00:00
Tibor Vass
f695e98cb7 Revert "Remove the rest of v1 manifest support"
This reverts commit 98fc09128b in order to
keep registry v2 schema1 handling and libtrust-key-based engine ID.

Because registry v2 schema1 was not officially deprecated and
registries are still relying on it, this patch puts its logic back.

However, registry v1 relics are not added back since v1 logic has been
removed a while ago.

This also fixes an engine upgrade issue in a swarm cluster. It was relying
on the Engine ID to be the same upon upgrade, but the mentioned commit
modified the logic to use UUID and from a different file.

Since the libtrust key is always needed to support v2 schema1 pushes,
that the old engine ID is based on the libtrust key, and that the engine ID
needs to be conserved across upgrades, adding a UUID-based engine ID logic
seems to add more complexity than it solves the problems.

Hence reverting the engine ID changes as well.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-18 00:36:01 +00:00
Tonis Tiigi
07b3aac902 builder-next: userns remap support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-06-10 21:49:17 -07:00
Sebastiaan van Stijn
c85fe2d224
Merge pull request #38522 from cpuguy83/fix_timers
Make sure timers are stopped after use.
2019-06-07 13:16:46 +02:00
Kirill Kolyshkin
f0fe0d6c4e
Merge pull request #39105 from cpuguy83/optimize_test_daemon_startup
Optimize test daemon startup
2019-05-23 11:58:47 -07:00
Rob Gulewich
072400fc4b Make cgroup namespaces configurable
This adds both a daemon-wide flag and a container creation property:
- Set the `CgroupnsMode: "host|private"` HostConfig property at
  container creation time to control what cgroup namespace the container
  is created in
- Set the `--default-cgroupns-mode=host|private` daemon flag to control
  what cgroup namespace containers are created in by default
- Set the default if the daemon flag is unset to "host", for backward
  compatibility
- Default to CgroupnsMode: "host" for client versions < 1.40

Signed-off-by: Rob Gulewich <rgulewich@netflix.com>
2019-05-07 10:22:16 -07:00
Brian Goff
595987fd08 Add log entries for daemon startup/shutdown
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-05-06 10:36:05 -07:00
Akihiro Suda
3518383ed9 dockerd: fix rootless detection (alternative to #39024)
The `--rootless` flag had a couple of issues:
* #38702: euid=0, $USER="root" but no access to cgroup ("rootful" Docker in rootless Docker)
* #39009: euid=0 but $USER="docker" (rootful boot2docker)

To fix #38702, XDG dirs are ignored as in rootful Docker, unless the
dockerd is directly running under RootlessKit namespaces.

RootlessKit detection is implemented by checking whether `$ROOTLESSKIT_STATE_DIR` is set.

To fix #39009, the non-robust `$USER` check is now completely removed.

The entire logic can be illustrated as follows:

```
withRootlessKit := getenv("ROOTLESSKIT_STATE_DIR")
rootlessMode := withRootlessKit || cliFlag("--rootless")
honorXDG := withRootlessKit
useRootlessKitDockerProxy := withRootlessKit
removeCgroupSpec := rootlessMode
adjustOOMScoreAdj := rootlessMode
```

Close #39024
Fix #38702 #39009

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-04-25 16:47:01 +09:00
Akihiro Suda
3bc02fc040 fix containerd WaitTimeout
`defer r.WaitTimeout(10s)` was in a wrong place and had caused the
daemon to hang for 10 seconds.

Fix #39025

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-04-08 18:44:14 +09:00
Tibor Vass
05c5d20a2c grpc: register BuildKit controller to /grpc
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-04-02 19:57:59 +00:00
John Howard
a3eda72f71
Merge pull request #38541 from Microsoft/jjh/containerd
Windows: Experimental: ContainerD runtime
2019-03-19 21:09:19 -07:00
John Howard
92bf0a5046 Windows:Add ETW logging hook
Signed-off-by: John Howard <jhoward@microsoft.com>
2019-03-12 18:41:55 -07:00
John Howard
85ad4b16c1 Windows: Experimental: Allow containerd for runtime
Signed-off-by: John Howard <jhoward@microsoft.com>

This is the first step in refactoring moby (dockerd) to use containerd on Windows.
Similar to the current model in Linux, this adds the option to enable it for runtime.
It does not switch the graphdriver to containerd snapshotters.

 - Refactors libcontainerd to a series of subpackages so that either a
  "local" containerd (1) or a "remote" (2) containerd can be loaded as opposed
  to conditional compile as "local" for Windows and "remote" for Linux.

 - Updates libcontainerd such that Windows has an option to allow the use of a
   "remote" containerd. Here, it communicates over a named pipe using GRPC.
   This is currently guarded behind the experimental flag, an environment variable,
   and the providing of a pipename to connect to containerd.

 - Infrastructure pieces such as under pkg/system to have helper functions for
   determining whether containerd is being used.

(1) "local" containerd is what the daemon on Windows has used since inception.
It's not really containerd at all - it's simply local invocation of HCS APIs
directly in-process from the daemon through the Microsoft/hcsshim library.

(2) "remote" containerd is what docker on Linux uses for it's runtime. It means
that there is a separate containerd service running, and docker communicates over
GRPC to it.

To try this out, you will need to start with something like the following:

Window 1:
	containerd --log-level debug

Window 2:
	$env:DOCKER_WINDOWS_CONTAINERD=1
	dockerd --experimental -D --containerd \\.\pipe\containerd-containerd

You will need the following binary from github.com/containerd/containerd in your path:
 - containerd.exe

You will need the following binaries from github.com/Microsoft/hcsshim in your path:
 - runhcs.exe
 - containerd-shim-runhcs-v1.exe

For LCOW, it will require and initrd.img and kernel in `C:\Program Files\Linux Containers`.
This is no different to the current requirements. However, you may need updated binaries,
particularly initrd.img built from Microsoft/opengcs as (at the time of writing), Linuxkit
binaries are somewhat out of date.

Note that containerd and hcsshim for HCS v2 APIs do not yet support all the required
functionality needed for docker. This will come in time - this is a baby (although large)
step to migrating Docker on Windows to containerd.

Note that the HCS v2 APIs are only called on RS5+ builds. RS1..RS4 will still use
HCS v1 APIs as the v2 APIs were not fully developed enough on these builds to be usable.
This abstraction is done in HCSShim. (Referring specifically to runtime)

Note the LCOW graphdriver still uses HCS v1 APIs regardless.

Note also that this does not migrate docker to use containerd snapshotters
rather than graphdrivers. This needs to be done in conjunction with Linux also
doing the same switch.
2019-03-12 18:41:55 -07:00
Justin Cormack
98fc09128b Remove the rest of v1 manifest support
As people are using the UUID in `docker info` that was based on the v1 manifest signing key, replace
with a UUID instead.

Remove deprecated `--disable-legacy-registry` option that was scheduled to be removed in 18.03.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2019-03-02 10:46:37 -08:00