Commit graph

7261 commits

Author SHA1 Message Date
Sebastiaan van Stijn
0f7c9cd27e
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
Both of these were deprecated in 55f675811a,
but the format of the GoDoc comments didn't follow the correct format, which
caused them not being picked up by tools as "deprecated".

This patch updates uses in the codebase to use the alternatives.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 16:42:13 +01:00
Sebastiaan van Stijn
9d5e754caa
move pkg/system: process to a separate package
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-04 01:50:23 +01:00
Sebastiaan van Stijn
970ad4e3c7
pkg/system: IsProcessZombie() ignore "os.ErrNotExist" errors
If the file doesn't exist, the process isn't running, so we should be able
to ignore that.

Also remove an intermediate variable.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-04 01:49:49 +01:00
Paweł Gronowski
a181a825c8
daemon/start: Revert passing ctx to ctr.Start
This caused integration tests to timeout in the CI

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2022-11-03 12:22:44 +01:00
Nicolas De Loof
def549c8f6
imageservice: Add context to various methods
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2022-11-03 12:22:40 +01:00
Sebastiaan van Stijn
98f36bba9a
Merge pull request #44398 from thaJeztah/daemon_logger_godoc
daemon/logger: fix godoc
2022-11-02 23:50:29 +01:00
Sebastiaan van Stijn
be1829f63d
daemon/logger: fix godoc
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-02 18:47:50 +01:00
Tianon Gravi
b76ccfef5f
Merge pull request #44289 from thaJeztah/windows_refactor_etwlogs
daemon/logger/etwlogs: rewrite to use go-winio/pkg/etw
2022-11-02 10:43:55 -07:00
Nicolas De Loof
8d0dc69027
implement docker system df
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2022-11-02 09:35:18 +01:00
Sebastiaan van Stijn
e7904c5faa
Merge pull request #44309 from thaJeztah/daemon_check_requirements
daemon: NewDaemon(): check system requirements early
2022-11-01 13:42:44 +01:00
Sebastiaan van Stijn
ef7e4ec3c6
Merge pull request #44317 from thaJeztah/daemon_mkdir
daemon: NewDaemon(): replace system.MkdirAll for os.Mkdir where possible
2022-11-01 13:41:16 +01:00
Cory Snider
ad4073edc1 daemon: fix docs for config-default constants
Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-28 15:52:57 -04:00
Sebastiaan van Stijn
542c735926
Merge pull request #44256 from thaJeztah/redundant_sprintfs
replace redundant fmt.Sprintf() with strconv
2022-10-25 16:48:15 -04:00
Brian Goff
6c5ca9779b
Merge pull request #44310 from thaJeztah/daemon_getPluginExecRoot
daemon: getPluginExecRoot(): pass config
2022-10-25 11:52:35 -07:00
Sebastiaan van Stijn
40b3fc727d
Merge pull request #44257 from tockn/master
fix typo
2022-10-23 00:07:40 +02:00
Sebastiaan van Stijn
b9921a5560
Merge pull request #44273 from thaJeztah/use_walkdir
use filepath.WalkDir instead of filepath.Walk
2022-10-21 02:28:56 +02:00
Sebastiaan van Stijn
413f66f1a3
Merge pull request #44308 from thaJeztah/add_DOCKER_INTEGRATION_USE_SNAPSHOTTER
daemon: add TEST_INTEGRATION_USE_SNAPSHOTTER for CI
2022-10-21 00:22:20 +02:00
Sebastiaan van Stijn
1c550c36b3
Merge pull request #44268 from thaJeztah/idtools_cleanup3
pkg/idtools: remove CanAccess(), and move to daemon
2022-10-20 21:58:17 +02:00
Sebastiaan van Stijn
51fe170224
daemon: NewDaemon() fix import colliding with local variable
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-18 16:04:43 +02:00
Sebastiaan van Stijn
27bd49f4bf
daemon: NewDaemon(): replace system.MkdirAll for os.Mkdir where possible
`system.MkdirAll()` is a special version of os.Mkdir to handle creating directories
using Windows volume paths (`"\\?\Volume{4c1b02c1-d990-11dc-99ae-806e6f6e6963}"`).
This may be important when `MkdirAll` is used, which traverses all parent paths to
create them if missing (ultimately landing on the "volume" path).

The daemon.NewDaemon() function used `system.MkdirAll()` in various places where
a subdirectory within `daemon.Root` was created. This appeared to be mostly out
of convenience (to not have to handle `os.ErrExist` errors). The `daemon.Root`
directory should already be set up in these locations, and should be set up with
correct permissions. Using `system.MkdirAll()` would potentially mask errors if
the root directory is missing, and instead set up parent directories (possibly
with incorrect permissions).

Because of the above, this patch changes `system.MkdirAll` to `os.Mkdir`. As we
are changing these lines, this patch also changes the legacy octal notation
(`0700`) to the now preferred `0o700`.

One location continues to use `system.MkdirAll`, as the temp-directory may be
configured to be outside of `daemon.Root`, but a redundant `os.Stat(realTmp)`
was removed, as `system.MkdirAll` is expected to handle this.

As we are changing these lines, this patch also changes the legacy octal notation
(`0700`) to the now preferred `0o700`.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-18 16:04:40 +02:00
Sebastiaan van Stijn
2400bc66ef
Merge pull request #44285 from cpuguy83/nix_ov2_reexec
Replace overlay2 mount reexec with in-proc impl
2022-10-18 14:39:05 +02:00
Sebastiaan van Stijn
19c5d21e6f
daemon: getPluginExecRoot(): pass config
This makes it more transparent that it's unused for Linux,
and we don't pass "root", which has no relation with the
path on Linux.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-17 15:22:10 +02:00
Sebastiaan van Stijn
17fb29c9e8
daemon: NewDaemon(): check system requirements early
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-17 15:15:55 +02:00
Sebastiaan van Stijn
7ff0f654fb
daemon: add TEST_INTEGRATION_USE_SNAPSHOTTER for CI
This allows us to run CI with the containerd snapshotter enabled, without
patching the daemon.json, or changing how tests set up daemon flags.

A warning log is added during startup, to inform if this variable is set,
as it should only be used for our integration tests.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-17 15:13:53 +02:00
Sebastiaan van Stijn
0a004fd361
daemon: NewDaemon(): log message if containerd snapshotter is enabled
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-17 15:00:10 +02:00
Sebastiaan van Stijn
69f72417f4
pkg/idtools: remove CanAccess(), and move to daemon
The implementation of CanAccess() is very rudimentary, and should
not be used for anything other than a basic check (and maybe not
even for that). It's only used in a single location in the daemon,
so move it there, and un-export it to not encourage others to use
it out of context.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 22:42:39 +02:00
Sebastiaan van Stijn
a9fa147a92
daemon/logger/etwlogs: rewrite to use go-winio/pkg/etw
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 14:14:18 +02:00
Tianon Gravi
6b2328e5cc
Merge pull request #44281 from thaJeztah/windows_filter_defer_in_loop
daemon/graphdriver/windows: Remove() don't use defer() in a loop
2022-10-14 18:14:32 +00:00
Tianon Gravi
82d563243f
Merge pull request #44280 from thaJeztah/windows_filter
daemon/graphdriver/windows: use go-winio.GetFileSystemType()
2022-10-14 18:12:42 +00:00
Brian Goff
34f459423a Replace overlay2 mount reexec with in-proc impl
Building off insights from the great work Cory Snider has been doing,
this replaces a reexec with a much lower overhead implementation which
performs the `Chddir` in a new goroutine that is locked to a specific
thread with CLONE_FS unshared.
The thread is thrown away afterwards and the Chdir does effectively the
same thing as what the reexec was being used for.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2022-10-12 00:38:50 +00:00
Sebastiaan van Stijn
75bdbf02a6
daemon/graphdriver/windows: Remove() don't use defer() in a loop
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-10 17:48:45 +02:00
Sebastiaan van Stijn
90431d1857
daemon/graphdriver/windows: use go-winio.GetFileSystemType()
go-winio now defines this function, so we can consume that.

Note that there's a difference between the old implementation and the original
one (added in 1cb9e9b44e). The old implementation
had special handling for win32 error codes, which was removed in the go-winio
implementation in 0966e1ad56

As `go-winio.GetFileSystemType()` calls `filepath.VolumeName(path)` internally,
this patch also removes the `string(home[0])`, which is redundant, and could
potentially panic if an empty string would be passed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-10 16:22:05 +02:00
Sebastiaan van Stijn
e35700eb50
daemon/graphdriver/overlay2: remove deprecated overrideKernelCheck
Commit 955c1f881a (Docker v17.12.0) replaced
detection of support for multiple lowerdirs (as required by overlay2) to not
depend on the kernel version. The `overlay2.override_kernel_check` was still
used to print a warning that older kernel versions may not have full support.

After this, commit e226aea280 (Docker v20.10.0,
backported to v19.03.7) removed uses of the `overlay2.override_kernel_check`
option altogether, but we were still parsing it.

This patch changes the `parseOptions()` function to not parse the option,
printing a deprecation warning instead. We should change this to be an error,
but the  `overlay2.override_kernel_check` option was not deprecated in the
documentation, so keeping it around for one more release.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-10 14:58:40 +02:00
Sebastiaan van Stijn
f595434abc
daemon/graphdriver/devmapper: use filepath.WalkDir instead of filepath.Walk
WalkDir is more performant as it doesn't perform an os.Lstat on every visited
file or directory.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-09 17:04:44 +02:00
Sebastiaan van Stijn
8f1bc0bccc
daemon/graphdriver/btrfs: use filepath.WalkDir instead of filepath.Walk
WalkDir is more performant as it doesn't perform an os.Lstat on every visited
file or directory.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-09 17:04:16 +02:00
Sebastiaan van Stijn
56e64270f3
daemon: use strconv instead of fmt.Sprintf()
Also cleaning up some errors

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-08 17:41:39 +02:00
Sebastiaan van Stijn
533ecb44b1
daemon/logger: use strconv instead of fmt.Sprintf()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-08 17:41:38 +02:00
Sebastiaan van Stijn
7fbf321c2a
daemon/graphdriver: use strconv instead of fmt.Sprintf
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-08 17:41:35 +02:00
Sebastiaan van Stijn
1515e02c8a
Merge pull request #44215 from corhere/fix-unlockosthread-pdeathsig
Stop subprocesses from getting unexpectedly killed
2022-10-06 20:08:53 +02:00
tockn
89a9d7d650 fix typo
fix typo

Signed-off-by: Takuto Sato <tockn.jp@gmail.com>
2022-10-06 11:09:06 +09:00
Cory Snider
1f22b15030 Lock OS threads when exec'ing with Pdeathsig
On Linux, when (os/exec.Cmd).SysProcAttr.Pdeathsig is set, the signal
will be sent to the process when the OS thread on which cmd.Start() was
executed dies. The runtime terminates an OS thread when a goroutine
exits after being wired to the thread with runtime.LockOSThread(). If
other goroutines are allowed to be scheduled onto a thread which called
cmd.Start(), an unrelated goroutine could cause the thread to be
terminated and prematurely signal the command. See
https://github.com/golang/go/issues/27505 for more information.

Prevent started subprocesses with Pdeathsig from getting signaled
prematurely by wiring the starting goroutine to the OS thread until the
subprocess has exited. No other goroutines can be scheduled onto a
locked thread so it will remain alive until unlocked or the daemon
process exits.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-05 12:18:03 -04:00
Sebastiaan van Stijn
5b6b42162b
pkg/fsutils: deprecate in favor of containerd/continuity/fs
The pkg/fsutils package was forked in containerd, and later moved to
containerd/continuity/fs. As we're moving more bits to containerd, let's also
use the same implementation to reduce code-duplication and to prevent them from
diverging.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-05 11:36:04 +02:00
Austin Vazquez
6f435e0bb9 Update awslogs driver batching documentation.
Before this change, the awslogs collectBatch and processEvent
function documentation still referenced the batchPublishFrequency
constant which was removed in favor of the configurable log stream
forceFlushInterval member.

Signed-off-by: Austin Vazquez <macedonv@amazon.com>
2022-10-03 23:37:14 +00:00
Sebastiaan van Stijn
6c1df4f9c5
Merge pull request #43973 from thaJeztah/containerd_less_panics
containerd-integration: prefer error over panic where possible
2022-10-03 23:28:22 +02:00
Brian Goff
4c0e0979b4 Fix live-restore w/ restart policies + volume refs
Before this change restarting the daemon in live-restore with running
containers + a restart policy meant that volume refs were not restored.
This specifically happens when the container is still running *and*
there is a restart policy that would make sure the container was running
again on restart.

The bug allows volumes to be removed even though containers are
referencing them. 😱

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2022-09-30 22:30:58 +00:00
Sebastiaan van Stijn
0249afc523
migrate pkg/pubsub to github.com/moby/pubsub
This package was moved to a separate repository, using the steps below:

    # install filter-repo (https://github.com/newren/git-filter-repo/blob/main/INSTALL.md)
    brew install git-filter-repo

    cd ~/projects

    # create a temporary clone of docker
    git clone https://github.com/docker/docker.git moby_pubsub_temp
    cd moby_pubsub_temp

    # for reference
    git rev-parse HEAD
    # --> 572ca799db

    # remove all code, except for pkg/pubsub, license, and notice, and rename pkg/pubsub to /
    git filter-repo --path pkg/pubsub/ --path LICENSE --path NOTICE --path-rename pkg/pubsub/:

    # remove canonical imports
    git revert -s -S 585ff0ebbe6bc25b801a0e0087dd5353099cb72e

    # initialize module
    go mod init github.com/moby/pubsub
    go mod tidy

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-30 18:40:19 +02:00
Brian Goff
02ee154558
Merge pull request #44208 from thaJeztah/container_cleanup_package_vars
daemon: replaced exported errors with errdefs
2022-09-30 09:23:36 -07:00
Cory Snider
c3a6de9ec8 logger/journald: unlock OS threads
Managed containerd processes are executed with SysProcAttr.Pdeathsig set
to syscall.SIGKILL so that the managed containerd is automatically
killed along with the daemon. At least, that is the intention. In
practice, the signal is sent to the process when the creating _OS
thread_ dies! If a goroutine exits while locked to an OS thread, the Go
runtime will terminate the thread. If that thread happens to be the
same thread which the subprocess was started from, the subprocess will
be signaled. Prevent the journald driver from sometimes unintentionally
killing child processes by ensuring that all runtime.LockOSThread()
calls are paired with runtime.UnlockOSThread().

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-09-28 11:46:59 -04:00
Sebastiaan van Stijn
ddb42f3ad2
daemon: fix empty-lines (revive)
daemon/network/filter_test.go:174:19: empty-lines: extra empty line at the end of a block (revive)
    daemon/restart.go:17:116: empty-lines: extra empty line at the end of a block (revive)
    daemon/daemon_linux_test.go:255:41: empty-lines: extra empty line at the end of a block (revive)
    daemon/reload_test.go:340:58: empty-lines: extra empty line at the end of a block (revive)
    daemon/oci_linux.go:495:101: empty-lines: extra empty line at the end of a block (revive)
    daemon/seccomp_linux_test.go:17:36: empty-lines: extra empty line at the start of a block (revive)
    daemon/container_operations.go:560:73: empty-lines: extra empty line at the end of a block (revive)
    daemon/daemon_unix.go:558:76: empty-lines: extra empty line at the end of a block (revive)
    daemon/daemon_unix.go:1092:64: empty-lines: extra empty line at the start of a block (revive)
    daemon/container_operations.go:587:24: empty-lines: extra empty line at the end of a block (revive)
    daemon/network.go:807:18: empty-lines: extra empty line at the end of a block (revive)
    daemon/network.go:813:42: empty-lines: extra empty line at the end of a block (revive)
    daemon/network.go:872:72: empty-lines: extra empty line at the end of a block (revive)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-28 01:58:51 +02:00
Sebastiaan van Stijn
05042ce472
daemon/images: fix empty-lines (revive)
daemon/images/image_squash.go:17:71: empty-lines: extra empty line at the start of a block (revive)
    daemon/images/store.go:128:27: empty-lines: extra empty line at the end of a block (revive)
    daemon/images/image_list.go:154:55: empty-lines: extra empty line at the start of a block (revive)
    daemon/images/image_delete.go:135:13: empty-lines: extra empty line at the end of a block (revive)
    daemon/images/image_search.go:25:64: empty-lines: extra empty line at the start of a block (revive)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-28 01:58:51 +02:00