Commit graph

44853 commits

Author SHA1 Message Date
Cory Snider
d1e3705c1a libnet/d/overlay: restore thread netns
func (*network) watchMiss() correctly locks its goroutine to an OS
thread before changing the thread's network namespace, but neglects to
restore the thread's network namespace before unlocking. Fix this
oversight by unlocking iff the thread's network namespace is
successfully restored.

Prevent the watchMiss goroutine from being locked to the main thread to
avoid the issues which would arise if such a situation was to occur.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-25 13:35:44 -04:00
Cory Snider
3e2f0c7a39 libnetwork: fixup thread locking in Linux tests
The parallel tests were unconditionally unlocking the test case
goroutine from the OS thread, irrespective of whether the thread's
network namespace was successfully restored. This was not a problem in
practice as the unpaired calls to runtime.LockOSThread() peppered
through the test case would have prevented the goroutine from being
unlocked. Unlock the goroutine from the thread iff the thread's network
namespace is successfully restored.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-25 13:35:44 -04:00
Cory Snider
afa41b16ea libnetwork/testutils: restore netns on teardown
testutils.SetupTestOSContext() sets the calling thread's network
namespace but neglected to restore it on teardown. This was not a
problem in practice as it called runtime.LockOSThread() twice but
runtime.UnlockOSThread() only once, so the tampered threads would be
terminated by the runtime when the test case returned and replaced with
a clean thread. Correct the utility so it restores the thread's network
namespace during teardown and unlocks the goroutine from the thread on
success.

Remove unnecessary runtime.LockOSThread() calls peppering test cases
which leverage testutils.SetupTestOSContext().

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-24 15:37:46 -04:00
Sebastiaan van Stijn
40b3fc727d
Merge pull request #44257 from tockn/master
fix typo
2022-10-23 00:07:40 +02:00
Sebastiaan van Stijn
fffa94787c
Merge pull request #44344 from thaJeztah/go1.18_compat
builder/remotecontext/git: allow building on go1.18
2022-10-21 19:38:54 +02:00
Sebastiaan van Stijn
4fdc1bb1fb
builder/remotecontext/git: allow building on go1.18
cmd.Environ() is new in go1.19, and not needed for this specific case.
Without this, trying to use this package in code that uses go1.18 will fail;

    builder/remotecontext/git/gitutils.go:216:23: cmd.Environ undefined (type *exec.Cmd has no field or method Environ)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-21 17:41:41 +02:00
Sebastiaan van Stijn
43b8dffb83
Merge pull request #44327 from thaJeztah/ghsa-ambiguous-pull-by-digest_master
Validate digest in repo for pull by digest
2022-10-21 14:19:55 +02:00
Sebastiaan van Stijn
b9921a5560
Merge pull request #44273 from thaJeztah/use_walkdir
use filepath.WalkDir instead of filepath.Walk
2022-10-21 02:28:56 +02:00
Sebastiaan van Stijn
08735b4aa8
Merge pull request #44324 from corhere/fix-git-file-leak
builder: Isolate Git from local system
2022-10-21 02:11:33 +02:00
Sebastiaan van Stijn
64cb636b06
Merge pull request #44337 from thaJeztah/buildkit_skip_unit
gha: buildkit: remove "skip-integration-tests" from matrix
2022-10-21 01:59:41 +02:00
Sebastiaan van Stijn
4f43cb660a
skip TestImagePullStoredfDigestForOtherRepo() on Windows and rootless
- On Windows, we don't build and run a local  test registry (we're not running
  docker-in-docker), so we need to skip this test.
- On rootless, networking doesn't support this (currently)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-21 01:48:59 +02:00
Brian Goff
27530efedb
Validate digest in repo for pull by digest
This is accomplished by storing the distribution source in the content
labels. If the distribution source is not found then we check to the
registry to see if the digest exists in the repo, if it does exist then
the puller will use it.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-21 01:48:59 +02:00
Sebastiaan van Stijn
92eca900b0
Revert "testutil/registry: remove unused WithStdout(), WithStErr() opts"
This reverts commit 1f21c4dd05.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-21 01:48:56 +02:00
Sebastiaan van Stijn
c93c9bca8e
Merge pull request #44336 from thaJeztah/buildkit_testskips
gha: update buildkit to v0.10.5-6-ge27c8e24 to skip some tests
2022-10-21 01:47:32 +02:00
Sebastiaan van Stijn
0f2956ab5d
Merge pull request #44302 from thaJeztah/sys_windows
pkg/system: optimize and refactor MkdirAllWithACL()
2022-10-21 00:36:58 +02:00
Sebastiaan van Stijn
413f66f1a3
Merge pull request #44308 from thaJeztah/add_DOCKER_INTEGRATION_USE_SNAPSHOTTER
daemon: add TEST_INTEGRATION_USE_SNAPSHOTTER for CI
2022-10-21 00:22:20 +02:00
Sebastiaan van Stijn
201fdf67ac
gha: update buildkit to v0.10.5-6-ge27c8e24 to skip some tests
full diff: https://github.com/moby/buildkit/compare/v0.10.5...v0.10.5-6-ge27c8e24

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-20 23:49:26 +02:00
Sebastiaan van Stijn
0760c6f4e1
gha: buildkit: make checks more readable
GitHub uses these parameters to construct a name; removing the ./ prefix
to make them more readable (and add them back where it's used)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-20 23:18:44 +02:00
Sebastiaan van Stijn
cfa2f9a2f2
gha: buildkit: remove "skip-integration-tests" from matrix
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-20 23:17:55 +02:00
Cory Snider
67d010bd2c builder: add missing doc comment
Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-20 16:47:18 -04:00
Cory Snider
94672c89cc builder: fix running git commands on Windows
Setting cmd.Env overrides the default of passing through the parent
process' environment, which works out fine most of the time, except when
it doesn't. For whatever reason, leaving out all the environment causes
git-for-windows sh.exe subprocesses to enter an infinite loop of
access violations during Cygwin initialization in certain environments
(specifically, our very own dev container image).

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-20 16:47:18 -04:00
Cory Snider
61acc9939f builder: make git config isolation opt-in
While it is undesirable for the system or user git config to be used
when the daemon clones a Git repo, it could break workflows if it was
unconditionally applied to docker/cli as well.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-20 16:47:18 -04:00
Cory Snider
72119f5d9b builder: isolate git from local system
Prevent git commands we run from reading the user or system
configuration, or cloning submodules from the local filesystem.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-20 16:47:18 -04:00
Cory Snider
0f7b0897cc builder: explicitly set CWD for all git commands
Keep It Simple! Set the working directory for git commands by...setting
the git process's working directory. Git commands can be run in the
parent process's working directory by passing the empty string.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-20 16:47:18 -04:00
Cory Snider
8deb92d653 builder: modernize TestCheckoutGit
Make the test more debuggable by logging all git command output and
running each table-driven test case as a subtest.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-10-20 16:47:18 -04:00
Sebastiaan van Stijn
1c550c36b3
Merge pull request #44268 from thaJeztah/idtools_cleanup3
pkg/idtools: remove CanAccess(), and move to daemon
2022-10-20 21:58:17 +02:00
Sebastiaan van Stijn
86ec93ccaf
Merge pull request #44320 from thaJeztah/bump_buildkit
vendor: github.com/moby/buildkit v0.10.5
2022-10-19 10:02:18 +02:00
Sebastiaan van Stijn
0fc17c42af
vendor: github.com/moby/buildkit v0.10.5
https://github.com/moby/buildkit/releases/tag/v0.10.5

full diff: https://github.com/moby/buildkit/compare/v0.10.4...v0.10.5

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-18 22:38:34 +02:00
Sebastiaan van Stijn
2400bc66ef
Merge pull request #44285 from cpuguy83/nix_ov2_reexec
Replace overlay2 mount reexec with in-proc impl
2022-10-18 14:39:05 +02:00
Tianon Gravi
6ad1e347a3
Merge pull request #44307 from thaJeztah/pkg_fileutils_fixes
pkg/fileutils: ReadSymlinkedDirectory: preserve underlying error, and improve test-coverage
2022-10-17 22:29:50 +00:00
Tianon Gravi
8dd5dae91a
Merge pull request #44305 from thaJeztah/distributable
registry: allow "allow-nondistributable-artifacts" for Docker Hub
2022-10-17 21:45:00 +00:00
Sebastiaan van Stijn
7ff0f654fb
daemon: add TEST_INTEGRATION_USE_SNAPSHOTTER for CI
This allows us to run CI with the containerd snapshotter enabled, without
patching the daemon.json, or changing how tests set up daemon flags.

A warning log is added during startup, to inform if this variable is set,
as it should only be used for our integration tests.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-17 15:13:53 +02:00
Sebastiaan van Stijn
0a004fd361
daemon: NewDaemon(): log message if containerd snapshotter is enabled
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-17 15:00:10 +02:00
Sebastiaan van Stijn
4fa853f5de
pkg/fileutils: ReadSymlinkedDirectory: preserve underlying error
We were discarding the underlying error, which made it impossible for
callers to detect (e.g.) an os.ErrNotExist.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-16 20:15:08 +02:00
Sebastiaan van Stijn
24e371c812
pkg/fileutils: improve tests
- use t.TempDir() to make sure we're testing from a clean state
- improve checks for errors to have the correct error-type where possible

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-16 20:15:08 +02:00
Sebastiaan van Stijn
30e5333ce3
registry: allow "allow-nondistributable-artifacts" for Docker Hub
Previously, Docker Hub was excluded when configuring "allow-nondistributable-artifacts".
With the updated policy announced by Microsoft, we can remove this restriction;
https://techcommunity.microsoft.com/t5/containers/announcing-windows-container-base-image-redistribution-rights/ba-p/3645201

There are plans to deprecated support for foreign layers altogether in the OCI,
and we should consider to make this option the default, but as that requires
deprecating the option (and possibly keeping an "opt-out" option), we can look
at that separately.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-16 01:08:49 +02:00
Sebastiaan van Stijn
d006242d73
Merge pull request #44251 from thaJeztah/pkg_dir_cleanup
pkg/directory: remove unused MoveToSubdir() utility, and some refactoring
2022-10-15 22:48:19 +02:00
Sebastiaan van Stijn
69f72417f4
pkg/idtools: remove CanAccess(), and move to daemon
The implementation of CanAccess() is very rudimentary, and should
not be used for anything other than a basic check (and maybe not
even for that). It's only used in a single location in the daemon,
so move it there, and un-export it to not encourage others to use
it out of context.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 22:42:39 +02:00
Sebastiaan van Stijn
ee34a8ac29
pkg/idtools: setPermissions() accept Identity as argument
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 22:42:39 +02:00
Sebastiaan van Stijn
bca90530fa
pkg/idtools: simplify if-statement
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 22:42:38 +02:00
Sebastiaan van Stijn
d68fa0382d
pkg/idtools: don't use system.Stat() on unix
Looks like we don't need the abstraction, so we can reduce the
dependency on pkg/system.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 22:42:35 +02:00
Sebastiaan van Stijn
1311687d0d
Merge pull request #44254 from thaJeztah/idtools_cleanup2
pkg/idtools: various cleanups
2022-10-15 22:42:09 +02:00
Sebastiaan van Stijn
081c00c7df
Merge pull request #44265 from thaJeztah/pkg_system_move_init_step1
pkg/system: cleanup, test-fixes and improvements and minor fixes
2022-10-15 21:28:13 +02:00
Sebastiaan van Stijn
b3a6ae71c1
Merge pull request #44266 from thaJeztah/pidfile_cleanup
pkg/pidfile: remove PIDFile type and cleanup
2022-10-15 21:26:17 +02:00
Sebastiaan van Stijn
43d6eb7173
pkg/pidfile: remove PIDFile type, rename New() to Write()
This type felt really redundant; `pidfile.New()` takes the path of the file to
create as an argument, so this is already known. The only thing the PIDFile
type provided was a `Remove()` method, which was just calling `os.Remove()` on
the path of the file.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 16:40:16 +02:00
Sebastiaan van Stijn
dd8983f96c
pkg/pidfile: reduce cyclomatic complexity, and small optimisation
Use bytes.TrimSpace instead of using the strings package, which is
more performant, and allows us to skip the intermediate variable.

Also combined some "if" statements to reduce cyclomatic complexity.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 15:11:40 +02:00
Sebastiaan van Stijn
4917bcc039
pkg/pidfile: don't ignore all errors when reading file
It's ok to ignore if the file doesn't exist, or if the file doesn't
have a PID in it, but we should produce an error if the file exists,
but we're unable to read it for other reasons.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 15:11:40 +02:00
Sebastiaan van Stijn
3ce2a7d026
pkg/pidfile: pkg/pidfile: use strconv instead of fmt.Sprintf(), and unconvert
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 15:11:40 +02:00
Sebastiaan van Stijn
f058afc861
pkg/system: synchronize mkdirall() with latest os.MkDirAll()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 13:24:43 +02:00
Sebastiaan van Stijn
2e66c0b6f0
pkg/system: create SecurityAttribute only once (Windows)
The same attribute was generated for each path that was created, but always
the same, so instead of generating it in each iteration, generate it once,
and pass it to our mkdirall() implementation.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-15 13:24:43 +02:00