Commit graph

3041 commits

Author SHA1 Message Date
Sebastiaan van Stijn
f68260ba22
Dockerfile: frozen images: update to bullseye, remove buildpack-dep
Update the frozen images to also be based on Debian bullseye. Using the "slim"
variant (which looks to have all we're currently using),  and remove the
buildpack-dep frozen image.

The buildpack-dep image is quite large, and it looks like we only use it to
compile some C binaries, which should work fine on a regular debian image;

    docker build -t debian:bullseye-slim-gcc -<<EOF
    FROM debian:bullseye-slim
    RUN apt-get update && apt-get install -y gcc libc6-dev --no-install-recommends
    EOF

    docker image ls

    REPOSITORY        TAG                 IMAGE ID       CREATED          SIZE
    debian            bullseye-slim-gcc   1851750242af   About a minute ago   255MB
    buildpack-deps    bullseye            fe8fece98de2   2 days ago           834MB

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-20 18:23:06 +02:00
Sebastiaan van Stijn
98caf09f0f
fix go-digest to make sure the algorithms are registered
Previously, ioutils imported the crypty/sha256 package, because it was
used by the HashData() utility. As a side-effect of that import, the
sha256 algorithm was registered through its `init()` function.

Now that the HashData() utility is removed, the import is no longer needed
in this package, but some parts of our code depended on the side-effect, and
without this, it fail to recognise the algorithms, unless something else
happens to import crypto/sha256 / crypto/sha512, which made our
tests fail:

```
=== Failed
=== FAIL: reference TestLoad (0.00s)
    store_test.go:53: failed to parse reference: unsupported digest algorithm

=== FAIL: reference TestSave (0.00s)
    store_test.go:82: failed to parse reference: unsupported digest algorithm

=== FAIL: reference TestAddDeleteGet (0.00s)
    store_test.go:174: could not parse reference: unsupported digest algorithm

=== FAIL: reference TestInvalidTags (0.00s)
    store_test.go:355: assertion failed: error is not nil: unsupported digest algorithm
```

While it would be better to do the import in the actual locations where it's
expected, there may be code-paths we overlook, so instead adding the import
here temporarily. Until the PR in go-digest has been merged and released.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-20 00:35:41 +02:00
Sebastiaan van Stijn
572498be56
move pkg/ioutils.HashData() to libnetwork/resolvconf
It's the only location it's used, so we might as well move it there.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-20 00:35:39 +02:00
Aaron Lehmann
ba2adeebdb
Merge pull request #42676 from aaronlehmann/patternmatcher-doublestar-bug
fileutils: Fix incorrect handling of "**/foo" pattern
2021-08-16 19:58:55 -07:00
Aaron Lehmann
c44b90f3bf Test fix for Windows compatibility
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-08-12 20:03:41 -07:00
Aaron Lehmann
97ede9df26 Rename Matches to MatchesOrParentMatches
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-08-12 18:10:04 -07:00
Aaron Lehmann
9bae4f2f24 Add more optimal MatchesUsingParentResult method, use it in pkg/archive
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-08-12 13:57:50 -07:00
Samuel Karp
8b43104937
Merge pull request #42672 from thaJeztah/pkg_archive_debuglogs 2021-08-05 12:04:59 -07:00
Brian Goff
51b06c6795
Merge pull request #42683 from thaJeztah/remove_lcow_step6
Remove LCOW (step 6)
2021-07-29 11:34:29 -07:00
Brian Goff
ad268e79c4
Merge pull request #42193 from lzhfromustc/3_23
discovery & test: Fix goroutine leaks by adding 1 buffer to channel
2021-07-28 15:25:37 -07:00
Sebastiaan van Stijn
e3491ab56a
pkg/archive: gzDecompress(): use local vars for MOBY_DISABLE_PIGZ
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-28 00:11:48 +02:00
Sebastiaan van Stijn
b321474747
pkg/archive: DetectCompression(): use bytes.HasPrefix()
The existing code was the exact equivalent of bytes.HasPrefix();

    // HasPrefix tests whether the byte slice s begins with prefix.
    func HasPrefix(s, prefix []byte) bool {
    	return len(s) >= len(prefix) && Equal(s[0:len(prefix)], prefix)
    }

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-28 00:11:46 +02:00
Sebastiaan van Stijn
ebeda658bc
pkg/archive: remove some debug logs
These seemed fairly redundant

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-28 00:11:43 +02:00
Sebastiaan van Stijn
13cb04e57c
remove various LCOW bits (container, image, pkg/containerfs)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 13:36:21 +02:00
Sebastiaan van Stijn
0998c7e363
system: remove LCOWSupported() utility
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 13:36:19 +02:00
Aaron Lehmann
90f8d1b675 fileutils: Fix incorrect handling of "**/foo" pattern
(*PatternMatcher).Matches includes a special case for when the pattern
matches a parent dir, even though it doesn't match the current path.
However, it assumes that the parent dir which would match the pattern
must have the same number of separators as the pattern itself. This
doesn't hold true with a patern like "**/foo". A file foo/bar would have
len(parentPathDirs) == 1, which is less than the number of path
len(pattern.dirs) == 2... therefore this check would be skipped.

Given that "**/foo" matches "foo", I think it's a bug that the "parent
subdir matches" check is being skipped in this case.

It seems safer to loop over the parent subdirs and check each against
the pattern. It's possible there is a safe optimization to check only a
certain subset, but the existing logic seems unsafe.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-07-26 11:28:10 -07:00
Sebastiaan van Stijn
28409ca6c7
replace pkg/signal with moby/sys/signal v0.5.0
This code was moved to the moby/sys repository

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-23 09:32:54 +02:00
Brian Goff
9a6ff685a8
Merge pull request #42641 from thaJeztah/make_signal_selfcontained 2021-07-19 14:46:15 -07:00
Justin Cormack
b337c70bdc
Merge pull request #42639 from thaJeztah/system_info_clean
pkg/sysinfo: assorted cleanup/refactoring for handling warnings and logging
2021-07-19 15:17:07 +01:00
Justin Cormack
fb21a1e474
Merge pull request #42580 from thaJeztah/reduce_TestClientWithRequestTimeout_flakiness
Reduce TestClientWithRequestTimeout flakiness
2021-07-19 15:11:43 +01:00
Sebastiaan van Stijn
6ff6913ac4
pkg/signal: remove gotest.tools dependency
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-15 18:24:23 +02:00
Sebastiaan van Stijn
0880df4644
pkg/signal: move Trap() to cmd/dockerd
It's the only location where this is used, and it's quite specific
to dockerd (not really a reusable function for external use), so
moving it into that package.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-15 18:11:00 +02:00
Sebastiaan van Stijn
ea5c94cdb9
pkg/signal: move signal.DumpStacks() to a separate package
It is not directly related to signal-handling, so can well live
in its own package.

Also added a variant that doesn't take a directory to write files
to, for easier consumption / better match to how it's used.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-15 18:09:43 +02:00
Sebastiaan van Stijn
4ace1998e5
pkg/sysinfo: use correct name for AppArmor in t.Skip()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 23:10:13 +02:00
Sebastiaan van Stijn
9b795c3e50
pkg/sysinfo.New(), daemon.RawSysInfo(): remove "quiet" argument
The "quiet" argument was only used in a single place (at daemon startup), and
every other use had to pass "false" to prevent this function from logging
warnings.

Now that SysInfo contains the warnings that occurred when collecting the
system information, we can make leave it up to the caller to use those
warnings (and log them if wanted).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 23:10:07 +02:00
Sebastiaan van Stijn
1fb62f455c
pkg/sysinfo: collect warnings in SysInfo struct
This allows the warnings to be consumed in other locations.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 17:28:25 +02:00
Sebastiaan van Stijn
208d3c6efb
pkg/sysinfo: move cg2Controllers to be a field in SysInfo and unify v1/v2
We pass the SysInfo struct to all functions. Adding cg2Controllers as a
(non-exported) field makes passing around this information easier.

Now that infoCollector and infoCollectorV2 have the same signature, we can
simplify some bits and use a single slice for all "collectors".

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:39:44 +02:00
Sebastiaan van Stijn
5cc20ad9e5
pkg/sysinfo: adjust Opt to set new field
This removes the need to have the opts type.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:39:26 +02:00
Sebastiaan van Stijn
ca27b473cc
pkg/sysinfo: move cg2GroupPath to be a field in SysInfo
We pass the SysInfo struct to all functions. Adding cg2GroupPath as a
(non-exported) field makes passing around this information easier.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:37:03 +02:00
Sebastiaan van Stijn
e70c5ea1a9
pkg/sysinfo.newV2() remove redundant path.Clean()
path.Join() already does path.Clean(), and the opts.cg2GroupPath
field is already cleaned as part of WithCgroup2GroupPath()

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:37:02 +02:00
Sebastiaan van Stijn
6677ab6a63
pkg/sysinfo: move cgMounts to be a field in SysInfo
We pass the SysInfo struct to all functions. Adding cgMounts as a
(non-exported) field makes passing around this information easier.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:37:00 +02:00
Sebastiaan van Stijn
10ce0d84c2
pkg/sysinfo.New() move v1 code to a newV1() function
This makes it clearer that this code is the cgroups v1 equivalent of newV2().

Also moves the "options" handling to newV2() because it's currently only used
for cgroupsv2.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:36:56 +02:00
jk-vb
1d055267a8
updated names-generator.go for alphabetization
re ordered some entries so they are in proper alphabetical order

Signed-off-by: Justin Keller <jkeller@vb-jkeller-mbp.local>
2021-07-13 23:52:51 +02:00
Sebastiaan van Stijn
00a4f67ddf
Reduce TestClientWithRequestTimeout flakiness
The test sometimes failed because no error was returned:

    === Failed
    === FAIL: pkg/plugins TestClientWithRequestTimeout (0.00s)
         client_test.go:254: assertion failed: expected an error, got nil: expected error

Possibly caused by a race condition, as the sleep was just 1 ms longer than the timeout;
this patch is increasing the sleep in the response to try to reduce flakiness.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-29 23:53:23 +02:00
Sebastiaan van Stijn
301be64d41
Merge pull request #42515 from geaaru/permit-override-name
ReplaceFileTarWrapper: permit to override file name
2021-06-24 21:01:08 +02:00
Sebastiaan van Stijn
46c591b045
pkg/system: deprecate some consts and move them to pkg/idtools
These consts were used in combination with idtools utilities, which
makes it a more logical location for these consts to live.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 23:16:15 +02:00
Sebastiaan van Stijn
26f5db7a1d
pkg/system: remove unused system.Unmount() utility
On Linux/Unix it was just a thin wrapper for unix.Unmount(), and a no-op on Windows.

This function was not used anywhere (also not externally), so removing this without
deprecating first.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 23:09:56 +02:00
Sebastiaan van Stijn
5f04517c48
pkg/system: remove deprecated GetOSVersion(), consts, SecurityInfo utils.
This removes the deprecated wrappers, so that the package no longer has
hcsshim as a dependency. These wrappers were no longer used in our code,
and were deprecated in the 20.10 release (giving external consumers to
replace the deprecated ones).

Note that there are two consts which were unused, but for which there is
no replacement in golang.org/x/sys;

    const (
        PROCESS_TRUST_LABEL_SECURITY_INFORMATION = 0x00000080
        ACCESS_FILTER_SECURITY_INFORMATION       = 0x00000100
    )

PROCESS_TRUST_LABEL_SECURITY_INFORMATION is documented as "reserved", and I could
not find clear documentation about ACCESS_FILTER_SECURITY_INFORMATION, so not sure
if they must be included in golang.org/x/sys: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/23e75ca3-98fd-4396-84e5-86cd9d40d343

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 23:09:55 +02:00
Sebastiaan van Stijn
472f21b923
replace uses of deprecated containerd/sys.RunningInUserNS()
This utility was moved to a separate package, which has no
dependencies.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 11:01:24 +02:00
Daniele Rondina
73aef6edfe
ReplaceFileTarWrapper: permit to override file name
Signed-off-by: Daniele Rondina <geaaru@sabayonlinux.org>
2021-06-11 11:00:12 +02:00
Sebastiaan van Stijn
bb17074119
reformat "nolint" comments
Unlike regular comments, nolint comments should not have a leading space.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:42 +02:00
Sebastiaan van Stijn
7b071e0557
pkg/archive: RebaseArchiveEntries(): ignore G110
pkg/archive/copy.go:357:16: G110: Potential DoS vulnerability via decompression bomb (gosec)
                if _, err = io.Copy(rebasedTar, srcTar); err != nil {
                            ^

Ignoring GoSec G110. See https://github.com/securego/gosec/pull/433
and https://cure53.de/pentest-report_opa.pdf, which recommends to
replace io.Copy with io.CopyN7. The latter allows to specify the
maximum number of bytes that should be read. By properly defining
the limit, it can be assured that a GZip compression bomb cannot
easily cause a Denial-of-Service.
After reviewing, this should not affect us, because here we do not
read into memory.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:36 +02:00
Sebastiaan van Stijn
f77213efc2
gosimple: S1039: unnecessary use of fmt.Sprintf
pkg/devicemapper/devmapper.go:383:28: S1039: unnecessary use of fmt.Sprintf (gosimple)
        if err := task.setMessage(fmt.Sprintf("@cancel_deferred_remove")); err != nil {
                                  ^
    integration/plugin/graphdriver/external_test.go:321:18: S1039: unnecessary use of fmt.Sprintf (gosimple)
                http.Error(w, fmt.Sprintf("missing id"), 409)
                              ^
    integration-cli/docker_api_stats_test.go:70:31: S1039: unnecessary use of fmt.Sprintf (gosimple)
            _, body, err := request.Get(fmt.Sprintf("/info"))
                                        ^
    integration-cli/docker_cli_build_test.go:4547:19: S1039: unnecessary use of fmt.Sprintf (gosimple)
                "--build-arg", fmt.Sprintf("FOO1=fromcmd"),
                               ^
    integration-cli/docker_cli_build_test.go:4548:19: S1039: unnecessary use of fmt.Sprintf (gosimple)
                "--build-arg", fmt.Sprintf("FOO2="),
                               ^
    integration-cli/docker_cli_build_test.go:4549:19: S1039: unnecessary use of fmt.Sprintf (gosimple)
                "--build-arg", fmt.Sprintf("FOO3"), // set in env
                               ^
    integration-cli/docker_cli_build_test.go:4668:32: S1039: unnecessary use of fmt.Sprintf (gosimple)
            cli.WithFlags("--build-arg", fmt.Sprintf("tag=latest")))
                                         ^
    integration-cli/docker_cli_build_test.go:4690:32: S1039: unnecessary use of fmt.Sprintf (gosimple)
            cli.WithFlags("--build-arg", fmt.Sprintf("baz=abc")))
                                         ^
    pkg/jsonmessage/jsonmessage_test.go:255:4: S1039: unnecessary use of fmt.Sprintf (gosimple)
                fmt.Sprintf("ID: status\n"),
                ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:27 +02:00
Sebastiaan van Stijn
6b0ecacd92
gosec: G404: Use of weak random number generator
These should be ok to ignore for the purpose they're used

    pkg/namesgenerator/names-generator.go:843:36: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
        name := fmt.Sprintf("%s_%s", left[rand.Intn(len(left))], right[rand.Intn(len(right))])
                                          ^
    pkg/namesgenerator/names-generator.go:849:36: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
            name = fmt.Sprintf("%s%d", name, rand.Intn(10))
                                             ^
    testutil/stringutils.go:11:18: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
            b[i] = letters[rand.Intn(len(letters))]
                           ^
    pkg/namesgenerator/names-generator.go:849:36: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
            name = fmt.Sprintf("%s%d", name, rand.Intn(10))
                                             ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:18 +02:00
Sebastiaan van Stijn
bc8acc5b08
pkg/fileutils: PatternMatcher.Matches(): remove debug logging
Trying to avoid logging code in "libraries" used elsewhere.
If this debug log is important, it should be easy to add in code
that's calling it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-09 11:56:00 +02:00
Brian Goff
a7ea29a5a6
Merge pull request #42451 from thaJeztah/remove_lcow_step1
Remove LCOW code (step 1)
2021-06-08 13:41:45 -07:00
Jakub Guzik
7ef6ece774 Fix setting swaplimit=true without checking
Signed-off-by: Jakub Guzik <jakubmguzik@gmail.com>
2021-06-03 22:54:10 +02:00
Sebastiaan van Stijn
e047d984dc
Remove LCOW code (step 1)
The LCOW implementation in dockerd has been deprecated in favor of re-implementation
in containerd (in progress). Microsoft started removing the LCOW V1 code from the
build dependencies we use in Microsoft/opengcs (soon to be part of Microsoft/hcshhim),
which means that we need to start removing this code.

This first step removes the lcow graphdriver, the LCOW initialization code, and
some LCOW-related utilities.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-03 21:16:21 +02:00
Sebastiaan van Stijn
05f520dd3c
pkg/signal: ignore SIGURG on all platforms
Other Unix platforms (e.g. Darwin) are also affected by the Go
runtime sending SIGURG.

This patch changes how we match the signal by just looking for the
"URG" name, which should handle any platform that has this signal
defined in the SignalMap.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-05-25 12:07:53 +02:00
Brian Goff
328d23f625
Merge pull request #42390 from thaJeztah/skip_defunct_test
pkg/fileutils: TestMatches: remove cases no longer valid for go1.16
2021-05-20 11:10:46 -07:00