Commit graph

3291 commits

Author SHA1 Message Date
Cory Snider
95824f2b5f pkg/containerfs: simplify ContainerFS type
Iterate towards dropping the type entirely.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-09-23 16:56:49 -04:00
Cory Snider
be4f4644a8 pkg/containerfs: drop Driver abstraction
The Driver abstraction was needed for Linux Containers on Windows,
support for which has since been removed.

There is no direct equivalent to Lchmod() in the standard library so
continue to use the containerd/continuity version.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-09-23 16:25:22 -04:00
Cory Snider
7014c0d65d pkg/containerfs: drop PathDriver abstraction
With LCOW support removed, there is no need to support non-native file
paths any longer.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-09-23 16:25:22 -04:00
Cory Snider
e37a2d1879 pkg/containerfs: delete Archiver, Driver
They were needed for Linux Containers on Windows, which is no longer
supported.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-09-23 16:25:22 -04:00
Cory Snider
a7c8fdc55b pkg/containerfs: make ResolveScopedPath a free fn
Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-09-23 16:25:22 -04:00
Cory Snider
fd16e003fb pkg/containerfs: Remove NewLocalDriver()
Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-09-23 16:25:22 -04:00
Cory Snider
4d48c00f94 pkg/containerfs: Trim away Driver interface part 1
The Driver interface was required for Linux Containers on Windows, which
is no longer supported.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-09-23 16:25:21 -04:00
Sebastiaan van Stijn
a39ba095eb
Merge pull request #44060 from thaJeztah/archive_cleanup
pkg/archive: make CanonicalTarNameForPath and alias for filepath.ToSlash
2022-09-21 18:01:24 +02:00
Bjorn Neergaard
85fa72c599 chore: update supported go version to 1.18+
The 1.16 `io/fs` compatibility code was being built on 1.18 and 1.19.
Drop it completely as 1.16 is long EOL, and additionally drop 1.17 as it
has been EOL for a month and 1.18 is both the minimum Go supported by
the 20.10 branch, as well as a very easy jump from 1.17.

Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
2022-09-15 13:52:39 -06:00
Sebastiaan van Stijn
f3e6b7065a
Merge pull request #44083 from thaJeztah/archive_cachebust
pkg/archive: remove backward compat hack for go < 1.9
2022-09-07 21:28:40 +02:00
Sebastiaan van Stijn
0f052eb4f5
namesgenerator: remove Valentina Tereshkova
While the name generator has been frozen for new additions in 624b3cfbe8,
this person has become controversial. Our intent is for this list to be inclusive
and non-controversial.

This patch removes the name from the list.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-06 13:37:32 +02:00
Sebastiaan van Stijn
609d87003a
pkg/archive: strip "write" bits again on Windows
1. Commit 1a22418f9f changed permissions to `0700`
   on Windows, or more factually, it removed `rw` (`chmod g-rw,o-rw`) and added
   executable bits (`chmod u+x`).
2. This was too restrictive, and b7dc9040f0 changed
   permissions to only remove the group- and world-writable bits to give read and
  execute access to everyone, but setting execute permissions for everyone.
3. However, this also removed the non-permission bits, so 41eb61d5c2
   updated the code to preserve those, and keep parity with Linux.

This changes it back to `2.`. I wonder (_think_) _permission_ bits (read, write)
can be portable, except for the _executable_ bit (which is not present on Windows).
The alternative could be to keep the permission bits, and only set the executable
bit (`perm | 0111`) for everyone (equivalent of `chmod +x`), but that likely would
be a breaking change.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-05 10:50:10 +02:00
Sebastiaan van Stijn
cddaa84777
pkg/archive: remove backward compat hack for go < 1.9
The fillGo18FileTypeBits func was added in 1a451d9a7b
to keep the tar headers consistent with headers created with go1.8 and older.

go1.8 and older incorrectly preserved all file-mode bits, including file-type,
instead of stripping those bits and only preserving the _permission_ bits, as
defined in;

- the GNU tar spec: https://www.gnu.org/software/tar/manual/html_node/Standard.html
- and POSIX: http://pubs.opengroup.org/onlinepubs/009695399/basedefs/tar.h.html

We decided at the time to copy the "wrong" behavior to prevent a cache-bust and
to keep the archives identical, however:

- It's not matching the standards, which causes differences between our tar
  implementation and the standard tar implementations, as well as implementations
  in other languages, such as Python (see docker/compose#883).
- BuildKit does not implement this hack.
- We don't _need_ this extra information (as it's already preserved in the
  type header; https://pkg.go.dev/archive/tar#pkg-constants

In short; let's remove this hack.

This reverts commit 1a451d9a7b.
This reverts commit 41eb61d5c2.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-04 21:44:02 +02:00
Sebastiaan van Stijn
c9c55df1f2
Merge pull request #44054 from thaJeztah/json_stream_export
pkg/jsonmessage: export "Stream" interface
2022-09-01 19:38:59 +02:00
Sebastiaan van Stijn
8b36298d7f
pkg/archive: remove tests for CanonicalTarNameForPath
Now that CanonicalTarNameForPath is an alias for filepath.ToSlash, they were
mostly redundant, and only testing Go's stdlib. Coverage for filepath.ToSlash is
provided through TestCanonicalTarName, which does a superset of CanonicalTarNameForPath,

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-30 10:41:53 +02:00
Sebastiaan van Stijn
d59758450b
pkg/archive: make CanonicalTarNameForPath and alias for filepath.ToSlash
filepath.ToSlash is already a no-op on non-Windows platforms, so there's no
need to provide multiple implementations.

We could consider deprecating this function, but it's used in the CLI, and
perhaps it's still useful to have a canonical location to perform this normalization.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-30 10:40:43 +02:00
Sebastiaan van Stijn
509f19f611
replace pkg/system Sequential funcs with moby/sys/sequential
Migrating these functions to allow them being shared between moby, docker/cli,
and containerd, and to allow using them without importing all of sys / system,
which (in containerd) also depends on hcsshim and more.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-30 09:33:49 +02:00
Sebastiaan van Stijn
5e0599cb6e
pkg/jsonmessage: export "Stream" interface
This interface is used as part of an exported function's signature,
so exporting the interface as well for callers to know what the argument
must have implemented.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-29 16:15:37 +02:00
Sebastiaan van Stijn
2640aec0d7
pkg/system: make IsAbs() platform-agnostic
filepath.IsAbs() will short-circuit on Linux/Unix, so having a single
implementation should not affect those platforms.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-27 15:11:27 +02:00
Sebastiaan van Stijn
df965f51bf
pkg/idtools: remove deprecated NewIdentityMapping, UIDS() and GIDS()
These were deprecated in 098a44c07f, which is
in the 22.06 branch, and no longer in use since e05f614267
so we can remove them from the master branch.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-08-09 18:03:29 +02:00
Sebastiaan van Stijn
4f08346686
fix formatting of "nolint" tags for go1.19
The correct formatting for machine-readable comments is;

    //<some alphanumeric identifier>:<options>[,<option>...][ // comment]

Which basically means:

- MUST NOT have a space before `<identifier>` (e.g. `nolint`)
- Identified MUST be alphanumeric
- MUST be followed by a colon
- MUST be followed by at least one `<option>`
- Optionally additional `<options>` (comma-separated)
- Optionally followed by a comment

Any other format will not be considered a machine-readable comment by `gofmt`,
and thus formatted as a regular comment. Note that this also means that a
`//nolint` (without anything after it) is considered invalid, same for `//#nosec`
(starts with a `#`).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-07-13 22:31:53 +02:00
Samuel Karp
0910306bf9
Merge pull request #43786 from thaJeztah/gofmt_119 2022-07-08 21:56:26 -07:00
Sebastiaan van Stijn
52c1a2fae8
gofmt GoDoc comments with go1.19
Older versions of Go don't format comments, so committing this as
a separate commit, so that we can already make these changes before
we upgrade to Go 1.19.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-07-08 19:56:23 +02:00
Sebastiaan van Stijn
e4e819b49c
pkg/parsers/operatingsystem: fix stray import
This was caught by goimports;

    goimports -w $(find . -type f -name '*.go'| grep -v "/vendor/")

CI doesn't run on these platforms, so didn't catch it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-07-08 18:42:20 +02:00
Sebastiaan van Stijn
5daceee6ca
Merge pull request #43624 from rumpl/feat-memdb-prefix
Use hashicorp/go-memdb instead of truncindex
2022-07-08 11:09:20 +02:00
Feroz Salam
1db286c5e8 Fix alphabetisation of possible names
Signed-off-by: Feroz Salam <feroz.salam@sourcegraph.com>
2022-06-03 15:07:52 +01:00
Sebastiaan van Stijn
553b0edb4c
fix unclosed file-handles in tests
These seemed to prevent cleaning up directories;

On arm64:

    === RUN   TestSysctlOverride
        testing.go:1090: TempDir RemoveAll cleanup: unlinkat /tmp/TestSysctlOverride2860094781/001/mounts/shm: device or resource busy
    --- FAIL: TestSysctlOverride (0.00s)

On Windows:

    === Failed
    === FAIL: github.com/docker/docker/daemon TestLoadOrCreateTrustKeyInvalidKeyFile (0.00s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestLoadOrCreateTrustKeyInvalidKeyFile2014634395\001\keyfile4156691647: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/daemon/graphdriver TestIsEmptyDir (0.01s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestIsEmptyDir1962964337\001\dir-with-empty-file\file2523853824: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/pkg/directory TestSizeEmptyFile (0.00s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeEmptyFile1562416712\001\file16507846: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/pkg/directory TestSizeNonemptyFile (0.00s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeNonemptyFile1240832785\001\file3265662846: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/pkg/directory TestSizeFileAndNestedDirectoryEmpty (0.00s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeFileAndNestedDirectoryEmpty2163416550\001\file3715413181: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/pkg/directory TestSizeFileAndNestedDirectoryNonempty (0.00s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeFileAndNestedDirectoryNonempty878205470\001\file3280422273: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/volume/service TestSetGetMeta (0.01s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSetGetMeta3332268057\001\db: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/volume/service TestList (0.03s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestList2846947953\001\volumes\metadata.db: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/volume/service TestRestore (0.02s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestRestore3368254142\001\volumes\metadata.db: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/daemon/graphdriver TestIsEmptyDir (0.00s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestIsEmptyDir2823795693\001\dir-with-empty-file\file2625561089: The process cannot access the file because it is being used by another process.

    === FAIL: github.com/docker/docker/pkg/directory TestSizeFileAndNestedDirectoryNonempty (0.00s)
        testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeFileAndNestedDirectoryNonempty4246252950\001\nested3442260313\file21164327: The process cannot access the file because it is being used by another process.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-05-31 21:53:38 +02:00
Akihiro Suda
0afc71fc90
archive: add human-readable hint to Lchown error
Before:
```
$ docker pull gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v0.7.0
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: lchown /usr/local/bin/docker-credential-gcr: invalid argument
```

After:
```
$ docker pull gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v0.7.0
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: failed to Lchown "/usr/local/bin/docker-credential-gcr" for UID 205001, GID 5000:
lchown /usr/local/bin/docker-credential-gcr: invalid argument (try increasing the number of subordinate IDs in /etc/subuid and /etc/subgid)
```

For issue 43576

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2022-05-30 11:27:08 +09:00
Akihiro Suda
a04e3326e7
Fix constant WARNING: No swap limit support on cgroup v2 hosts
Fix issue 43646

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2022-05-27 10:51:54 +09:00
Djordje Lukic
70dc392bfa
Use hashicorp/go-memdb instead of truncindex
memdb already knows how to search by prefix so there is no need to keep
a separate list of container ids in the truncindex

Benchmarks:

$ go test -benchmem -run=^$ -count 5 -tags linux -bench ^BenchmarkDBGetByPrefix100$ github.com/docker/docker/container
goos: linux
goarch: amd64
pkg: github.com/docker/docker/container
cpu: Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
BenchmarkDBGetByPrefix100-6        16018             73935 ns/op           33888 B/op       1100 allocs/op
BenchmarkDBGetByPrefix100-6        16502             73150 ns/op           33888 B/op       1100 allocs/op
BenchmarkDBGetByPrefix100-6        16218             74014 ns/op           33856 B/op       1100 allocs/op
BenchmarkDBGetByPrefix100-6        15733             73370 ns/op           33792 B/op       1100 allocs/op
BenchmarkDBGetByPrefix100-6        16432             72546 ns/op           33744 B/op       1100 allocs/op
PASS
ok      github.com/docker/docker/container      9.752s

$ go test -benchmem -run=^$ -count 5 -tags linux -bench ^BenchmarkTruncIndexGet100$ github.com/docker/docker/pkg/truncindex
goos: linux
goarch: amd64
pkg: github.com/docker/docker/pkg/truncindex
cpu: Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
BenchmarkTruncIndexGet100-6        16862             73732 ns/op           44776 B/op       1173 allocs/op
BenchmarkTruncIndexGet100-6        16832             73629 ns/op           45184 B/op       1179 allocs/op
BenchmarkTruncIndexGet100-6        17214             73571 ns/op           45160 B/op       1178 allocs/op
BenchmarkTruncIndexGet100-6        16113             71680 ns/op           45360 B/op       1182 allocs/op
BenchmarkTruncIndexGet100-6        16676             71246 ns/op           45056 B/op       1184 allocs/op
PASS
ok      github.com/docker/docker/pkg/truncindex 9.759s

$ go test -benchmem -run=^$ -count 5 -tags linux -bench ^BenchmarkDBGetByPrefix500$ github.com/docker/docker/container
goos: linux
goarch: amd64
pkg: github.com/docker/docker/container
cpu: Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
BenchmarkDBGetByPrefix500-6         1539            753541 ns/op          169381 B/op       5500 allocs/op
BenchmarkDBGetByPrefix500-6         1624            749975 ns/op          169458 B/op       5500 allocs/op
BenchmarkDBGetByPrefix500-6         1635            761222 ns/op          169298 B/op       5500 allocs/op
BenchmarkDBGetByPrefix500-6         1693            727856 ns/op          169297 B/op       5500 allocs/op
BenchmarkDBGetByPrefix500-6         1874            710813 ns/op          169570 B/op       5500 allocs/op
PASS
ok      github.com/docker/docker/container      6.711s

$ go test -benchmem -run=^$ -count 5 -tags linux -bench ^BenchmarkTruncIndexGet500$ github.com/docker/docker/pkg/truncindex
goos: linux
goarch: amd64
pkg: github.com/docker/docker/pkg/truncindex
cpu: Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
BenchmarkTruncIndexGet500-6         1934            780328 ns/op          224073 B/op       5929 allocs/op
BenchmarkTruncIndexGet500-6         1713            713935 ns/op          225011 B/op       5937 allocs/op
BenchmarkTruncIndexGet500-6         1780            702847 ns/op          224090 B/op       5943 allocs/op
BenchmarkTruncIndexGet500-6         1736            711086 ns/op          224027 B/op       5929 allocs/op
BenchmarkTruncIndexGet500-6         2448            508694 ns/op          222322 B/op       5914 allocs/op
PASS
ok      github.com/docker/docker/pkg/truncindex 6.877s

Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
2022-05-20 18:22:21 +02:00
Cory Snider
c609523a8c pkg/filenotify: delete unused package
Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-05-19 15:23:18 -04:00
Kir Kolyshkin
8a5c13155e all: use unix.ByteSliceToString for utsname fields
This also fixes the GetOperatingSystem function in
pkg/parsers/operatingsystem which mistakenly truncated utsname.Machine
to the index of \0 in utsname.Sysname.

Fixes: 7aeb3efcb4
Cc: Tobias Klauser <tklauser@distanz.ch>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2022-05-18 17:13:20 -07:00
Eng Zer Jun
7873c27cfb
all: replace strings.Replace with strings.ReplaceAll
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-05-09 19:45:40 +08:00
Sebastiaan van Stijn
ea1eb449b7
daemon: killWithSignal, killPossiblyDeadProcess: accept syscall.Signal
This helps reducing some type-juggling / conversions further up
the stack.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-05-05 00:53:52 +02:00
Eng Zer Jun
36049a04d2
test: use T.Setenv to set env vars in tests
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-23 17:44:16 +08:00
Sebastiaan van Stijn
0e4f473a9f
pkg/chrootarchive: remove redundant init() stub for Windows
The package already has some windows files (so it's not empty), and
this init was not needed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 12:16:23 +02:00
Sebastiaan van Stijn
5f89a6a78e
pkg/urlutil: deprecate, and move to builder/remotecontext/urlutil
pkg/urlutil (despite its poorly chosen name) is not really intended as a generic
utility to handle URLs, and should only be used by the builder to handle (remote)
build contexts.

- IsURL() only does a very rudimentary check for http(s):// prefixes, without any
  other validation, but due to its name may give incorrect expectations.
- IsGitURL() is written specifically with docker build remote git contexts in
  mind, and has handling for backward-compatibility, where strings that are
  not URLs, but start with "github.com/" are accepted.

Because of the above, this patch:

- moves the package inside builder/remotecontext, close to where it's intended
  to be used (ideally this would be part of build/remotecontext itself, but this
  package imports many other dependencies, which would introduce those as extra
  dependencies in the CLI).
- deprecates pkg/urlutil, but adds aliases as there are some external consumers.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-12 19:58:05 +02:00
Sebastiaan van Stijn
074bc1c3ab
pkg/urlutil: remove unused IsTransportURL()
This function is no longer used (either internally, or externally), so
can be removed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-12 19:57:32 +02:00
CrazyMax
5d9e99ead3
missing windows arm64 arch detection
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-04-09 16:10:42 +02:00
Sebastiaan van Stijn
0a3336fd7d
Merge pull request #43366 from corhere/finish-identitymapping-refactor
Finish refactor of UID/GID usage to a new struct
2022-03-25 14:51:05 +01:00
Sebastiaan van Stijn
54eeff6eb3
Merge pull request #43385 from thaJeztah/move_IsWindowsClient
pkg/system: remove deprecated/unused consts and move IsWindowsClient()
2022-03-18 15:29:32 +01:00
Sebastiaan van Stijn
9bf40d7edd
pkg/system: move IsWindowsClient to pkg/parsers/operatingsystem
This function was only used in a single place, and pkg/parsers/operatingsystem
already copied the `verNTWorkstation` const, so we might as well move this function
there as well to "unclutter" pkg/system.

The function had no external users, so not adding an alias / stub.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-17 10:26:50 +01:00
Sebastiaan van Stijn
5d10c6ec67
Update handling of deprecated kernel (tcp) memory options
- Omit `KernelMemory` and `KernelMemoryTCP` fields in `/info` response if they're
  not supported, or when using API v1.42 or up.
- Re-enable detection of `KernelMemory` (as it's still needed for older API versions)
- Remove warning about kernel memory TCP in daemon logs (a warning is still returned
  by the `/info` endpoint, but we can consider removing that).
- Prevent incorrect "Minimum kernel memory limit allowed" error if the value was
  reset because it's not supported by the host.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-17 09:56:39 +01:00
aiordache
af6307fbda
Remove KernelMemory option from /containers/create and /update endpoints
- remove KernelMemory option from `v1.42` api docs
 - remove KernelMemory warning on `/info`
 - update changes for `v1.42`
 - remove `KernelMemory` field from endpoints docs

Signed-off-by: aiordache <anca.iordache@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-17 09:55:36 +01:00
Akihiro Suda
7f99438ce7
Merge pull request #43194 from thaJeztah/bump_golang_1.18
Update golang to 1.18.0
2022-03-17 12:52:16 +09:00
Sebastiaan van Stijn
85c4d633db
pkg/system: remove deprecated (and unused) windows consts
These consts were deprecated in 46c591b045, and
although that has not been in a release yet (we usually deprecate for at least
one release before removing), doing a search showed that there were no external
consumers of these consts, so it should be fine to remove them.

This patch removes the consts that were moded to pkg/idtools;

- SeTakeOwnershipPrivilege
- ContainerAdministratorSidString
- ContainerUserSidString

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-16 12:42:57 +01:00
Sebastiaan van Stijn
1133d55770
Merge pull request #43321 from corhere/43284-report-displayversion
pkg/parsers: support Windows 11: report DisplayVersion; drop ProductName
2022-03-16 12:41:50 +01:00
Sebastiaan van Stijn
4203a97aad
staticcheck: ignore "SA1019: strings.Title is deprecated"
This function is marked deprecated in Go 1.18; however, the suggested replacement
brings in a large amount of new code, and most strings we generate will be ASCII,
so this would only be in case it's used for some user-provided string. We also
don't have a language to use, so would be using the "default".

Adding a `//nolint` comment to suppress the linting failure instead.

    daemon/logger/templates/templates.go:23:14: SA1019: strings.Title is deprecated: The rule Title uses for word boundaries does not handle Unicode punctuation properly. Use golang.org/x/text/cases instead. (staticcheck)
        "title":    strings.Title,
                    ^
    pkg/plugins/pluginrpc-gen/template.go:67:9: SA1019: strings.Title is deprecated: The rule Title uses for word boundaries does not handle Unicode punctuation properly. Use golang.org/x/text/cases instead. (staticcheck)
        return strings.Title(s)
               ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-16 12:11:54 +01:00
Cory Snider
098a44c07f Finish refactor of UID/GID usage to a new struct
Finish the refactor which was partially completed with commit
34536c498d, passing around IdentityMapping structs instead of pairs of
[]IDMap slices.

Existing code which uses []IDMap relies on zero-valued fields to be
valid, empty mappings. So in order to successfully finish the
refactoring without introducing bugs, their replacement therefore also
needs to have a useful zero value which represents an empty mapping.
Change IdentityMapping to be a pass-by-value type so that there are no
nil pointers to worry about.

The functionality provided by the deprecated NewIDMappingsFromMaps
function is required by unit tests to to construct arbitrary
IdentityMapping values. And the daemon will always need to access the
mappings to pass them to the Linux kernel. Accommodate these use cases
by exporting the struct fields instead. BuildKit currently depends on
the UIDs and GIDs methods so we cannot get rid of them yet.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-03-14 16:28:57 -04:00
Sebastiaan van Stijn
8539d06209
Merge pull request #43347 from corhere/42452-sysinfo-remove-libcontainer
pkg/sysinfo: remove libcontainer dependency
2022-03-09 18:25:00 +01:00
Cory Snider
b0b71dbe1c pkg/sysinfo: remove libcontainer dependency
Reimplement GetCgroupMounts using the github.com/containerd/cgroups and
github.com/moby/sys/mountinfo packages.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-03-07 18:09:09 -05:00
Sebastiaan van Stijn
d89bfac728
remove deprecated pkg/symlink
This package was deprecated in dc3c382b34, which
was part of the 20.10 release, so consumers of this package should've been
able to migrate to the new location.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-06 00:24:22 +01:00
Sebastiaan van Stijn
7449ca078b
remove deprecated pkg/locker
This package was deprecated in 5ca758199d, which
was part of the 20.10 release, so consumers of this package should've been
able to migrate to the new location.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-06 00:20:46 +01:00
Sebastiaan van Stijn
4a6dc85e40
remove deprecated pkg/term, pkg/term/windows
This package was deprecated in 41d4112e89, which
was part of the 20.10 release, so consumers of this package should've been
able to migrate to the new location.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-06 00:18:46 +01:00
Sebastiaan van Stijn
97a235196e
remove deprecated pkg/mount
This package was deprecated in 99beb2ca02, which
was part of the 20.10 release, so consumers of this package should've been
able to migrate to the new location.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-06 00:13:05 +01:00
Cory Snider
9aacaeb667 pkg/parsers: support Windows 11; drop ProductName
Microsoft has stopped updating the ProductName registry value in Windows
11; it reads as Windows 10. And Microsoft has made it very difficult to
look up the real product name programmatically so that applications do
not attempt to parse it. (Ever wonder why they skipped Windows 9?) The
only documented and supported mechanisms require WMI or WinRT. The
product name has no bearing on application compatibility so it is not
worth doing any heroics to display the correct name. The build number
and Update Build Revision is sufficient information to identify a
specific build of Windows. Stop displaying the ProductName so as not to
confuse users with incorrect information.

Microsoft has frozen the ReleaseId registry value at 2009 when they
switched to semi-annual releases and alpha-numeric versions. The release
version as displayed by winver.exe and Settings -> System -> About on
Windows 20H2 and newer can be found in the new DisplayVersion registry
value. Replicate the way winver.exe displays the version by
preferentially reporting the DisplayVersion if present and reporting if
it is a Windows Server edition.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-03-04 16:08:34 -05:00
Sebastiaan van Stijn
dee9f422c8
pkg/system: remove github.com/docker/go-units dependency
This is not "very" important, but this dependency was only used
for a single const, which could be satisfied with a comment.

Not very urgent, as github.com/docker/go-units is likely imported
through other ways already (but it's nice to have the package be
more isolated).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-03 00:22:32 +01:00
Sebastiaan van Stijn
25ee00c494
pkg/system: move EnsureRemoveAll() to pkg/containerfs
pkg/system historically has been a bit of a kitchen-sink of things that were
somewhat "system" related, but didn't have a good place for. EnsureRemoveAll()
is one of those utilities. EnsureRemoveAll() is used to both unmount and remove
a path, for which it depends on both github.com/moby/sys/mount, which in turn
depends on github.com/moby/sys/mountinfo.

pkg/system is imported in the CLI, but neither EnsureRemoveAll(), nor any of its
moby/sys dependencies are used on the client side, so let's move this function
somewhere else, to remove those dependencies from the CLI.

I looked for plausible locations that were related; it's used in:

- daemon
- daemon/graphdriver/XXX/
- plugin

I considered moving it into a (e.g.) "utils" package within graphdriver (but not
a huge fan of "utils" packages), and given that it felt (mostly) related to
cleaning up container filesystems, I decided to move it there.

Some things to follow-up on after this:

- Verify if this function is still needed (it feels a bit like a big hammer in
  a "YOLO, let's try some things just in case it fails")
- Perhaps it should be integrated in `containerfs.Remove()` (so that it's used
  automatically)
- Look if there's other implementations (and if they should be consolidated),
  although (e.g.) the one in containerd is a copy of ours:
  https://github.com/containerd/containerd/blob/v1.5.9/pkg/cri/server/helpers_linux.go#L200

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-03 00:22:26 +01:00
Sebastiaan van Stijn
c72c1ca62c
Merge pull request #43185 from corhere/42402-safer-fileinfo
Remove local fork of archive/tar package
2022-02-24 10:36:14 +01:00
Sebastiaan van Stijn
705f9b68cc
some cleaning up of isolation checks, and platform information
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 22:58:37 +01:00
Cory Snider
833139f390 pkg/archive: audit gosec file-traversal lints
The recently-upgraded gosec linter has a rule for archive extraction
code which may be vulnerable to directory traversal attacks, a.k.a. Zip
Slip. Gosec's detection is unfortunately prone to false positives,
however: it flags any filepath.Join call with an argument derived from a
tar.Header value, irrespective of whether the resultant path is used for
filesystem operations or if directory traversal attacks are guarded
against.

All of the lint errors reported by gosec appear to be false positives.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-02-18 15:42:22 -05:00
Cory Snider
e9bbc41dd1 Remove local fork of archive/tar package
A copy of Go's archive/tar packge was vendored with a patch applied to
mitigate CVE-2019-14271. Vendoring standard library packages is not
supported by Go in module-aware mode, which is getting in the way of
maintenance. A different approach to mitigate the vulnerability is
needed which does not involve vendoring parts of the standard library.

glibc implements name service lookups such as users, groups and DNS
using a scheme known as Name Service Switch. The services are
implemented as modules, shared libraries which glibc dynamically links
into the process the first time a function requiring the module is
called. This is the crux of the vulnerability: if a process linked
against glibc chroots, then calls one of the functions implemented with
NSS for the first time, glibc may load NSS modules out of the chrooted
filesystem.

The API underlying the `docker cp` command is implemented by forking a
new process which chroots into the container's rootfs and writes a tar
stream of files from the container over standard output. It utilizes the
Go standard library's archive/tar package to write the tar stream. It
makes use of the tar.FileInfoHeader function to construct a tar.Header
value from an fs.FileInfo value. In modern versions of Go on *nix
platforms, FileInfoHeader will attempt to resolve the file's UID and GID
to their respective user and group names by calling the os/user
functions LookupId and LookupGroupId. The cgo implementation of os/user
on *nix performs lookups by calling the corresponding libc functions. So
when linked against glibc, calls to tar.FileInfoHeader after the
process has chrooted into the container's rootfs can have the side
effect of loading NSS modules from the container! Without any
mitigations, a malicious container image author can trivially get
arbitrary code execution by leveraging this vulnerability and escape the
chroot (which is not a sandbox) into the host.

Mitigate the vulnerability without patching or forking archive/tar by
hiding the OS-dependent file info from tar.FileInfoHeader which it needs
to perform the lookups. Without that information available it falls back
to populating the tar.Header with only the information obtainable
directly from the FileInfo value without making any calls into os/user.

Fixes #42402

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-02-18 13:40:19 -05:00
Sebastiaan van Stijn
5c8d50a132
Merge pull request #43210 from tianon/frozen-names-generator
Freeze the namesgenerator package against new additions
2022-02-10 20:38:44 +01:00
Tianon Gravi
624b3cfbe8 Freeze the namesgenerator package against new additions
See the added comment/documentation within the package for more details.

Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2022-02-10 09:56:08 -08:00
Sebastiaan van Stijn
347a85787b
pkg/namesgenerator: remove names-generator binary
This binary was added in 0800650665, but no motivation,
other than "this could be nice". Searching if it's used by anyone, I found only
two occurrences, both of which were just examples, or indexing results;

- eae845e40e/tree-examples/docker.txt (L1229)
- 8b1e0b89c7/sync/cache/librariesioGoGo.json (L2200)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-04 10:37:19 +01:00
Sebastiaan van Stijn
ab9796b0e9
pkg/system: rewrite IsWindowsClient() using golang.org/x/sys/windows
Looks like we should be able to use the implementation from x/sys/windows.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-26 13:39:03 +01:00
Sebastiaan van Stijn
be463cbd6c
pkg/system: remove unused HasWin32KSupport()
This was added in 194eaa5c0f to check image
compatibility based on Platform.Features;

    // For now, hard code that all base images except nanoserver depend on win32k support
    if imageData.Name != "nanoserver" {
        imageData.OSFeatures = append(imageData.OSFeatures, "win32k")
    }

But no longer used since 1f59bc8c03 and
d231260868

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-26 13:11:58 +01:00
Aaron Lehmann
40bb983175
Merge pull request #43095 from aaronlehmann/avoid-regexp-simple-cases
fileutils: Avoid compiling a regexp for simple patterns
2022-01-20 17:43:07 -08:00
Aaron Lehmann
38e2e756c9 Avoid platform-specific NewPatternMatcher function in TestCompile
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2022-01-20 09:08:16 -08:00
Aaron Lehmann
8d250be008 Add unit test for (*Pattern).compile
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2022-01-12 10:31:35 -08:00
Sebastiaan van Stijn
e202ac3f38
remove pkg/discovery as it's now unused
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-06 18:28:28 +01:00
Tobias Klauser
cfd26afabe
Use syscall.Timespec.Unix
Use the syscall method instead of repeating the type conversions for
the syscall.Stat_t Atim/Mtim members. This also allows to drop the
//nolint: unconvert comments.

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
2022-01-03 16:51:02 +01:00
Aaron Lehmann
dd66dcad9c fileutils: Avoid compiling a regexp for simple patterns
If we detect that a pattern is either an exact match, prefix match, or
suffix match, use an optimized code path instead of compiling a regexp.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-12-20 16:32:13 -08:00
Tonis Tiigi
26dafe43a4 system: unbreak build for darwin
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-12-15 22:15:07 -08:00
Aaron Lehmann
55da5245de Fix missing parent info case in MatchesUsingParentResults
Unfortunately, this check was missing in the original version. It could
cause a positive match to be overwritten by checking parent dirs.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-11-26 10:22:19 -08:00
Aaron Lehmann
0f1b68df16 Adjust deprecation comments
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-11-24 07:53:10 -08:00
Aaron Lehmann
4555d3aa54 pkg/fileutils: Track incremental pattern match results against each pattern
The existing code does not correctly handle the case where a file
matches one of the patterns, but should not match overall because of an
exclude pattern that applied to a parent directory (see
https://github.com/docker/buildx/issues/850).

Fix this by independently tracking the results of matching against each
pattern. A file should be considered to match any pattern that matched a
parent dir.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-11-19 15:58:13 -08:00
Sebastiaan van Stijn
89a3c427aa
Merge pull request #42961 from milas/fix-path-match-escape
pkg/fileutils: escape additional regex meta characters
2021-10-28 20:25:21 +02:00
Akihiro Suda
4283e93e64
Merge pull request #41215 from cpuguy83/better_plugin_errors
Add more error details on plugin get
2021-10-24 12:37:07 +09:00
Milas Bowman
195558de5a pkg/fileutils: escape additional regex meta characters
There were a couple characters being explicitly escaped, but it
wasn't comprehensive.

This is now the set difference between the Golang regex meta
characters and the `filepath` match meta characters with the
exception of `\`, which already has special logic due to being
the path separator on Windows.

Signed-off-by: Milas Bowman <milasb@gmail.com>
2021-10-22 11:31:11 -04:00
Sebastiaan van Stijn
872c64cd66
Merge pull request #42862 from dkkb/feature/zstd_with_skippable_frame
compression: support zstd with skippable frame
2021-10-21 20:29:12 +02:00
Da McGrady
23abee412b
compression: support zstd with skippable frame
As a matter of fact, there are two frame formats defined by Zstandard: Zstandard frames and Skippable frames.
So we should probably support zstd algorithms with skippable frames.
See https://tools.ietf.org/id/draft-kucherawy-dispatch-zstd-00.html#rfc.section.2 for more details.

Signed-off-by: Da McGrady <dabkb@aol.com>
2021-10-15 17:23:55 +08:00
Tonis Tiigi
cec4e69813
chrootarchive: don't create parent dirs outside of chroot
If chroot is used with a special root directory then create
destination directory within chroot. This works automatically
already due to extractor creating parent paths and is only
used currently with cp where parent paths are actually required
and error will be shown to user before reaching this point.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 52d285184068998c22632bfb869f6294b5613a58)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 80f1169eca)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-10-05 09:57:04 +02:00
Artem Khramov
f3d3994a4b
Port pkg/system/mknod.go to FreeBSD
Because FreeBSD uses 64-bit device nodes (see
https://reviews.freebsd.org/rS318736), Linux implementation of
`system.Mknod` & `system.Mkdev` is not sufficient.

This change adds freebsd-specific implementations for `Mknod` and
Mkdev`.

Signed-off-by: Artem Khramov <akhramov@pm.me>
2021-09-22 09:47:35 +03:00
Akihiro Suda
6014c1e29d
Merge pull request #41759 from giuseppe/zstd-compression
compression: add support for the zstd algorithm
2021-09-17 17:01:45 +09:00
Giuseppe Scrivano
e187eb2bb5
compression: add support for the zstd algorithm
zstd is a compression algorithm that has a very fast decoder, while
providing also good compression ratios.  The fast decoder makes it
suitable for container images, as decompressing the tarballs is a very
expensive operation.

https://github.com/opencontainers/image-spec/pull/788 added support
for zstd to the OCI image specs.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2021-09-16 17:03:47 +02:00
Sebastiaan van Stijn
f586a473cf
pkg/namesgenerator: replace uses of fmt.Sprintf()
Looks like we don't need sprintf for how it's used. Replacing sprintf makes it
more performant (~2.4x as fast), and less memory, allocations:

    BenchmarkGetRandomName-8      	 8203230	       142.4 ns/op	      37 B/op	       2 allocs/op
    BenchmarkGetRandomNameOld-8   	 3499509	       342.9 ns/op	      85 B/op	       5 allocs/op

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-09-14 15:15:29 +02:00
Sebastiaan van Stijn
8fa6126f75
Merge pull request #42543 from rainrambler/patch-1
use defer to unlock mutex
2021-09-01 13:26:30 +02:00
Brian Goff
94e00d09da
Merge pull request #42797 from thaJeztah/go116_compat
pkg/plugins: fix compatibility with go1.16
2021-08-31 08:19:59 -07:00
Sebastiaan van Stijn
303ea8e820
pkg/plugins: fix compatibility with go1.16
commit c55a4ac779 changed the ioutil utilities
to use the new os variants, per recommendation from the go 1.16 release notes:
https://golang.org/doc/go1.16#ioutil

> we encourage new code to use the new definitions in the io and os packages.
> Here is a list of the new locations of the names exported by io/ioutil:

However, the devil is in the detail, and io.ReadDir() is not a direct
replacement for ioutil.ReadDir();

> ReadDir => os.ReadDir (note: returns a slice of os.DirEntry rather than a slice of fs.FileInfo)

go1.16 added a io.FileInfoToDirEntry() utility to concert a DirEntry to
a FileInfo, but it's not available in go1.16

This patch copies the FileInfoToDirEntry code, and uses it for go1.16.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-31 15:42:54 +02:00
Akihiro Suda
fecf45b09a
Merge pull request #42796 from thaJeztah/containerd_seccomp_check
pkg/sysinfo: use containerd/pkg/seccomp.IsEnabled()
2021-08-29 03:05:59 +09:00
Sebastiaan van Stijn
accec292c1
pkg/sysinfo: use containerd/pkg/seccomp.IsEnabled()
This replaces the local SeccompSupported() utility for the implementation in containerd,
which performs the same check.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-27 15:21:52 +02:00
rainrambler
7ce810c771
use defer to unlock mutex
use defer to unlock mutex (clean up)

Signed-off-by: Anyu Wang <wanganyu@outlook.com>
Signed-off-by: rainrambler <wanganyu@outlook.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-27 14:39:46 +02:00
Eng Zer Jun
c55a4ac779
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-27 14:56:57 +08:00
Sebastiaan van Stijn
686be57d0a
Update to Go 1.17.0, and gofmt with Go 1.17
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-24 23:33:27 +02:00
Sebastiaan van Stijn
8207c05cfc
Merge pull request #41479 from olljanat/ci-win-containerd-support
Windows CI: Add support for testing with containerd
2021-08-24 22:29:14 +02:00
Brian Goff
a44a8e54ce
Merge pull request #42717 from thaJeztah/move_defaults 2021-08-24 09:33:22 -07:00
Sebastiaan van Stijn
f68260ba22
Dockerfile: frozen images: update to bullseye, remove buildpack-dep
Update the frozen images to also be based on Debian bullseye. Using the "slim"
variant (which looks to have all we're currently using),  and remove the
buildpack-dep frozen image.

The buildpack-dep image is quite large, and it looks like we only use it to
compile some C binaries, which should work fine on a regular debian image;

    docker build -t debian:bullseye-slim-gcc -<<EOF
    FROM debian:bullseye-slim
    RUN apt-get update && apt-get install -y gcc libc6-dev --no-install-recommends
    EOF

    docker image ls

    REPOSITORY        TAG                 IMAGE ID       CREATED          SIZE
    debian            bullseye-slim-gcc   1851750242af   About a minute ago   255MB
    buildpack-deps    bullseye            fe8fece98de2   2 days ago           834MB

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-20 18:23:06 +02:00
Sebastiaan van Stijn
98caf09f0f
fix go-digest to make sure the algorithms are registered
Previously, ioutils imported the crypty/sha256 package, because it was
used by the HashData() utility. As a side-effect of that import, the
sha256 algorithm was registered through its `init()` function.

Now that the HashData() utility is removed, the import is no longer needed
in this package, but some parts of our code depended on the side-effect, and
without this, it fail to recognise the algorithms, unless something else
happens to import crypto/sha256 / crypto/sha512, which made our
tests fail:

```
=== Failed
=== FAIL: reference TestLoad (0.00s)
    store_test.go:53: failed to parse reference: unsupported digest algorithm

=== FAIL: reference TestSave (0.00s)
    store_test.go:82: failed to parse reference: unsupported digest algorithm

=== FAIL: reference TestAddDeleteGet (0.00s)
    store_test.go:174: could not parse reference: unsupported digest algorithm

=== FAIL: reference TestInvalidTags (0.00s)
    store_test.go:355: assertion failed: error is not nil: unsupported digest algorithm
```

While it would be better to do the import in the actual locations where it's
expected, there may be code-paths we overlook, so instead adding the import
here temporarily. Until the PR in go-digest has been merged and released.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-20 00:35:41 +02:00
Sebastiaan van Stijn
572498be56
move pkg/ioutils.HashData() to libnetwork/resolvconf
It's the only location it's used, so we might as well move it there.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-20 00:35:39 +02:00
Olli Janatuinen
1285c6d125 Windows CI: Add support for testing with containerd
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2021-08-17 07:09:40 -07:00
Aaron Lehmann
ba2adeebdb
Merge pull request #42676 from aaronlehmann/patternmatcher-doublestar-bug
fileutils: Fix incorrect handling of "**/foo" pattern
2021-08-16 19:58:55 -07:00
Aaron Lehmann
c44b90f3bf Test fix for Windows compatibility
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-08-12 20:03:41 -07:00
Aaron Lehmann
97ede9df26 Rename Matches to MatchesOrParentMatches
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-08-12 18:10:04 -07:00
Aaron Lehmann
9bae4f2f24 Add more optimal MatchesUsingParentResult method, use it in pkg/archive
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-08-12 13:57:50 -07:00
Sebastiaan van Stijn
e53f65a916
pkg/signal: remove DefaultStopSignal const
This const was previously living in pkg/signal, but with that package
being moved to its own module, it didn't make much sense to put docker's
defaults in a generic module.

The const from the "signal" package is currenlty used *both* by the CLI
and the daemon as a default value when creating containers. This put up
some questions:

a. should the default be non-exported, and private to the container
   package? After all, it's a _default_ (so should be used if _NOT_ set).
b. should the client actually setting a default, or instead just omit
   the value, unless specified by the user? having the client set a
   default also means that the daemon cannot change the default value
   because the client (or older clients) will override it.
c. consider defaults from the client and defaults of the daemon to be
   separate things, and create a default const in the CLI.

This patch implements option "a" (option "b" will be done separately,
as it involves the CLI code). This still leaves "c" open as an option,
if the CLI wants to set its own default.

Unfortunately, this change means we'll have to drop the alias for the
deprecated pkg/signal.DefaultStopSignal const, but a comment was left
instead, which can assist consumers of the const to find why it's no
longer there (a search showed the Docker CLI as the only consumer though).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-11 10:31:29 +02:00
Samuel Karp
8b43104937
Merge pull request #42672 from thaJeztah/pkg_archive_debuglogs 2021-08-05 12:04:59 -07:00
Brian Goff
51b06c6795
Merge pull request #42683 from thaJeztah/remove_lcow_step6
Remove LCOW (step 6)
2021-07-29 11:34:29 -07:00
Brian Goff
ad268e79c4
Merge pull request #42193 from lzhfromustc/3_23
discovery & test: Fix goroutine leaks by adding 1 buffer to channel
2021-07-28 15:25:37 -07:00
Sebastiaan van Stijn
e3491ab56a
pkg/archive: gzDecompress(): use local vars for MOBY_DISABLE_PIGZ
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-28 00:11:48 +02:00
Sebastiaan van Stijn
b321474747
pkg/archive: DetectCompression(): use bytes.HasPrefix()
The existing code was the exact equivalent of bytes.HasPrefix();

    // HasPrefix tests whether the byte slice s begins with prefix.
    func HasPrefix(s, prefix []byte) bool {
    	return len(s) >= len(prefix) && Equal(s[0:len(prefix)], prefix)
    }

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-28 00:11:46 +02:00
Sebastiaan van Stijn
ebeda658bc
pkg/archive: remove some debug logs
These seemed fairly redundant

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-28 00:11:43 +02:00
Sebastiaan van Stijn
13cb04e57c
remove various LCOW bits (container, image, pkg/containerfs)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 13:36:21 +02:00
Sebastiaan van Stijn
0998c7e363
system: remove LCOWSupported() utility
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 13:36:19 +02:00
Aaron Lehmann
90f8d1b675 fileutils: Fix incorrect handling of "**/foo" pattern
(*PatternMatcher).Matches includes a special case for when the pattern
matches a parent dir, even though it doesn't match the current path.
However, it assumes that the parent dir which would match the pattern
must have the same number of separators as the pattern itself. This
doesn't hold true with a patern like "**/foo". A file foo/bar would have
len(parentPathDirs) == 1, which is less than the number of path
len(pattern.dirs) == 2... therefore this check would be skipped.

Given that "**/foo" matches "foo", I think it's a bug that the "parent
subdir matches" check is being skipped in this case.

It seems safer to loop over the parent subdirs and check each against
the pattern. It's possible there is a safe optimization to check only a
certain subset, but the existing logic seems unsafe.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-07-26 11:28:10 -07:00
Sebastiaan van Stijn
28409ca6c7
replace pkg/signal with moby/sys/signal v0.5.0
This code was moved to the moby/sys repository

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-23 09:32:54 +02:00
Brian Goff
9a6ff685a8
Merge pull request #42641 from thaJeztah/make_signal_selfcontained 2021-07-19 14:46:15 -07:00
Justin Cormack
b337c70bdc
Merge pull request #42639 from thaJeztah/system_info_clean
pkg/sysinfo: assorted cleanup/refactoring for handling warnings and logging
2021-07-19 15:17:07 +01:00
Justin Cormack
fb21a1e474
Merge pull request #42580 from thaJeztah/reduce_TestClientWithRequestTimeout_flakiness
Reduce TestClientWithRequestTimeout flakiness
2021-07-19 15:11:43 +01:00
Sebastiaan van Stijn
6ff6913ac4
pkg/signal: remove gotest.tools dependency
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-15 18:24:23 +02:00
Sebastiaan van Stijn
0880df4644
pkg/signal: move Trap() to cmd/dockerd
It's the only location where this is used, and it's quite specific
to dockerd (not really a reusable function for external use), so
moving it into that package.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-15 18:11:00 +02:00
Sebastiaan van Stijn
ea5c94cdb9
pkg/signal: move signal.DumpStacks() to a separate package
It is not directly related to signal-handling, so can well live
in its own package.

Also added a variant that doesn't take a directory to write files
to, for easier consumption / better match to how it's used.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-15 18:09:43 +02:00
Sebastiaan van Stijn
4ace1998e5
pkg/sysinfo: use correct name for AppArmor in t.Skip()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 23:10:13 +02:00
Sebastiaan van Stijn
9b795c3e50
pkg/sysinfo.New(), daemon.RawSysInfo(): remove "quiet" argument
The "quiet" argument was only used in a single place (at daemon startup), and
every other use had to pass "false" to prevent this function from logging
warnings.

Now that SysInfo contains the warnings that occurred when collecting the
system information, we can make leave it up to the caller to use those
warnings (and log them if wanted).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 23:10:07 +02:00
Sebastiaan van Stijn
1fb62f455c
pkg/sysinfo: collect warnings in SysInfo struct
This allows the warnings to be consumed in other locations.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 17:28:25 +02:00
Sebastiaan van Stijn
208d3c6efb
pkg/sysinfo: move cg2Controllers to be a field in SysInfo and unify v1/v2
We pass the SysInfo struct to all functions. Adding cg2Controllers as a
(non-exported) field makes passing around this information easier.

Now that infoCollector and infoCollectorV2 have the same signature, we can
simplify some bits and use a single slice for all "collectors".

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:39:44 +02:00
Sebastiaan van Stijn
5cc20ad9e5
pkg/sysinfo: adjust Opt to set new field
This removes the need to have the opts type.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:39:26 +02:00
Sebastiaan van Stijn
ca27b473cc
pkg/sysinfo: move cg2GroupPath to be a field in SysInfo
We pass the SysInfo struct to all functions. Adding cg2GroupPath as a
(non-exported) field makes passing around this information easier.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:37:03 +02:00
Sebastiaan van Stijn
e70c5ea1a9
pkg/sysinfo.newV2() remove redundant path.Clean()
path.Join() already does path.Clean(), and the opts.cg2GroupPath
field is already cleaned as part of WithCgroup2GroupPath()

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:37:02 +02:00
Sebastiaan van Stijn
6677ab6a63
pkg/sysinfo: move cgMounts to be a field in SysInfo
We pass the SysInfo struct to all functions. Adding cgMounts as a
(non-exported) field makes passing around this information easier.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:37:00 +02:00
Sebastiaan van Stijn
10ce0d84c2
pkg/sysinfo.New() move v1 code to a newV1() function
This makes it clearer that this code is the cgroups v1 equivalent of newV2().

Also moves the "options" handling to newV2() because it's currently only used
for cgroupsv2.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-14 16:36:56 +02:00
jk-vb
1d055267a8
updated names-generator.go for alphabetization
re ordered some entries so they are in proper alphabetical order

Signed-off-by: Justin Keller <jkeller@vb-jkeller-mbp.local>
2021-07-13 23:52:51 +02:00
Sebastiaan van Stijn
00a4f67ddf
Reduce TestClientWithRequestTimeout flakiness
The test sometimes failed because no error was returned:

    === Failed
    === FAIL: pkg/plugins TestClientWithRequestTimeout (0.00s)
         client_test.go:254: assertion failed: expected an error, got nil: expected error

Possibly caused by a race condition, as the sleep was just 1 ms longer than the timeout;
this patch is increasing the sleep in the response to try to reduce flakiness.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-29 23:53:23 +02:00
Sebastiaan van Stijn
301be64d41
Merge pull request #42515 from geaaru/permit-override-name
ReplaceFileTarWrapper: permit to override file name
2021-06-24 21:01:08 +02:00
Sebastiaan van Stijn
46c591b045
pkg/system: deprecate some consts and move them to pkg/idtools
These consts were used in combination with idtools utilities, which
makes it a more logical location for these consts to live.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 23:16:15 +02:00
Sebastiaan van Stijn
26f5db7a1d
pkg/system: remove unused system.Unmount() utility
On Linux/Unix it was just a thin wrapper for unix.Unmount(), and a no-op on Windows.

This function was not used anywhere (also not externally), so removing this without
deprecating first.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 23:09:56 +02:00
Sebastiaan van Stijn
5f04517c48
pkg/system: remove deprecated GetOSVersion(), consts, SecurityInfo utils.
This removes the deprecated wrappers, so that the package no longer has
hcsshim as a dependency. These wrappers were no longer used in our code,
and were deprecated in the 20.10 release (giving external consumers to
replace the deprecated ones).

Note that there are two consts which were unused, but for which there is
no replacement in golang.org/x/sys;

    const (
        PROCESS_TRUST_LABEL_SECURITY_INFORMATION = 0x00000080
        ACCESS_FILTER_SECURITY_INFORMATION       = 0x00000100
    )

PROCESS_TRUST_LABEL_SECURITY_INFORMATION is documented as "reserved", and I could
not find clear documentation about ACCESS_FILTER_SECURITY_INFORMATION, so not sure
if they must be included in golang.org/x/sys: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/23e75ca3-98fd-4396-84e5-86cd9d40d343

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 23:09:55 +02:00
Sebastiaan van Stijn
472f21b923
replace uses of deprecated containerd/sys.RunningInUserNS()
This utility was moved to a separate package, which has no
dependencies.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 11:01:24 +02:00
Daniele Rondina
73aef6edfe
ReplaceFileTarWrapper: permit to override file name
Signed-off-by: Daniele Rondina <geaaru@sabayonlinux.org>
2021-06-11 11:00:12 +02:00
Sebastiaan van Stijn
bb17074119
reformat "nolint" comments
Unlike regular comments, nolint comments should not have a leading space.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:42 +02:00
Sebastiaan van Stijn
7b071e0557
pkg/archive: RebaseArchiveEntries(): ignore G110
pkg/archive/copy.go:357:16: G110: Potential DoS vulnerability via decompression bomb (gosec)
                if _, err = io.Copy(rebasedTar, srcTar); err != nil {
                            ^

Ignoring GoSec G110. See https://github.com/securego/gosec/pull/433
and https://cure53.de/pentest-report_opa.pdf, which recommends to
replace io.Copy with io.CopyN7. The latter allows to specify the
maximum number of bytes that should be read. By properly defining
the limit, it can be assured that a GZip compression bomb cannot
easily cause a Denial-of-Service.
After reviewing, this should not affect us, because here we do not
read into memory.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:36 +02:00
Sebastiaan van Stijn
f77213efc2
gosimple: S1039: unnecessary use of fmt.Sprintf
pkg/devicemapper/devmapper.go:383:28: S1039: unnecessary use of fmt.Sprintf (gosimple)
        if err := task.setMessage(fmt.Sprintf("@cancel_deferred_remove")); err != nil {
                                  ^
    integration/plugin/graphdriver/external_test.go:321:18: S1039: unnecessary use of fmt.Sprintf (gosimple)
                http.Error(w, fmt.Sprintf("missing id"), 409)
                              ^
    integration-cli/docker_api_stats_test.go:70:31: S1039: unnecessary use of fmt.Sprintf (gosimple)
            _, body, err := request.Get(fmt.Sprintf("/info"))
                                        ^
    integration-cli/docker_cli_build_test.go:4547:19: S1039: unnecessary use of fmt.Sprintf (gosimple)
                "--build-arg", fmt.Sprintf("FOO1=fromcmd"),
                               ^
    integration-cli/docker_cli_build_test.go:4548:19: S1039: unnecessary use of fmt.Sprintf (gosimple)
                "--build-arg", fmt.Sprintf("FOO2="),
                               ^
    integration-cli/docker_cli_build_test.go:4549:19: S1039: unnecessary use of fmt.Sprintf (gosimple)
                "--build-arg", fmt.Sprintf("FOO3"), // set in env
                               ^
    integration-cli/docker_cli_build_test.go:4668:32: S1039: unnecessary use of fmt.Sprintf (gosimple)
            cli.WithFlags("--build-arg", fmt.Sprintf("tag=latest")))
                                         ^
    integration-cli/docker_cli_build_test.go:4690:32: S1039: unnecessary use of fmt.Sprintf (gosimple)
            cli.WithFlags("--build-arg", fmt.Sprintf("baz=abc")))
                                         ^
    pkg/jsonmessage/jsonmessage_test.go:255:4: S1039: unnecessary use of fmt.Sprintf (gosimple)
                fmt.Sprintf("ID: status\n"),
                ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:27 +02:00
Sebastiaan van Stijn
6b0ecacd92
gosec: G404: Use of weak random number generator
These should be ok to ignore for the purpose they're used

    pkg/namesgenerator/names-generator.go:843:36: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
        name := fmt.Sprintf("%s_%s", left[rand.Intn(len(left))], right[rand.Intn(len(right))])
                                          ^
    pkg/namesgenerator/names-generator.go:849:36: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
            name = fmt.Sprintf("%s%d", name, rand.Intn(10))
                                             ^
    testutil/stringutils.go:11:18: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
            b[i] = letters[rand.Intn(len(letters))]
                           ^
    pkg/namesgenerator/names-generator.go:849:36: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
            name = fmt.Sprintf("%s%d", name, rand.Intn(10))
                                             ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:18 +02:00
Sebastiaan van Stijn
bc8acc5b08
pkg/fileutils: PatternMatcher.Matches(): remove debug logging
Trying to avoid logging code in "libraries" used elsewhere.
If this debug log is important, it should be easy to add in code
that's calling it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-09 11:56:00 +02:00
Brian Goff
a7ea29a5a6
Merge pull request #42451 from thaJeztah/remove_lcow_step1
Remove LCOW code (step 1)
2021-06-08 13:41:45 -07:00
Jakub Guzik
7ef6ece774 Fix setting swaplimit=true without checking
Signed-off-by: Jakub Guzik <jakubmguzik@gmail.com>
2021-06-03 22:54:10 +02:00
Sebastiaan van Stijn
e047d984dc
Remove LCOW code (step 1)
The LCOW implementation in dockerd has been deprecated in favor of re-implementation
in containerd (in progress). Microsoft started removing the LCOW V1 code from the
build dependencies we use in Microsoft/opengcs (soon to be part of Microsoft/hcshhim),
which means that we need to start removing this code.

This first step removes the lcow graphdriver, the LCOW initialization code, and
some LCOW-related utilities.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-03 21:16:21 +02:00
Sebastiaan van Stijn
05f520dd3c
pkg/signal: ignore SIGURG on all platforms
Other Unix platforms (e.g. Darwin) are also affected by the Go
runtime sending SIGURG.

This patch changes how we match the signal by just looking for the
"URG" name, which should handle any platform that has this signal
defined in the SignalMap.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-05-25 12:07:53 +02:00
Brian Goff
328d23f625
Merge pull request #42390 from thaJeztah/skip_defunct_test
pkg/fileutils: TestMatches: remove cases no longer valid for go1.16
2021-05-20 11:10:46 -07:00
Sebastiaan van Stijn
b7ebf32ba3
pkg/signal.CatchAll: ignore SIGURG on Linux
Do not handle SIGURG on Linux, as in go1.14+, the go runtime issues
SIGURG as an interrupt to support preemptable system calls on Linux.

This issue was caught in TestCatchAll, which could fail when updating to Go 1.14 or above;

    === Failed
    === FAIL: pkg/signal TestCatchAll (0.01s)
        signal_linux_test.go:32: assertion failed: urgent I/O condition (string) != continued (string)
        signal_linux_test.go:32: assertion failed: continued (string) != hangup (string)
        signal_linux_test.go:32: assertion failed: hangup (string) != child exited (string)
        signal_linux_test.go:32: assertion failed: child exited (string) != illegal instruction (string)
        signal_linux_test.go:32: assertion failed: illegal instruction (string) != floating point exception (string)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-05-19 19:42:18 +02:00
Sebastiaan van Stijn
2842639e0e
pkg/fileutils: TestMatches: remove cases no longer valid for go1.16
These tests were no longer valid on Go 1.16; related to https://tip.golang.org/doc/go1.16#path/filepath

> The Match and Glob functions now return an error if the unmatched part of
> the pattern has a syntax error. Previously, the functions returned early on
> a failed match, and thus did not report any later syntax error in the pattern.

Causing the test to fail:

    === RUN   TestMatches
        fileutils_test.go:388: assertion failed: error is not nil: syntax error in pattern: pattern="a\\" text="a"
    --- FAIL: TestMatches (0.00s)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-05-18 09:37:13 +02:00
Sebastiaan van Stijn
8f4b3b0ad4
pkg/archive temporarily skip some tests on Windowsd V19H1 (1903) and up
These tests fail, possibly due to changes in the kernel. Temporarily skipping
these tests, so that we at least have some coverage on these windows versions
in this repo, and we can look into this specific issue separately.;

    === FAIL: github.com/docker/docker/pkg/archive TestChangesDirsEmpty (0.21s)
        changes_test.go:261: Reported changes for identical dirs: [{\dirSymlink C}]

    === FAIL: github.com/docker/docker/pkg/archive TestChangesDirsMutated (0.14s)
        changes_test.go:391: unexpected change "C \\dirSymlink" "\\dirnew"

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-09 01:07:27 +02:00
Akihiro Suda
6322dfc217
archive: do not use overlayWhiteoutConverter for UserNS
overlay2 no longer sets `archive.OverlayWhiteoutFormat` when
running in UserNS, so we can remove the complicated logic in the
archive package.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-03-29 14:47:12 +09:00
lzhfromustc
5ffcd162b5 discovery & test: Fix goroutine leaks by adding 1 buffer to channel
Signed-off-by: Ziheng Liu <lzhfromustc@gmail.com>
2021-03-24 10:32:39 -04:00
Chris Kreussling (Flatbush Gardener)
7024564c48 Correct name (right) sort order
Move "montalcini" to between "mirzakhani" and "moore"

Signed-off-by: Chris Kreussling (Flatbush Gardener) <xrisfg@gmail.com>
2021-03-09 08:22:14 -05:00
Shengjing Zhu
f23c1c297d pkg/archive: TestUntarParentPathPermissions requires root
=== RUN   TestUntarParentPathPermissions
    archive_unix_test.go:171: assertion failed: error is not nil: chown /tmp/TestUntarParentPathPermissions694189715/foo: operation not permitted

Signed-off-by: Shengjing Zhu <zhsj@debian.org>
2021-02-27 21:53:16 +08:00
Sebastiaan van Stijn
328de0b8d9
Update documentation links
- Using "/go/" redirects for some topics, which allows us to
  redirect to new locations if topics are moved around in the
  documentation.
- Updated some old URLs to their new location.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-25 12:11:50 +01:00
Sebastiaan van Stijn
25ada76437
pkg/archive: Unpack() use 0755 permissions for missing directories
Commit edb62a3ace fixed a bug in MkdirAllAndChown()
that caused the specified permissions to not be applied correctly. As a result
of that bug, the configured umask would be applied.

When extracting archives, Unpack() used 0777 permissions when creating missing
parent directories for files that were extracted.
Before edb62a3ace, this resulted in actual
permissions of those directories to be 0755 on most configurations (using a
default 022 umask).

Creating these directories should not depend on the host's umask configuration.
This patch changes the permissions to 0755 to match the previous behavior,
and to reflect the original intent of using 0755 as default.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-17 16:24:52 +01:00
Sebastiaan van Stijn
01ae718aef
Merge pull request #41984 from tonistiigi/pax-parent
archive: avoid creating parent dirs for XGlobalHeader
2021-02-12 17:58:08 +01:00
Grant Millar
2ad187fd4a Fix userns-remap option when username & UID match
Signed-off-by: Grant Millar <rid@cylo.io>
2021-02-10 15:58:34 +00:00
Tonis Tiigi
ba7906aef3 archive: avoid creating parent dirs for XGlobalHeader
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-02-04 18:38:51 -08:00
Brian Goff
66dffbec86
Ensure MkdirAllAndChown also sets perms
Generally if we ever need to change perms of a dir, between versions,
this ensures the permissions actually change when we think it should
change without having to handle special cases if it already existed.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit edb62a3ace)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-02 13:01:20 +01:00
Sebastiaan van Stijn
e422445418
Merge pull request #41892 from AkihiroSuda/fix-41803
pkg/archive: allow mknodding FIFO inside userns
2021-01-28 08:26:05 +01:00
Tibor Vass
d5209b29b9
Merge pull request #41927 from tiborvass/execabs
Use golang.org/x/sys/execabs
2021-01-26 09:15:54 -08:00
Akihiro Suda
d5d5cccb7e
pkg/archive: allow mknodding FIFO inside userns
Fix #41803

Also attempt to mknod devices.
Mknodding devices are likely to fail, but still worth trying when
running with a seccomp user notification.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-01-26 17:43:25 +09:00
Tibor Vass
7ca0cb7ffa Use golang.org/x/sys/execabs
Signed-off-by: Tibor Vass <tibor@docker.com>
2021-01-25 19:13:12 +00:00
Eric Mountain
95eb490780
Use v2 capabilities in layer archives
When building images in a user-namespaced container, v3 capabilities are
stored including the root UID of the creator of the user-namespace.

This UID does not make sense outside the build environment however. If
the image is run in a non-user-namespaced runtime, or if a user-namespaced
runtime uses a different UID, the capabilities requested by the effective
bit will not be honoured by `execve(2)` due to this mismatch.

Instead, we convert v3 capabilities to v2, dropping the root UID on the
fly.

Signed-off-by: Eric Mountain <eric.mountain@datadoghq.com>
2020-12-23 14:16:41 +01:00
Arnaud Rebillout
c7e7426796 pkg/chrootarchive: Skip privileged tests when non-root
These tests fail when run by a non-root user

  === RUN   TestUntarWithMaliciousSymlinks
      archive_unix_test.go:63: assertion failed: expected error to contain "open /safe/host-file: no such file or directory", got "Error processing tar file(exit status 1): Error creating mount namespace before pivot: operation not permitted"
  --- FAIL: TestUntarWithMaliciousSymlinks (0.00s)
  === RUN   TestTarWithMaliciousSymlinks
      archive_unix_test.go:90: /tmp/TestTarWithMaliciousSymlinks400408188
  === RUN   TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root/safe_host-file
      archive_unix_test.go:155: assertion failed: error is not nil: error processing tar file: Error after fallback to chroot: operation not permitted: exit status 1
  === RUN   TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root/safe/_host-file
      archive_unix_test.go:155: assertion failed: error is not nil: error processing tar file: Error after fallback to chroot: operation not permitted: exit status 1
  === RUN   TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root/safe_
      archive_unix_test.go:155: assertion failed: error is not nil: error processing tar file: Error after fallback to chroot: operation not permitted: exit status 1
  === RUN   TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root/safe/_
      archive_unix_test.go:155: assertion failed: error is not nil: error processing tar file: Error after fallback to chroot: operation not permitted: exit status 1
  === RUN   TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root_safe/host-file
      archive_unix_test.go:155: assertion failed: error is not nil: error processing tar file: Error after fallback to chroot: operation not permitted: exit status 1
  === RUN   TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root_/safe/host-file
      archive_unix_test.go:155: assertion failed: error is not nil: error processing tar file: Error after fallback to chroot: operation not permitted: exit status 1
  === RUN   TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root_
      archive_unix_test.go:155: assertion failed: error is not nil: error processing tar file: Error after fallback to chroot: operation not permitted: exit status 1
  --- FAIL: TestTarWithMaliciousSymlinks (0.02s)
      --- FAIL: TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root/safe_host-file (0.00s)
      --- FAIL: TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root/safe/_host-file (0.00s)
      --- FAIL: TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root/safe_ (0.00s)
      --- FAIL: TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root/safe/_ (0.00s)
      --- FAIL: TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root_safe/host-file (0.00s)
      --- FAIL: TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root_/safe/host-file (0.00s)
      --- FAIL: TestTarWithMaliciousSymlinks//tmp/TestTarWithMaliciousSymlinks400408188/root_ (0.00s)
  FAIL
  FAIL	github.com/docker/docker/pkg/chrootarchive	0.042s

Signed-off-by: Arnaud Rebillout <elboulangero@gmail.com>
2020-12-15 09:47:48 +07:00
Sebastiaan van Stijn
be67385209
Merge pull request #41626 from cuonglm/master
pkg/devicemapper: fix invalid usage of reflect.SliceHeader
2020-11-16 23:05:50 +01:00
Sebastiaan van Stijn
2b41e84719
pkg/system: fix compile on darwin (macOS)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-10 20:49:09 +01:00
Sebastiaan van Stijn
0e8023ddea
Merge pull request #41638 from TBBle/37352-fix-Windows-CI-pipeline
Reinstate Windows CI Pipeline usefulness
2020-11-10 02:52:54 +01:00
Sebastiaan van Stijn
6458f750e1
use containerd/cgroups to detect cgroups v2
libcontainer does not guarantee a stable API, and is not intended
for external consumers.

this patch replaces some uses of libcontainer/cgroups with
containerd/cgroups.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-09 15:00:32 +01:00
Paul "TBBle" Hampson
a158b53d86 Separate non-Windows mount code from common code
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-08 23:15:15 +11:00
Sebastiaan van Stijn
dc3c382b34
replace pkg/symlink with github.com/moby/sys/symlink
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-03 11:17:12 +01:00
Cuong Manh Le
c208f03fbd
pkg/devicemapper: fix invalid usage of reflect.SliceHeader
The current usage of reflect.SliceHeader violates rule 6th of
unsafe.Pointer conversion. In short, reflect.SliceHeader could not be
used as plain struct.

See https://golang.org/pkg/unsafe/#Pointer

Signed-off-by: Cuong Manh Le <cuong.manhle.vn@gmail.com>
2020-11-03 14:20:52 +07:00
Sebastiaan van Stijn
e766361271
pkg/mount: update deprecated wrappers
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-29 23:04:20 +01:00
Sebastiaan van Stijn
32d506b394
vendor: moby/sys mountinfo/v0.4.0
full diff: https://github.com/moby/sys/compare/mountinfo/v0.1.3...mountinfo/v0.4.0

> Note that this dependency uses submodules, providing "github.com/moby/sys/mount"
> and "github.com/moby/sys/mountinfo". Our vendoring tool (vndr) currently doesn't
> support submodules, so we vendor the top-level moby/sys repository (which contains
> both) and pick the most recent tag, which could be either `mountinfo/vXXX` or
> `mount/vXXX`.

github.com/moby/sys/mountinfo v0.4.0
--------------------------------------------------------------------------------

Breaking changes:

- `PidMountInfo` is now deprecated and will be removed before v1.0; users should switch to `GetMountsFromReader`

Fixes and improvements:

- run filter after all fields are parsed
- correct handling errors from bufio.Scan
- documentation formatting fixes

github.com/moby/sys/mountinfo v0.3.1
--------------------------------------------------------------------------------

- mount: use MNT_* flags from golang.org/x/sys/unix on freebsd
- various godoc and CI fixes
- mountinfo: make GetMountinfoFromReader Linux-specific
- Add support for OpenBSD in addition to FreeBSD
- mountinfo: use idiomatic naming for fields

github.com/moby/sys/mountinfo v0.2.0
--------------------------------------------------------------------------------

Bug fixes:

- Fix path unescaping for paths with double quotes

Improvements:

- Mounted: speed up by adding fast paths using openat2 (Linux-only) and stat
- Mounted: relax path requirements (allow relative, non-cleaned paths, symlinks)
- Unescape fstype and source fields
- Documentation improvements

Testing/CI:

- Unit tests: exclude darwin
- CI: run tests under Fedora 32 to test openat2
- TestGetMounts: fix for Ubuntu build system
- Makefile: fix ignoring test failures
- CI: add cross build

github.com/moby/sys/mount v0.1.1
--------------------------------------------------------------------------------

https://github.com/moby/sys/releases/tag/mount%2Fv0.1.1

Improvements:

- RecursiveUnmount: add a fast path (#26)
- Unmount: improve doc
- fix CI linter warning on Windows

Testing/CI:

- Unit tests: exclude darwin
- Makefile: fix ignoring test failures
- CI: add cross build

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-29 23:02:28 +01:00
fanjiyun
1b8826beee docker stats: fix 'panic: close of closed channel'
Signed-off-by: fanjiyun <fan.jiyun@zte.com.cn>
2020-10-24 11:48:56 +08:00
Tianon Gravi
654cad4d9d Also trim "~..." from AppArmor versions
Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2020-10-08 17:03:51 -07:00
Akihiro Suda
ee079e4692
pkg/aaparser: support parsing version like "3.0.0-beta1"
Fix #41517

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-10-02 17:33:52 +09:00
Kir Kolyshkin
2502db66d0 pkg/system: make EnsureRemoveAll unix-specific
The tricks performed by EnsureRemoveAll only make sense for Linux and
other Unices, so separate it out, and make EnsureRemoveAll for Windows
just an alias of os.RemoveAll.

This makes sure RecursiveUnmount is not called on Windows.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2020-09-21 01:23:28 -07:00
Brian Goff
f99814d749
Merge pull request #41448 from iamleot/stat_bsd
Unify same fromStatT() implementation for FreeBSD and NetBSD
2020-09-15 16:53:08 -07:00
Sebastiaan van Stijn
1155b6bc7a
Merge pull request #41395 from cpuguy83/no_libseccomp
Remove dependency in dockerd on libseccomp
2020-09-15 17:37:04 +02:00
Leonardo Taccari
3d9120f744 Unify same fromStatT() implementation for FreeBSD and NetBSD
Suggested by Sebastiaan van Stijn, thanks!

Signed-off-by: Leonardo Taccari <leot@NetBSD.org>
2020-09-14 18:33:04 +02:00
Leonardo Taccari
6efa9e0575 Add support for NetBSD
Signed-off-by: Leonardo Taccari <leot@NetBSD.org>
2020-09-12 14:28:29 +02:00
Brian Goff
df7031b669 Memoize seccomp value for SysInfo
As it turns out, we call this function every time someone calls `docker
info`, every time a contianer is created, and every time a container is
started.
Certainly this should be refactored as a whole, but for now, memoize the
seccomp value.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-09-11 22:48:46 +00:00
Sebastiaan van Stijn
5ca758199d
replace pkg/locker with github.com/moby/locker
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-09-10 22:15:40 +02:00
Sebastiaan van Stijn
ea9886cec4
pkg/idtools: refactor to avoid string-splitting
The package used a lot of string-formatting, followed by string-splitting.
This looked to originate from attempts to use templating to allow future
extensibility (9a3ab0358e).

Looking at the history of the package, only a single update was made to
these templates, 5 years go, which makes it unlikely that more templating
will be needed.

This patch simplifies the handling of arguments to use `[]string` instead
of a single `string` (and splitting to a `[]string`). This both simplifies
the code somewhat, and prevents user/group-names containing spaces to be
splitted (causing, e.g. `getent` to fail).

Note that user/group-names containing spaces are invalid (or at least
discouraged), there are situations where such names may be used, so we
should avoid breaking on such names.

Before this change, a user/group name with a space in its name would fail;

    dockerd --userns-remap="user:domain users"
    INFO[2020-08-19T10:26:59.288868661+02:00] Starting up
    Error during groupname lookup for "domain users": getent unable to find entry "domain" in group database

With this change:

    # Add some possibly problematic usernames for testing
    # need to do this manually, as `adduser` / `useradd` won't accept these names
    echo 'user name1002:1002::/home/one:/bin/false' >> /etc/passwd; \
    echo 'user name1002:' >> /etc/group; \
    echo 'user name:1266401166:65536' >> /etc/subuid; \
    echo 'user name:1266401153:65536' >> /etc/subgid; \
    echo 'user$HOME1003:1003::/home/one:/bin/false' >> /etc/passwd; \
    echo 'user$HOME1003:' >> /etc/group; \
    echo 'user$HOME:1266401166:65536' >> /etc/subuid; \
    echo 'user$HOME:1266401153:65536' >> /etc/subgid; \
    echo 'user'"'"'name1004:1004::/home/one:/bin/false' >> /etc/passwd; \
    echo 'user'"'"'name1004:' >> /etc/group; \
    echo 'user'"'"'name:1266401166:65536' >> /etc/subuid; \
    echo 'user'"'"'name:1266401153:65536' >> /etc/subgid; \
    echo 'user"name1005:1005::/home/one:/bin/false' >> /etc/passwd; \
    echo 'user"name1005:' >> /etc/group; \
    echo 'user"name:1266401166:65536' >> /etc/subuid; \
    echo 'user"name:1266401153:65536' >> /etc/subgid;

    # Start the daemon using those users
    dockerd --userns-remap="user name:user name"
    dockerd --userns-remap='user$HOME:user$HOME'
    dockerd --userns-remap="user'name":"user'name"
    dockerd --userns-remap='user"name':'user"name'

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-08-20 12:24:38 +02:00
Till Wegmüller
e3ab236309 Add solaris support to lstat
Signed-off-by: Till Wegmüller <toasterson@gmail.com>
2020-08-12 11:40:58 +02:00
Brian Goff
cadd72d37d
Merge pull request #41208 from thaJeztah/remove_shellwords 2020-07-17 15:46:24 -07:00
Brian Goff
260c26b7be
Merge pull request #41016 from kolyshkin/cgroup-init 2020-07-16 11:26:52 -07:00
Brian Goff
b1a3fe4934 Add more error details on plugin get
Seeing this `ErrNotImplements` in some of our logs and it's not very
helpful because we don't know what plugin is causing it or even what the
requested interface is.

```
{"message":"legacy plugin: Plugin does not implement the requested driver"}
```

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-07-15 13:10:30 -07:00
Sebastiaan van Stijn
3f36764844
Merge pull request #40994 from gaurav1086/rm_test_fix_goroutine_leak
rm_test: fix goroutine leak
2020-07-14 17:41:43 +02:00
Sebastiaan van Stijn
d4b7b4b0a7
Merge pull request #40941 from cpuguy83/down_with_the_func_init
Don't use init() to set unpigz path
2020-07-14 15:44:08 +02:00
Sebastiaan van Stijn
c9c9e0ab90
pkg/parsers: remove uses of mattn/go-shellwords
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-07-14 14:03:10 +02:00
Sebastiaan van Stijn
818e0b2fcf
pkg/parsers: add unit test for Darwin
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-07-14 14:02:54 +02:00
Kir Kolyshkin
afbeaf6f29 pkg/sysinfo: rm duplicates
The CPU CFS cgroup-aware scheduler is one single kernel feature, not
two, so it does not make sense to have two separate booleans
(CPUCfsQuota and CPUCfsPeriod). Merge these into CPUCfs.

Same for CPU realtime.

For compatibility reasons, /info stays the same for now.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2020-06-26 16:19:52 -07:00
Sebastiaan van Stijn
ab6b92b6b2
chrootarchive: fix "conversion from int to string yields a string of one rune"
update test to fix go 1.15 linting failure:

    pkg/chrootarchive/archive_test.go:103:32: conversion from int to string yields a string of one rune

relates to golang/go 32479

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-06-25 13:38:42 +02:00
Sebastiaan van Stijn
66bb1c4644
pkg/sysinfo: use containerd/sys to detect UserNamespaces
The implementation in libcontainer/system is quite complicated,
and we only use it to detect if user-namespaces are enabled.

In addition, the implementation in containerd uses a sync.Once,
so that detection (and reading/parsing `/proc/self/uid_map`) is
only performed once.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-06-15 13:07:48 +02:00
Sebastiaan van Stijn
9ff990a2b9
pkg/chrootarchive: use containerd/sys to detect UserNamespaces
The implementation in libcontainer/system is quite complicated,
and we only use it to detect if user-namespaces are enabled.

In addition, the implementation in containerd uses a sync.Once,
so that detection (and reading/parsing `/proc/self/uid_map`) is
only performed once.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-06-15 13:07:23 +02:00