The existing code does not correctly handle the case where a file
matches one of the patterns, but should not match overall because of an
exclude pattern that applied to a parent directory (see
https://github.com/docker/buildx/issues/850).
Fix this by independently tracking the results of matching against each
pattern. A file should be considered to match any pattern that matched a
parent dir.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
As a matter of fact, there are two frame formats defined by Zstandard: Zstandard frames and Skippable frames.
So we should probably support zstd algorithms with skippable frames.
See https://tools.ietf.org/id/draft-kucherawy-dispatch-zstd-00.html#rfc.section.2 for more details.
Signed-off-by: Da McGrady <dabkb@aol.com>
zstd is a compression algorithm that has a very fast decoder, while
providing also good compression ratios. The fast decoder makes it
suitable for container images, as decompressing the tarballs is a very
expensive operation.
https://github.com/opencontainers/image-spec/pull/788 added support
for zstd to the OCI image specs.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Update the frozen images to also be based on Debian bullseye. Using the "slim"
variant (which looks to have all we're currently using), and remove the
buildpack-dep frozen image.
The buildpack-dep image is quite large, and it looks like we only use it to
compile some C binaries, which should work fine on a regular debian image;
docker build -t debian:bullseye-slim-gcc -<<EOF
FROM debian:bullseye-slim
RUN apt-get update && apt-get install -y gcc libc6-dev --no-install-recommends
EOF
docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
debian bullseye-slim-gcc 1851750242af About a minute ago 255MB
buildpack-deps bullseye fe8fece98de2 2 days ago 834MB
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The existing code was the exact equivalent of bytes.HasPrefix();
// HasPrefix tests whether the byte slice s begins with prefix.
func HasPrefix(s, prefix []byte) bool {
return len(s) >= len(prefix) && Equal(s[0:len(prefix)], prefix)
}
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
pkg/archive/copy.go:357:16: G110: Potential DoS vulnerability via decompression bomb (gosec)
if _, err = io.Copy(rebasedTar, srcTar); err != nil {
^
Ignoring GoSec G110. See https://github.com/securego/gosec/pull/433
and https://cure53.de/pentest-report_opa.pdf, which recommends to
replace io.Copy with io.CopyN7. The latter allows to specify the
maximum number of bytes that should be read. By properly defining
the limit, it can be assured that a GZip compression bomb cannot
easily cause a Denial-of-Service.
After reviewing, this should not affect us, because here we do not
read into memory.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These tests fail, possibly due to changes in the kernel. Temporarily skipping
these tests, so that we at least have some coverage on these windows versions
in this repo, and we can look into this specific issue separately.;
=== FAIL: github.com/docker/docker/pkg/archive TestChangesDirsEmpty (0.21s)
changes_test.go:261: Reported changes for identical dirs: [{\dirSymlink C}]
=== FAIL: github.com/docker/docker/pkg/archive TestChangesDirsMutated (0.14s)
changes_test.go:391: unexpected change "C \\dirSymlink" "\\dirnew"
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
overlay2 no longer sets `archive.OverlayWhiteoutFormat` when
running in UserNS, so we can remove the complicated logic in the
archive package.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
=== RUN TestUntarParentPathPermissions
archive_unix_test.go:171: assertion failed: error is not nil: chown /tmp/TestUntarParentPathPermissions694189715/foo: operation not permitted
Signed-off-by: Shengjing Zhu <zhsj@debian.org>
Commit edb62a3ace fixed a bug in MkdirAllAndChown()
that caused the specified permissions to not be applied correctly. As a result
of that bug, the configured umask would be applied.
When extracting archives, Unpack() used 0777 permissions when creating missing
parent directories for files that were extracted.
Before edb62a3ace, this resulted in actual
permissions of those directories to be 0755 on most configurations (using a
default 022 umask).
Creating these directories should not depend on the host's umask configuration.
This patch changes the permissions to 0755 to match the previous behavior,
and to reflect the original intent of using 0755 as default.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fix#41803
Also attempt to mknod devices.
Mknodding devices are likely to fail, but still worth trying when
running with a seccomp user notification.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
When building images in a user-namespaced container, v3 capabilities are
stored including the root UID of the creator of the user-namespace.
This UID does not make sense outside the build environment however. If
the image is run in a non-user-namespaced runtime, or if a user-namespaced
runtime uses a different UID, the capabilities requested by the effective
bit will not be honoured by `execve(2)` due to this mismatch.
Instead, we convert v3 capabilities to v2, dropping the root UID on the
fly.
Signed-off-by: Eric Mountain <eric.mountain@datadoghq.com>
The implementation in libcontainer/system is quite complicated,
and we only use it to detect if user-namespaces are enabled.
In addition, the implementation in containerd uses a sync.Once,
so that detection (and reading/parsing `/proc/self/uid_map`) is
only performed once.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`func init()` is evil here, and the logrus calls are being made before
the logger is even setup.
It also means in order to use pigz you have to restart the daemon.
Instead this takes a small hit and resolves pigz on each extraction.
In the grand scheme of decompressing this is a very small hit.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Switch to moby/sys/mount and mountinfo. Keep the pkg/mount for potential
outside users.
This commit was generated by the following bash script:
```
set -e -u -o pipefail
for file in $(git grep -l 'docker/docker/pkg/mount"' | grep -v ^pkg/mount); do
sed -i -e 's#/docker/docker/pkg/mount"#/moby/sys/mount"#' \
-e 's#mount\.\(GetMounts\|Mounted\|Info\|[A-Za-z]*Filter\)#mountinfo.\1#g' \
$file
goimports -w $file
done
```
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It makes sense to use mount package here because
- it no longer requires /proc to be mounted
- it provides verbose errors so the caller doesn't have to
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There is a race condition in pkg/archive when using `cmd.Start` for pigz
and xz where the `*bufio.Reader` could be returned to the pool while the
command is still writing to it, and then picked up and used by a new
command.
The command is wrapped in a `CommandContext` where the process will be
killed when the context is cancelled, however this is not instantaneous,
so there's a brief window while the command is still running but the
`*bufio.Reader` was already returned to the pool.
wrapReadCloser calls `cancel()`, and then `readBuf.Close()` which
eventually returns the buffer to the pool. However, because cmdStream
runs `cmd.Wait` in a go routine that we never wait for to finish, it is
not safe to return the reader to the pool yet. We need to ensure we
wait for `cmd.Wait` to finish!
Signed-off-by: Stephen Benjamin <stephen@redhat.com>
also renamed the non-windows variant of this file to be
consistent with other files in this package
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As reported in docker/for-linux/issues/484, since Docker 18.06
docker cp with a destination file name fails with the following error:
> archive/tar: cannot encode header: Format specifies USTAR; and USTAR cannot encode Name="a_very_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_long_filename_that_is_101_characters"
The problem is caused by changes in Go 1.10 archive/tar, which
mis-guesses the tar stream format as USTAR (rather than PAX),
which, in turn, leads to inability to specify file names
longer than 100 characters.
This tar stream is sent by TarWithOptions() (which, since we switched to
Go 1.10, explicitly sets format=PAX for every file, see FileInfoHeader(),
and before Go 1.10 it was PAX by default). Unfortunately, the receiving
side, RebaseArchiveEntries(), which calls tar.Next(), mistakenly guesses
header format as USTAR, which leads to the above error.
The fix is easy: set the format to PAX in RebaseArchiveEntries()
where we read the tar stream and change the file name.
A unit test is added to prevent future regressions.
NOTE this code is not used by dockerd, but rather but docker cli
(also possibly other clients), so this needs to be re-vendored
to cli in order to take effect.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Ubuntu kernel supports overlayfs in user namespaces.
However, Docker had previously crafting overlay opaques directly
using mknod(2) and setxattr(2), which are not supported in userns.
Tested with LXD, Ubuntu 18.04, kernel 4.15.0-36-generic #39-Ubuntu.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Signed-off-by: John Howard <jhoward@microsoft.com>
If fixes an error in sameFsTime which was using `==` to compare two times. The correct way is to use go's built-in timea.Equals(timeb).
In changes_windows, it uses sameFsTime to compare mTim of a `system.StatT` to allow TestChangesDirsMutated to operate correctly now.
Note there is slight different between the Linux and Windows implementations of detecting changes. Due to https://github.com/moby/moby/issues/9874,
and the fix at https://github.com/moby/moby/pull/11422, Linux does not consider a change to the directory time as a change. Windows on NTFS
does. See https://github.com/moby/moby/pull/37982 for more information. The result in `TestChangesDirsMutated`, `dir3` is NOT considered a change
in Linux, but IS considered a change on Windows. The test mutates dir3 to have a mtime of +1 second.
With a handful of tests still outstanding, this change ports most of the unit tests under pkg/archive to Windows.
It provides an implementation of `copyDir` in tests for Windows. To make a copy similar to Linux's `cp -a` while preserving timestamps
and links to both valid and invalid targets, xcopy isn't sufficient. So I used robocopy, but had to circumvent certain exit codes that
robocopy exits with which are warnings. Link to article describing this is in the code.
