Otherwise a malformed or empty digest may cause a panic.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit a7d4af84bd)
Signed-off-by: Tibor Vass <tibor@docker.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 611eb6ffb3)
Renamed constant defaultAppArmorProfile to defaultApparmorProfile.
Signed-off-by: Tibor Vass <tibor@docker.com>
Various dirs in /var/lib/docker contain data that needs to be mounted
into a container. For this reason, these dirs are set to be owned by the
remapped root user, otherwise there can be permissions issues.
However, this uneccessarily exposes these dirs to an unprivileged user
on the host.
Instead, set the ownership of these dirs to the real root (or rather the
UID/GID of dockerd) with 0701 permissions, which allows the remapped
root to enter the directories but not read/write to them.
The remapped root needs to enter these dirs so the container's rootfs
can be configured... e.g. to mount /etc/resolve.conf.
This prevents an unprivileged user from having read/write access to
these dirs on the host.
The flip side of this is now any user can enter these directories.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit e908cc3901)
Cherry-pick conflict with eb14d936bf:
Kept old `container` variable name.
Signed-off-by: Tibor Vass <tibor@docker.com>
The remapped root does not need access to this dir.
Having this owned by the remapped root opens the host up to an
uprivileged user on the host being able to escalate privileges.
While it would not be normal for the remapped UID to be used outside of
the container context, it could happen.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit bfedd27259)
Signed-off-by: Tibor Vass <tibor@docker.com>
Generally if we ever need to change perms of a dir, between versions,
this ensures the permissions actually change when we think it should
change without having to handle special cases if it already existed.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit edb62a3ace)
Signed-off-by: Tibor Vass <tibor@docker.com>
Fix#41803
Also attempt to mknod devices.
Mknodding devices are likely to fail, but still worth trying when
running with a seccomp user notification.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit d5d5cccb7e)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
full diff: https://github.com/containerd/containerd/compare/v1.3.7...v1.3.8
Release notes:
containerd 1.3.8
----------------------
Welcome to the v1.3.8 release of containerd!
The eighth patch release for containerd 1.3 includes several bug fixes and updates.
Notable Updates
- Fix metrics monitoring of v2 runtime tasks
- Fix nil pointer error when restoring checkpoint
- Fix devmapper device deletion on rollback
- Fix integer overflow on Windows
- Update seccomp default profile
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The new shim socket path convention hardcodes `/run/containerd`:
https://github.com/containerd/containerd/pull/4343
`dockerd-rootless.sh` is updated to hide the rootful `/run/containerd`
from the mount namespace of the rootless dockerd.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 794aa20983)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Fix#41517
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit ee079e4692)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The cloud logging client should be closed when the log driver is closed. Otherwise dockerd will keep a gRPC connection to the logging endpoint open indefinitely.
This results in a slow leak of tcp sockets (1) and memory (~200Kb) any time that a container using `--log-driver=gcplogs` is terminates.
Signed-off-by: Patrick Haas <patrickhaas@google.com>
(cherry picked from commit ef553e14a4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The test was looking for the wrong file name.
Since compression happens asyncronously, sometimes the test would
succeed and sometimes fail.
This change makes sure to wait for the compressed version of the file
since we can't know when the compression is going to occur.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit c6d860ace6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
A nil interface in Go is not the same as a nil pointer that satisfies
the interface. libcontainer/user has special handling for missing
/etc/{passwd,group} files but this is all based on nil interface checks,
which were broken by Docker's usage of the API.
When combined with some recent changes in runc that made read errors
actually be returned to the caller, this results in spurrious -EINVAL
errors when we should detect the situation as "there is no passwd file".
Signed-off-by: Aleksa Sarai <asarai@suse.de>
(cherry picked from commit 3108ae6226)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 0a2b9b5464...db3c7e526a
- Use golang.org/x/sys/unix instead of syscall
- Set O_CLOEXEC when opening a network namespace
- Fixes "the container‘s netns fds leak, causing the container netns to not
clean up successfully after the container stops"
- Allows to create and delete named network namespaces
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 818bad6ef2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Same update as was vendored in e26e1cc5c1 on
master.
full diff: 7109fa855b...0a2b9b5464
- Add support for Go modules
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 4cb720ef64...a1e4f48e71
Brings in the cherry-picks from moby/buildkit#1596 and moby/buildkit#1598 :
- Add --force flag in git fetch command
- Fix socket handling during copy (Treat unix sockets as regular files)
- Remotecache: Only visit each item once when walking results.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
"Container's image" term is rather ambiguous: it can be both a name and an ID.
Looking at the sources[1], it's actually an image ID, so bring some clarity.
[1]: a6a47d1a49/daemon/inspect.go (L170)
Signed-off-by: Nikolay Edigaryev <edigaryev@gmail.com>
(cherry picked from commit c44fb42377)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change brings in a single new commit from Microsoft/hcsshim. The
commit fixes an issue when unpacking a Windows container layer which
could result in incorrect directory timestamps.
This manifested most significantly in an impact to startup times of
some Windows container images (such as anything based on servercore).
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
(cherry picked from commit 2865478487)
Signed-off-by: Ameya Gawde <agawde@mirantis.com>
full diff: 153d0769a1...026aabaa65
- Fix 'failed to get network during CreateEndpoint'
- log error instead if disabling IPv6 router advertisement failed
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
A newer runc changed [1] a couple of certain error messages checked in this
test to be lowercased, which lead to a mismatch in this test case.
Fix is to remove "The" (which was replaced with "the").
[1] https://github.com/opencontainers/runc/pull/2441
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 56de0489fc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The defer function was checking for the local `err` variable, not
on the error that was returned by the function. As a result, the
sandbox would never be cleaned up for containers that used "none"
networking, and a failiure occured during setup.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b98b8df886)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit e5ad8b14daf0a1ddb12c0b83d153531afffb908b)
Signed-off-by: Tibor Vass <tibor@docker.com>
full diff: https://github.com/checkpoint-restore/criu/compare/v3.13...v3.14
New features
- C/R of memfd memory mappings and file descriptors.
- Add time namespace support.
- Add the read pre-dump mode which uses process_vm_readv.
- Add --cgroup-yard option
- Add support of the cgroup v2 freezer.
- Add support of opened O_PATH fds.
Bugfixes
- Fix C/R ia32 processes on AMD
- Fix cross-compilation
- Many fixes here and there
Improvements
- Use clone3() with set_tid to restore processes
- Clean up compel headers.
- Use the new mount API
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a342010823)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Building gotestsum started to fail after the repository removed some
dependencies on master.
What happens is that first, we `go get` the package (with go modules disabled);
GO111MODULE=off go get -d gotest.tools/gotestsum
Which gets the latest version from master, and fetches the dependencies used
on master. Then we checkout the version we want to install (for example `v0.3.5`)
and run go build.
However, `v0.3.5` depends on logrus, and given that we ran `go get` for `master`,
that dependency was not fetched, and build fails.
This patch modifies the installer to use go modules (alternatively we could
probably run `go get .` after checking out the `v0.3.5` version),
We need to modify all installers, as it looks like this is a standard pattern
we use, but other dependencies were not failing (yet), so this patch only
addresses the immediate failure.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 1d9da1b233)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
slirp4netns v0.3.X turned out not to work with RootlessKit >= v0.7.1:
https://github.com/rootless-containers/rootlesskit/issues/143
As slirp4netns v0.3.X reached EOL on Mar 31, 2020, RootlessKit is not
going to fix support for slirp4netns v0.3.X.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit c86abee1a4)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Commit 12c7541f1f updated the
opencontainers/selinux dependency to v1.3.1, which had a breaking
change in the errors that were returned.
Before v1.3.1, the "raw" `syscall.ENOTSUP` was returned if the
underlying filesystem did not support xattrs, but later versions
wrapped the error, which caused our detection to fail.
This patch uses `errors.Is()` to check for the underlying error.
This requires github.com/pkg/errors v0.9.1 or above (older versions
could use `errors.Cause()`, but are not compatible with "native"
wrapping of errors in Go 1.13 and up, and could potentially cause
these errors to not being detected again.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 49f8a4224c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This prevents getting into a situation where a container log cannot make
progress because we tried to rotate a file, got an error, and now the
file is closed. The next time we try to write a log entry it will try
and rotate again but error that the file is already closed.
I wonder if there is more we can do to beef up this rotation logic.
Found this issue while investigating missing logs with errors in the
docker daemon logs like:
```
Failed to log message for json-file: error closing file: close <file>:
file already closed
```
I'm not sure why the original rotation failed since the data was no
longer available.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 3989f91075)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
full diff: https://github.com/checkpoint-restore/criu/compare/v3.12...v3.13
Here we have some bugfixes, huuuge *.py patch for coding style
and nice set of new features like 32bit for ARM, TLS for page
server and new mode for CGroups.
New features
- VDSO: arm32 support
- Add TLS support for page server communications
- "Ignore" mode for --manage-cgroups
- Restore SO_BROADCAST option for inet sockets
Bugfixes
- Auxiliary events were left in inotify queues
- Lazy-pages daemon didn't detect stack pages and surrounders properly and marked them as "lazy"
- Memory and resource leakage were detected by coverity, cppcheck and clang
Improvements
- Use gettimeofday() directly from vdso for restore timings
- Reformat all .py code into pep8 style
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f508db4833)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- relates to moby/buildkit 1111
- relates to moby/buildkit 1079
- relates to docker/buildx 129
full diff: 9461782956...e31b211e4f
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e7183dbfe9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The pages that were linked to have moved, so changing the
links to point to docs.docker.com instead.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e9348898d3)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The TEST_FILTER variable allows running a single integration or integration-cli
test. However, it failed to work properly for integration-cli tests.
Before:
-----------
# Filtering "integration" tests works:
make TEST_FILTER=TestInspectCpusetInConfigPre120 test-integration
...
DONE 1 tests in 18.331s
# But running a single test in "integration-cli" did not:
make TEST_FILTER=TestSwarmNetworkCreateIssue27866 test-integration
...
DONE 0 tests in 17.314s
Trying to manually add the `/` prefix, didn't work either, because that made the
"grep" fail to find which test-suites to run/skip:
make TEST_FILTER=/TestSwarmNetworkCreateIssue27866 test-integration
---> Making bundle: test-integration (in bundles/test-integration)
make: *** [test-integration] Error 1
After:
-----------
make TEST_FILTER=TestInspectCpusetInConfigPre120 test-integration
...
DONE 1 tests in 18.331s
make TEST_FILTER=TestSwarmNetworkCreateIssue27866 test-integration
...
DONE 12 tests in 26.527s
Note that the `12` tests is still a bit misleading, because every _suite_ is
started (which is counted as a test), but no tests are run. This is still
something that could be improved on.
This patch also makes a small modification to the code that's setting
`integration_api_dirs`, and no longer runs `go list` if not needed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e7805653b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/golang/go/compare/go1.13.6...go1.13.7
go1.13.7 (released 2020/01/28) includes two security fixes. One mitigates
the CVE-2020-0601 certificate verification bypass on Windows. The other affects
only 32-bit architectures.
https://github.com/golang/go/issues?q=milestone%3AGo1.13.7+label%3ACherryPickApproved
- X.509 certificate validation bypass on Windows 10
A Windows vulnerability allows attackers to spoof valid certificate chains when
the system root store is in use. These releases include a mitigation for Go
applications, but it’s strongly recommended that affected users install the
Windows security update to protect their system.
This issue is CVE-2020-0601 and Go issue golang.org/issue/36834.
- Panic in crypto/x509 certificate parsing and golang.org/x/crypto/cryptobyte
On 32-bit architectures, a malformed input to crypto/x509 or the ASN.1 parsing
functions of golang.org/x/crypto/cryptobyte can lead to a panic.
The malformed certificate can be delivered via a crypto/tls connection to a
client, or to a server that accepts client certificates. net/http clients can
be made to crash by an HTTPS server, while net/http servers that accept client
certificates will recover the panic and are unaffected.
Thanks to Project Wycheproof for providing the test cases that led to the
discovery of this issue. The issue is CVE-2020-7919 and Go issue golang.org/issue/36837.
This is also fixed in version v0.0.0-20200124225646-8b5121be2f68 of golang.org/x/crypto/cryptobyte.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 878db479be)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
go1.13.5 (released 2019/12/04) includes fixes to the go command, the runtime, the
linker, and the net/http package. See the Go 1.13.5 milestone on our issue tracker
for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.13.5+label%3ACherryPickApproved
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a218e9b7b0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/golang/go/compare/go1.13...go1.13.1
```
Hi gophers,
We have just released Go 1.13.1 and Go 1.12.10 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.1).
net/http (through net/textproto) used to accept and normalize invalid HTTP/1.1 headers with a space before the colon, in violation of RFC 7230. If a Go server is used behind an uncommon reverse proxy that accepts and forwards but doesn't normalize such invalid headers, the reverse proxy and the server can interpret the headers differently. This can lead to filter bypasses or request smuggling, the latter if requests from separate clients are multiplexed onto the same upstream connection by the proxy. Such invalid headers are now rejected by Go servers, and passed without normalization to Go client applications.
The issue is CVE-2019-16276 and Go issue golang.org/issue/34540.
Thanks to Andrew Stucki, Adam Scarr (99designs.com), and Jan Masarik (masarik.sh) for discovering and reporting this issue.
Downloads are available at https://golang.org/dl for all supported platforms.
Alla prossima,
Filippo on behalf of the Go team
```
From the patch: 6e6f4aaf70
```
net/textproto: don't normalize headers with spaces before the colon
RFC 7230 is clear about headers with a space before the colon, like
X-Answer : 42
being invalid, but we've been accepting and normalizing them for compatibility
purposes since CL 5690059 in 2012.
On the client side, this is harmless and indeed most browsers behave the same
to this day. On the server side, this becomes a security issue when the
behavior doesn't match that of a reverse proxy sitting in front of the server.
For example, if a WAF accepts them without normalizing them, it might be
possible to bypass its filters, because the Go server would interpret the
header differently. Worse, if the reverse proxy coalesces requests onto a
single HTTP/1.1 connection to a Go server, the understanding of the request
boundaries can get out of sync between them, allowing an attacker to tack an
arbitrary method and path onto a request by other clients, including
authentication headers unknown to the attacker.
This was recently presented at multiple security conferences:
https://portswigger.net/blog/http-desync-attacks-request-smuggling-reborn
net/http servers already reject header keys with invalid characters.
Simply stop normalizing extra spaces in net/textproto, let it return them
unchanged like it does for other invalid headers, and let net/http enforce
RFC 7230, which is HTTP specific. This loses us normalization on the client
side, but there's no right answer on the client side anyway, and hiding the
issue sounds worse than letting the application decide.
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8eb23cde95)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Adding separate documents for older API versions, so that these don't have to
be collected from each tag/release branch. For each version of the API, I picked
the highest release that uses the API (to make sure to include possible fixes
in the swagger);
git mv api/swagger.yaml api/swagger-current.yaml
git checkout v18.05.0-ce -- api/swagger.yaml
git mv api/swagger.yaml docs/api/v1.37.yaml
git checkout v18.02.0-ce -- api/swagger.yaml
git mv api/swagger.yaml docs/api/v1.36.yaml
git checkout v18.01.0-ce -- api/swagger.yaml
git mv api/swagger.yaml docs/api/v1.35.yaml
git checkout v17.11.0-ce -- api/swagger.yaml
git mv api/swagger.yaml docs/api/v1.34.yaml
git checkout v17.10.0-ce -- api/swagger.yaml
git mv api/swagger.yaml docs/api/v1.33.yaml
git checkout v17.09.1-ce -- api/swagger.yaml
git mv api/swagger.yaml docs/api/v1.32.yaml
git checkout v17.07.0-ce -- api/swagger.yaml
git mv api/swagger.yaml docs/api/v1.31.yaml
git checkout v17.06.2-ce -- api/swagger.yaml
git mv api/swagger.yaml docs/api/v1.30.yaml
git mv api/swagger-current.yaml api/swagger.yaml
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2b8ae08571)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Relates to https://patchwork.kernel.org/patch/10756415/
Added to whitelist:
- `clock_getres_time64` (equivalent of `clock_getres`, which was whitelisted)
- `clock_gettime64` (equivalent of `clock_gettime`, which was whitelisted)
- `clock_nanosleep_time64` (equivalent of `clock_nanosleep`, which was whitelisted)
- `futex_time64` (equivalent of `futex`, which was whitelisted)
- `io_pgetevents_time64` (equivalent of `io_pgetevents`, which was whitelisted)
- `mq_timedreceive_time64` (equivalent of `mq_timedreceive`, which was whitelisted)
- `mq_timedsend_time64 ` (equivalent of `mq_timedsend`, which was whitelisted)
- `ppoll_time64` (equivalent of `ppoll`, which was whitelisted)
- `pselect6_time64` (equivalent of `pselect6`, which was whitelisted)
- `recvmmsg_time64` (equivalent of `recvmmsg`, which was whitelisted)
- `rt_sigtimedwait_time64` (equivalent of `rt_sigtimedwait`, which was whitelisted)
- `sched_rr_get_interval_time64` (equivalent of `sched_rr_get_interval`, which was whitelisted)
- `semtimedop_time64` (equivalent of `semtimedop`, which was whitelisted)
- `timer_gettime64` (equivalent of `timer_gettime`, which was whitelisted)
- `timer_settime64` (equivalent of `timer_settime`, which was whitelisted)
- `timerfd_gettime64` (equivalent of `timerfd_gettime`, which was whitelisted)
- `timerfd_settime64` (equivalent of `timerfd_settime`, which was whitelisted)
- `utimensat_time64` (equivalent of `utimensat`, which was whitelisted)
Not added to whitelist:
- `clock_adjtime64` (equivalent of `clock_adjtime`, which was not whitelisted)
- `clock_settime64` (equivalent of `clock_settime`, which was not whitelisted)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 89fabf0f24)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 7c1e88399e...481103c879
- Fix error handling for task deletion
- Fix fd leak of shim log
- Fix killall when use pidnamespace
- Improve ARM platform matching
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This makes sure that we don't become vulnerable to CVE-2018-17419 or
CVE-2019-19794 in the future. While we are not currently vulnerable to
either, there is a risk that a PR could be made which uses one of the
vulnerable methods in the future, so it's worth going ahead and updating
to ensure that a simple PR that would easily pass code review doesn't
lead to a vulnerability.
Signed-off-by: Sam Whited <sam@samwhited.com>
Currently we show this deprecation notice for any error returned by a
registry.
Registries can return an error for any number of reasons.
Instead let's show the deprecation notice only if the fallback was
successful.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 6859bc7eee)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: e004a1415a...aa1ba87e99
changes:
- Use standard include paths instead of hard-coding
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5125f8b304)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
CI runs on Ubuntu 16.04 machines, which use iptables (legacy), but
Debian buster uses nftables. Because of this, DNS resolution does not
work if the daemon configures iptables.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit bb0472bd23)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The btrfs-tools was a transitional package, and no longer exists:
> Package btrfs-tools
> stretch (oldstable) (admin): transitional dummy package
> 4.7.3-1: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x
It must be replaced either by `btrfs-progs` or `libbtrfs-dev` (which has just the development headers)
> Package: libbtrfs-dev (4.20.1-2)
> Checksumming Copy on Write Filesystem utilities (development headers)
Note that the `libbtrfs-dev` package is not available on Debian stretch
(only in stretch-backports)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4e3ab9e9fb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
```
---> Making bundle: .integration-daemon-stop (in bundles/test-integration)
++++ cat bundles/test-integration/docker.pid
+++ kill 13137
+++ /etc/init.d/apparmor stop
Leaving: AppArmorNo profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.
To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run 'aa-teardown'."
script returned exit code 255
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5dbfae6949)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Ordering the COPY lines to optimize for layer sharing
when these dependencies are updated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8edbe5dec2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Indenting with tabs can cause the formatting to go wonky,
because the first line of any command is "indented" with spaces,
but following lines are not, therefore they can be mis-aligned with
the first line.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a42b4144bc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https:/github.com/LK4D4/vndr/compare/v0.1.0...v0.1.1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 486161a63a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
With this change, go packages/modules that use versioned
import paths (github.com/foo/bar/v2), but don't use a directory
in the repository, can now be supported.
For example:
```
github.com/coreos/go-systemd/v22 v22.0.0
```
will vendor the github.com/coreos/go-systemd repository
into `vendor/github.com/coreos/go-systemd/v22`.
full diff: f5ab8fc5fb...v0.1.0
- LK4D4/vndr#83 migrate bitbucket to api 2.0
- fixesLK4D4/vndr#82https://api.bitbucket.org/1.0/repositories/ww/goautoneg: 410 Gone
- LK4D4/vndr#86 Replace sort.Sort with sort.Strings
- LK4D4/vndr#87 support `github.com/coreos/go-systemd/v22`
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d4f05c168d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This version avoids doing name lookups on creating tarball that
should be avoided in to not hit loading glibc shared libraries.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
from XDG_CONFIG_HOME
when in rootless mode
Signed-off-by: Dmitry Sharshakov <d3dx12.xx@gmail.com>
(cherry picked from commit f4fa98f583)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
full diff: https://github.com/grpc/grpc-go/compare/v1.23.0...v1.23.1
- grpc/grpc-go#3018 server: set and advertise max frame size of 16KB
- grpc/grpc-go#3017 grpclb: fix deadlock in grpclb connection cache
- Before the fix, if the timer to remove a SubConn fires at the
same time NewSubConn cancels the timer, it caused a mutex leak
and deadlock.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 39ad39d220)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Dont assign the --bip value directly to the subnet
for the default bridge. Instead use the network value
from the ParseCIDR output
Addresses: https://github.com/moby/moby/issues/40392
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
(cherry picked from commit f800d5f786)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 0f9f99500c removed the
use of SQLite for managing container links, and commit
f8119bb7a7 removed the migration
tool, and SQLite dependency.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e553a03627)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fixes issues where one goroutine tries to delete or rename a file
while another goroutine has the file open (e.g. a log reader).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit a5f237c2b5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix tries to address the issue raised in 39353 where
docker crash when creating namespaces with UID in /etc/subuid and /etc/subgid.
The issue was that, mapping to `/etc/sub[u,g]id` in docker does not
allow numeric ID.
This fix fixes the issue by probing other combinations (uid:groupname, username:gid, uid:gid)
when normal username:groupname fails.
This fix fixes 39353.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
(cherry picked from commit f09dc2f4fc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Specify base64url rather than base64. Also correct other links to the base64url section of RFC4648
Signed-off-by: Mike Bush <mpbush@gmail.com>
(cherry picked from commit f282dde877)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
WithBlock makes sure that the following containerd request is reliable.
In one edge case with high load pressure, kernel kills dockerd, containerd
and containerd-shims caused by OOM. When both dockerd and containerd
restart, but containerd will take time to recover all the existing
containers. Before containerd serving, dockerd will failed with gRPC
error. That bad thing is that restore action will still ignore the
any non-NotFound errors and returns running state for
already stopped container. It is unexpected behavior. And
we need to restart dockerd to make sure that anything is OK.
It is painful. Add WithBlock can prevent the edge case. And
n common case, the containerd will be serving in shortly.
It is not harm to add WithBlock for containerd connection.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 9f73396dab)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The latest `ltsc2019` image (`10.0.17763.1039`) appear to be broken,
and even a `RUN Write-Host hello` hangs.
Temporarily switching back to an older version so that CI doesn't fail.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit fa2417984b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before the collection goroutine wakes up every 1 second (as configured).
This sleep interval is in case there are no stats to collect we don't
end up in a tight loop.
Instead use a condition variable to signal that a collection is needed.
This prevents us from waking the goroutine needlessly when there is no
one looking for stats.
For now I've kept the sleep just moved it to the end of the loop, which
gives some space between collections.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit e75e6b0e31)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
full diff: 5215b1806f...v1.3.1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 12c7541f1f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The thirteenth patch release for `containerd` 1.2 fixes a regression introduced
in v1.2.12 that caused container/shim to hang on single core machines, fixes an
issue with blkio, and updates the Golang runtime to 1.12.17.
* Fix container pid race condition
* Update containerd/cgroups dependency to address blkio issue
* Set octet-stream content-type on PUT request
* Pin to libseccomp 2.3.3 to preserve compatibility with hosts that do not have libseccomp 2.4 or higher installed
* Update Golang runtime to 1.12.17, which includes a fix to the runtime
full diff: https://github.com/containerd/containerd/compare/v1.2.12...v1.2.13
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Support for these constraints was added in docker 1.13.0
(API v1.25), but never documented.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ed439e4a31)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This makes sure that things like `--tmpfs` mounts over an anonymous
volume don't create volumes uneccessarily.
One method only checks mountpoints, the other checks both mountpoints
and tmpfs... the usage of these should likely be consolidated.
Ideally, processing for `--tmpfs` mounts would get merged in with the
rest of the mount parsing. I opted not to do that for this change so the
fix is minimal and can potentially be backported with fewer changes of
breaking things.
Merging the mount processing for tmpfs can be handled in a followup.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit f464c31668)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Now that we do check if overlay is working by performing an actual
overlayfs mount, there's no need in extra checks for the kernel version
or the filesystem type. Actual mount check is sufficient.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit e226aea280)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Before this commit, overlay check was performed by looking for
`overlay` in /proc/filesystem. This obviously might not work
for rootless Docker (fs is there, but one can't use it as non-root).
This commit changes the check to perform the actual mount, by reusing
the code previously written to check for multiple lower dirs support.
The old check is removed from both drivers, as well as the additional
check for the multiple lower dirs support in overlay2 since it's now
a part of the main check.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 649e4c8889)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This moves supportsMultipleLowerDir() to overlayutils
so it can be used from both overlay and overlay2.
The only changes made were:
* replace logger with logrus
* don't use workDirName mergedDirName constants
* add mnt var to improve readability a bit
This is a preparation for the next commit.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit d5687079ad)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Signed-off-by: Danny Milosavljevic <dannym@scratchpost.org>
(cherry picked from commit 074eca1d79)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
`--userland-proxy-path` is automatically set by dockerd: e6c1820ef5/cmd/dockerd/config_unix.go (L46)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 9bd1ae024a)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
full diff: https://github.com/containerd/containerd/compare/v1.2.11...v1.2.12
Welcome to the v1.2.12 release of containerd!
The twelfth patch release for containerd 1.2 includes an updated runc with
a fix for CVE-2019-19921, an updated version of the opencontainers/selinux
dependency, which includes a fix for CVE-2019-16884, an updated version of the
gopkg.in/yaml.v2 dependency to address CVE-2019-11253, and a Golang update.
Notable Updates
- Update the runc vendor to v1.0.0-rc10 which includes a mitigation for CVE-2019-19921.
- Update the opencontainers/selinux which includes a mitigation for CVE-2019-16884.
- Update Golang runtime to 1.12.16, mitigating the CVE-2020-0601 certificate verification
bypass on Windows, and CVE-2020-7919, which only affects 32-bit architectures.
- Update Golang runtime to 1.12.15, which includes a fix to the runtime (Go 1.12.14,
Go 1.12.15) and and the net/http package (Go 1.12.15)
- A fix to prevent SIGSEGV when starting containerd-shim containerd/containerd#3960
- Fixes to exec containerd/containerd#3755
- Prevent docker exec hanging if an earlier docker exec left a zombie process
- Prevent High system load/CPU utilization with liveness and readiness probes
- Prevent Docker healthcheck causing high CPU utilization
CRI fixes:
- Update the gopkg.in/yaml.v2 vendor to v2.2.8 with a mitigation for CVE-2019-11253
API
- Fix API filters to properly handle and return parse errors containerd/containerd#3950
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 88737f569e...69ecbb4d6d
Includes 69ecbb4d6d
(forward-port of 8b5121be2f),
which fixes CVE-2020-7919:
- Panic in crypto/x509 certificate parsing and golang.org/x/crypto/cryptobyte
On 32-bit architectures, a malformed input to crypto/x509 or the ASN.1 parsing
functions of golang.org/x/crypto/cryptobyte can lead to a panic.
The malformed certificate can be delivered via a crypto/tls connection to a
client, or to a server that accepts client certificates. net/http clients can
be made to crash by an HTTPS server, while net/http servers that accept client
certificates will recover the panic and are unaffected.
Thanks to Project Wycheproof for providing the test cases that led to the
discovery of this issue. The issue is CVE-2020-7919 and Go issue golang.org/issue/36837.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b606c8e440)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/golang/go/compare/go1.12.15...go1.12.16
go1.12.16 (released 2020/01/28) includes two security fixes. One mitigates the
CVE-2020-0601 certificate verification bypass on Windows. The other affects only
32-bit architectures.
https://github.com/golang/go/issues?q=milestone%3AGo1.12.16+label%3ACherryPickApproved
- X.509 certificate validation bypass on Windows 10
A Windows vulnerability allows attackers to spoof valid certificate chains when
the system root store is in use. These releases include a mitigation for Go
applications, but it’s strongly recommended that affected users install the
Windows security update to protect their system.
This issue is CVE-2020-0601 and Go issue golang.org/issue/36834.
- Panic in crypto/x509 certificate parsing and golang.org/x/crypto/cryptobyte
On 32-bit architectures, a malformed input to crypto/x509 or the ASN.1 parsing
functions of golang.org/x/crypto/cryptobyte can lead to a panic.
The malformed certificate can be delivered via a crypto/tls connection to a
client, or to a server that accepts client certificates. net/http clients can
be made to crash by an HTTPS server, while net/http servers that accept client
certificates will recover the panic and are unaffected.
Thanks to Project Wycheproof for providing the test cases that led to the
discovery of this issue. The issue is CVE-2020-7919 and Go issue golang.org/issue/36837.
This is also fixed in version v0.0.0-20200124225646-8b5121be2f68 of golang.org/x/crypto/cryptobyte.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Found these by doing a `grep -R 'using the force'` on a full test run.
There's still a few more which are running against the main test daemon,
so it is difficult to find which test they belong to.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit fcd65ebf49)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fix TODO: move into startMetricsServer()
Fix errors.Wrap return nil when passed err is nil
Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com>
Signed-off-by: HuanHuan Ye <logindaveye@gmail.com>
(cherry picked from commit 88c554f950)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
While working on other tests I noticed that environment.Execution cannot
be used for anything but the pre-configured daemon, however this can
come in handy for being able share daemons across multiple tests that
currently spin up a new daemon.
The execution env also seems to be misused in some of these cases.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 1381956499)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If `unix.Lgetxattr` returns an error, then `sz == -1` which will cause a
runtime panic if `errno == unix.ERANGE`.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
(cherry picked from commit 4138cd22ab)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The previous description stated that an array of names / ids could be passed when the API in reality expects objects in the form of NetworkAttachmentConfig. This is fixed by updating the description and adding a definition for NetworkAttachmentConfig.
Signed-off-by: Hannes Ljungberg <hannes@5monkeys.se>
(cherry picked from commit 4d09fab232)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This information was added to an older version of the API
documentation (through 164ab2cfc9 and
5213a0a67e), but only added in the
"docs" branch.
This patch copies the information to the swagger file.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 79c877cfa7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
To match ContainerJSONBase api/types/types.go
Signed-off-by: Daniel Black <daniel@linux.ibm.com>
(cherry picked from commit 7b4b940470)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This parameter was introduced 4 years ago in b857dadb33
as part of https://github.com/moby/moby/pull/15711, but has never made it to the API docs.
Signed-off-by: Jan Chren (rindeal) <dev.rindeal@gmail.com>
(cherry picked from commit 9608dc5470)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fix the indentation to allow jane-openapi generate to work
Signed-off-by: Jeremy Leherpeur <jeremy.leherpeur@yousign.fr>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit cf315bedc5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Unlike Linux which uses a temp dir as GOPATH, Windows
uses c:\go. Among other things, this blocks go get.
Moving GOPATH to c:\gopath and updating references in
comments and documentation.
Currently the change is being scoped narrowly. In the
future GOPATH value could be passed as a parameter to
the ps1 scripts.
Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>
(cherry picked from commit ecf91f0d7f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
btrfs_noversion was added in d7c37b5a28
for distributions that did not have the `btrfs/version.h` header file.
Seeing how all of the distributions we currently support do have the
`btrfs/version.h` file we should probably just remove this build flag
altogether.
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
(cherry picked from commit e665263b10)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When cleaning up IPC mounts, the daemon could log a warning if the IPC mount was not found;
```
cleanup: failed to unmount IPC: umount /var/lib/docker/containers/90f408e26e205d30676655a08504dddc0d17f5713c1dd4654cf67ded7d3bbb63/mounts/shm, flags: 0x2: no such file or directory"
```
These warnings are safe to ignore, but can cause some confusion; `container.UnmountIpcMount()`
already attempted to suppress these warnings, however, `mount.Unmount()` returns a `mountError`,
which nests the original error, therefore detecting failed.
This parch uses `errors.Cause()` to get the _underlying_ error to detect if it's a "is not exist".
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 060f387c0b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `statusCodeFromGRPCError` and `statusCodeFromDistributionError`
helpers are used by `GetHTTPErrorStatusCode`, which already recurses
if the error implements the `Causer` interface.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 32f4fdfb5c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before this change:
dockerd -H unix:///run/docker.sock -H unix:///run/docker.sock -H unix:///run/docker.sock
...
INFO[2019-07-13T00:02:36.195090937Z] Daemon has completed initialization
INFO[2019-07-13T00:02:36.215940441Z] API listen on /run/docker.sock
INFO[2019-07-13T00:02:36.215933172Z] API listen on /run/docker.sock
INFO[2019-07-13T00:02:36.215990566Z] API listen on /run/docker.sock
After this change:
dockerd -H unix:///run/docker.sock -H unix:///run/docker.sock -H unix:///run/docker.sock
...
INFO[2019-07-13T00:01:37.533579874Z] Daemon has completed initialization
INFO[2019-07-13T00:01:37.567045771Z] API listen on /run/docker.sock
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d470252e87)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Could be misinterpreted as something not too kosher
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
(cherry picked from commit 8be39cd277)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The ImageCollectionTest.test_pull_multiple test performs a `docker pull` without
a `:tag` specified) to pull all tags of the given repository (image).
After pulling the image, the image(s) pulled are checked to verify if the list
of images contains the `:latest` tag.
However, the test assumes that all tags of the image are tags for the same
version of the image (same digest), and thus a *single* image is returned, which
is not always the case.
Currently, the `hello-world:latest` and `hello-world:linux` tags point to a
different digest, therefore the `client.images.pull()` returns multiple images:
one image for digest, making the test fail:
=================================== FAILURES ===================================
____________________ ImageCollectionTest.test_pull_multiple ____________________
tests/integration/models_images_test.py:90: in test_pull_multiple
assert len(images) == 1
E AssertionError: assert 2 == 1
E + where 2 = len([<Image: 'hello-world:linux'>, <Image: 'hello-world:latest'>])
This patch temporarily skips the broken test until it is fixed upstream.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f2b25e498f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
go1.12.14 (released 2019/12/04) includes a fix to the runtime. See the Go 1.12.14
milestone on our issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.12.14+label%3ACherryPickApproved
Update Golang 1.12.13
------------------------
go1.12.13 (released 2019/10/31) fixes an issue on macOS 10.15 Catalina where the
non-notarized installer and binaries were being rejected by Gatekeeper. Only macOS
users who hit this issue need to update.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Among other things, this is required to pull in
microsoft/hcsshim#718
Also fixesmicrosoft/hcsshim#737
which was caught by checks while attempting to bump
up hcsshim version.
Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>
(cherry picked from commit a7b6c3f0bf)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fixed failures in TestRunAttachFailedNoLeak caused by case mismatch
Signed-off-by: vikrambirsingh <vikrambir.singh@docker.com>
(cherry picked from commit c530c9cbb0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7c40c0a922)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
follow-up to 27552ceb15, where this
was left as a review comment, but the PR was already merged.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9a7e96b5b7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/containerd/containerd/compare/v1.2.10...v1.2.11
The eleventh patch release for containerd 1.2 includes an updated runc with
an additional fix for CVE-2019-16884 and a Golang update.
Notable Updates
-----------------------
- Update the runc vendor to v1.0.0-rc9 which includes an additional mitigation
for CVE-2019-16884.
More details on the runc CVE in opencontainers/runc#2128, and the additional
mitigations in opencontainers/runc#2130.
- Add local-fs.target to service file to fix corrupt image after unexpected host
reboot. Reported in containerd/containerd#3671, and fixed by containerd/containerd#3746.
- Update Golang runtime to 1.12.13, which includes security fixes to the crypto/dsa
package made in Go 1.12.11 (CVE-2019-17596), and fixes to the go command, runtime,
syscall and net packages (Go 1.12.12).
CRI fixes:
-----------------------
- Fix shim delete error code to avoid unnecessary retries in the CRI plugin. Discovered
in containerd/cri#1309, and fixed by containerd/containerd#3732 and containerd/containerd#3739.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 3e425f80a8...v1.0.0-rc9
- opencontainers/runc#1951 Add SCMP_ACT_LOG as a valid Seccomp action
- opencontainers/runc#2130 *: verify operations on /proc/... are on procfs
This is an additional mitigation for CVE-2019-16884. The primary problem
is that Docker can be coerced into bind-mounting a file system on top of
/proc (resulting in label-related writes to /proc no longer happening).
While we are working on mitigations against permitting the mounts, this
helps avoid our code from being tricked into writing to non-procfs
files. This is not a perfect solution (after all, there might be a
bind-mount of a different procfs file over the target) but in order to
exploit that you would need to be able to tweak a config.json pretty
specifically (which thankfully Docker doesn't allow).
Specifically this stops AppArmor from not labeling a process silently
due to /proc/self/attr/... being incorrectly set, and stops any
accidental fd leaks because /proc/self/fd/... is not real.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The [gelf payload specification](http://docs.graylog.org/en/2.4/pages/gelf.html#gelf-payload-specification)
demands that the field `short_message` *MUST* be set by the client library.
Since docker logging via the gelf driver sends messages line by line, it can happen that messages with an empty
`short_message` are passed on. This causes strict downstream processors (like graylog) to raise an exception.
The logger now skips messages with an empty line.
Resolves: #40232
See also: #37572
Signed-off-by: Jonas Heinrich <Jonas@JonasHeinrich.com>
(cherry picked from commit 5c6b913ff1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
also updates libnetwork to d9a6682a4dbb13b1f0d8216c425fe9ae010a0f23
full diff:
3eb39382bf...d9a6682a4d
- docker/libnetwork#2482 [19.03 backport] Shorten controller ID in exec-root to not hit UNIX_PATH_MAX
- docker/libnetwork#2483 [19.03 backport] Fix panic in drivers/overlay/encryption.go
Signed-off-by: Grant Millar <rid@cylo.io>
(cherry picked from commit df7b8f458a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When trying to build with some pretty typical version strings this was
causing failures trying to generate the windows resource file.
The resource file is already gated by an `ifdef` for this var, so
instead of blindly setting based on "VERSION", which can contain some
characters which are incompatible (e.g. 1.2.3.rc.0 will fail due to the
".rc").
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit ce931f28ea)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The validate step in CI was broken, due to a combination of
086b4541cf, fbdd437d29,
and 85733620eb being merged to master.
```
api/types/filters/parse.go:39:1: exported method `Args.Keys` should have comment or be unexported (golint)
func (args Args) Keys() []string {
^
daemon/config/builder.go:19:6: exported type `BuilderGCFilter` should have comment or be unexported (golint)
type BuilderGCFilter filters.Args
^
daemon/config/builder.go:21:1: exported method `BuilderGCFilter.MarshalJSON` should have comment or be unexported (golint)
func (x *BuilderGCFilter) MarshalJSON() ([]byte, error) {
^
daemon/config/builder.go:35:1: exported method `BuilderGCFilter.UnmarshalJSON` should have comment or be unexported (golint)
func (x *BuilderGCFilter) UnmarshalJSON(data []byte) error {
^
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9d726f1c18)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If anything marshals the daemon config now or in the future
this commit ensures the correct canonical form for the builder
GC policies' filters.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 85733620eb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
For backwards compatibility, the old incorrect object format for
builder.GC.Rule.Filter still works but is deprecated in favor of array of
strings akin to what needs to be passed on the CLI.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit fbdd437d29)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
buildkit supports entitlements like network-host and security-insecure.
this patch aims to make it configurable through daemon.json file.
by default network-host is enabled & secuirty-insecure is disabled.
Signed-off-by: Kunal Kushwaha <kunal.kushwaha@gmail.com>
(cherry picked from commit 8b7bbf180f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 7019b60d0d added these
env-vars to other stages, but forgot to update the DCO stage,
which also does a diff to validate commits that are in a PR.
Also adding openssh-client, for situations where the upstream
needs to be accessed through an ssh connection.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7c5fd83c22)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit e76dea157e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before this change we just accept that any error is "not found" and it
could be something else, but even if it it is just a "not found" kind of
error this should be dealt with from the container store and not the
event processor.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 54e30a62d3)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously we were re-using schema2.DeserializedManifest to handle oci
manifests. The issue lies in the fact that distribution started
validating the media type string during json deserialization. This
change broke our usage of that type.
Instead distribution now provides direct support for oci schemas, so use
that instead of our custom handlers.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit e443512ce4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change caused a regression, causing the DOCKER-USER chain
to not be created, despite iptables being enabled on the daemon.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is a continuation of 2a08f33166247da9d4c09d4c6c72cbb8119bf8df;
When running CI in other repositories (e.g. Docker's downstream
docker/engine repository), or other branches, the validation
scripts were calculating the list of changes based on the wrong
information.
This lead to weird failures in CI in a branch where these values
were not updated ':-) (CI on a pull request failed because it detected
that new tests were added to the deprecated `integration-cli` test-suite,
but the pull request did not actually make changes in that area).
This patch uses environment variables set by Jenkins to sets the
correct target repository (and branch) to compare to.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7019b60d0d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The patch made in docker/libnetwork#2450 caused a breaking change in the
networking behaviour, causing Kubernetes installations on Docker Desktop
(and possibly other setups) to fail.
Rolling back this change in the 19.03 branch while we investigate if there
are alternatives.
diff: 45c710223c...96bcc0dae8
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When slirp4netns v0.4.0+ is used, now slirp4netns is hardened using
mount namespace ("sandbox") and seccomp to mitigate potential
vulnerabilities.
bump up rootlesskit: 2fcff6ceae...791ac8cb20
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit e20b7323fb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Also updated the related docs.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5175ed54e5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The docker-py tests were broken, because the version of
py-test that was used, used a dependency that had a new
major release with a breaking change.
Unfortunately, it was not pinned to a specific version,
so when the dependency did the release, py-test broke;
```
22:16:47 Traceback (most recent call last):
22:16:47 File "/usr/local/bin/pytest", line 10, in <module>
22:16:47 sys.exit(main())
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 61, in main
22:16:47 config = _prepareconfig(args, plugins)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 182, in _prepareconfig
22:16:47 config = get_config()
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 156, in get_config
22:16:47 pluginmanager.import_plugin(spec)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
22:16:47 __import__(importspec)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/tmpdir.py", line 25, in <module>
22:16:47 class TempPathFactory(object):
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/tmpdir.py", line 35, in TempPathFactory
22:16:47 lambda p: Path(os.path.abspath(six.text_type(p)))
22:16:47 TypeError: attrib() got an unexpected keyword argument 'convert'
```
docker-py master has a fix for this (bumping the version of
`py-test`), but it's not in a release yet, and the docker cli that's used
in our CI is pinned to 17.06, which doesn't support building from a remote
git repository from a specific git commit.
To fix the immediate situation, this patch switches the docker-py
tests to run from the master branch.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 48353e16fe)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
The `/session` endpoint left experimental in API V1.39 through
239047c2d3 and
01c9e7082e, but the API reference
was not updated accordingly.
This updates the API documentation to match the change.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6756f5f378)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/golang/go/compare/go1.12.9...go1.12.10
```
Hi gophers,
We have just released Go 1.13.1 and Go 1.12.10 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.1).
net/http (through net/textproto) used to accept and normalize invalid HTTP/1.1 headers with a space before the colon, in violation of RFC 7230. If a Go server is used behind an uncommon reverse proxy that accepts and forwards but doesn't normalize such invalid headers, the reverse proxy and the server can interpret the headers differently. This can lead to filter bypasses or request smuggling, the latter if requests from separate clients are multiplexed onto the same upstream connection by the proxy. Such invalid headers are now rejected by Go servers, and passed without normalization to Go client applications.
The issue is CVE-2019-16276 and Go issue golang.org/issue/34540.
Thanks to Andrew Stucki, Adam Scarr (99designs.com), and Jan Masarik (masarik.sh) for discovering and reporting this issue.
Downloads are available at https://golang.org/dl for all supported platforms.
Alla prossima,
Filippo on behalf of the Go team
```
From the patch: 6e6f4aaf70
```
net/textproto: don't normalize headers with spaces before the colon
RFC 7230 is clear about headers with a space before the colon, like
X-Answer : 42
being invalid, but we've been accepting and normalizing them for compatibility
purposes since CL 5690059 in 2012.
On the client side, this is harmless and indeed most browsers behave the same
to this day. On the server side, this becomes a security issue when the
behavior doesn't match that of a reverse proxy sitting in front of the server.
For example, if a WAF accepts them without normalizing them, it might be
possible to bypass its filters, because the Go server would interpret the
header differently. Worse, if the reverse proxy coalesces requests onto a
single HTTP/1.1 connection to a Go server, the understanding of the request
boundaries can get out of sync between them, allowing an attacker to tack an
arbitrary method and path onto a request by other clients, including
authentication headers unknown to the attacker.
This was recently presented at multiple security conferences:
https://portswigger.net/blog/http-desync-attacks-request-smuggling-reborn
net/http servers already reject header keys with invalid characters.
Simply stop normalizing extra spaces in net/textproto, let it return them
unchanged like it does for other invalid headers, and let net/http enforce
RFC 7230, which is HTTP specific. This loses us normalization on the client
side, but there's no right answer on the client side anyway, and hiding the
issue sounds worse than letting the application decide.
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
```
docker/integration-cli/checker/checker.go
Line 12: warning: exported type Compare should have comment or be unexported (golint)
Line 14: warning: exported function False should have comment or be unexported (golint)
Line 20: warning: exported function True should have comment or be unexported (golint)
Line 26: warning: exported function Equals should have comment or be unexported (golint)
Line 32: warning: exported function Contains should have comment or be unexported (golint)
Line 38: warning: exported function Not should have comment or be unexported (golint)
Line 52: warning: exported function DeepEquals should have comment or be unexported (golint)
Line 58: warning: exported function HasLen should have comment or be unexported (golint)
Line 64: warning: exported function IsNil should have comment or be unexported (golint)
Line 70: warning: exported function GreaterThan should have comment or be unexported (golint)
Line 76: warning: exported function NotNil should have comment or be unexported (golint)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6397dd4d31)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
On account of being flaky on both RS1 and RS5.
Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com>
Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>
(cherry picked from commit 7de4e13089)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `-test.timeout=5m` was glued directly after the current `TESTFLAGS`,
causing them to be non-functional;
Before:
make TESTDEBUG=1 TESTDIRS='github.com/docker/docker/pkg/filenotify' TESTFLAGS='-test.run TestPollerEvent' test-unit
+ mkdir -p bundles
+ gotestsum --format=standard-quiet --jsonfile=bundles/go-test-report.json --junitfile=bundles/junit-report.xml -- -tags 'netgo seccomp libdm_no_deferred_remove' -cover -coverprofile=bundles/profile.out -covermode=atomic -test.run TestPollerEvent-test.timeout=5m github.com/docker/docker/pkg/filenotify
testing: warning: no tests to run
ok github.com/docker/docker/pkg/filenotify 0.003s coverage: 0.0% of statements [no tests to run]
DONE 0 tests in 0.298s
After:
make TESTDEBUG=1 TESTDIRS='github.com/docker/docker/pkg/filenotify' TESTFLAGS='-test.run TestPollerEvent' test-unit
+ mkdir -p bundles
+ gotestsum --format=standard-quiet --jsonfile=bundles/go-test-report.json --junitfile=bundles/junit-report.xml -- -tags 'netgo seccomp libdm_no_deferred_remove' -cover -coverprofile=bundles/profile.out -covermode=atomic -test.run TestPollerEvent -test.timeout=5m github.com/docker/docker/pkg/filenotify
ok github.com/docker/docker/pkg/filenotify 0.608s coverage: 44.7% of statements
DONE 1 tests in 0.922s
This was introduced in 42f0a0db75
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0620990307)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Replaced using a bit of grep-ing;
```
find . -name "*_test.go" -exec sed -E -i 's#assert.Assert\((.*), fmt.Sprintf\((.*)\)\)$#assert.Assert\(\1, \2\)#g' '{}' \;
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0fabf3e41e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
After the commit faaffd5d6d ("Windows:Disable 2 restart test when
Hyper-V") some tests became skipped on linux:
SKIP: docker_cli_restart_test.go:167: DockerSuite.TestRestartContainerSuccess (unmatched requirement IsolationIsProcess)
SKIP: docker_cli_restart_test.go:240: DockerSuite.TestRestartPolicyAfterRestart (unmatched requirement IsolationIsProcess)
But AFAIU it is highly unlikely that we actually meant to skip them on linux.
https://github.com/moby/moby/issues/39625
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
(cherry picked from commit b469933b06)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- remove -check.* flags
- use (per-test) -timeout flag
- allow user to override TEST_SKIP_* regardless of TESTFLAGS
- remove test-imports validation
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 7cd028f2d0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
go run rm-gocheck.go redress '[^/]\bcheck\.Suite\(.*\{\s*$' \
"integration-cli/check_test.go" "integration-cli/docker_cli_external_volume_driver_unix_test.go" "integration-cli/docker_cli_network_unix_test.go"
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 6a8a9738ec)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The following "rm-gocheck:"-prefixed commits were generated by
go run rm-gocheck.go --commit
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 8f64611c83)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Tests fixed in this patch used to compile and pass successfully,
despite checking if non-nullable types are not nil.
These would have become compile errors once go-check is removed.
About TestContainerAPIPsOmitFields:
Basically what happened is that this test got refactored to start using the API types
and API client library instead of custom types and stdlib's http functions.
This test used to test an API regression which could possibly be a unit test.
However because PublicPort and IP are not nullable types, this test became useless.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit e07a3f2917)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Don't use two-stage mount in TestContainersAPICreateMountsCreate();
apparently it was written before mount.Mount() could accept propagation
flags.
While at it, remove rw as this is the default.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 1cfdb2ffb8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The only option we supply is either BIND or a mount propagation flag,
so it makes sense to specify the flag value directly, rather than using
parseOptions() every time.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ec248fe61d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
It is sufficient to add "rprivate" to mount flags.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit a6773f69f2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
For bind mounts, fstype argument to mount(2) is ignored.
Usual convention is either empty string or "none".
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 4e65b17ac4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Eliminate double call to parseOptions() from Mount()
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 80fce834ad)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Current code in MakeMount parses /proc/self/mountinfo twice:
first in call to Mounted(), then in call to Mount(). Use
ForceMount() to eliminate such double parsing.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit aa60541877)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
TestLogBlocking is intended to test that the Log method blocks by
default. It does this by mocking out the internals of the
awslogs.logStream and replacing one of its internal channels with one
that is controlled by the test. The call to Log occurs inside a
goroutine. Go may or may not schedule the goroutine immediately and the
blocking may or may not be observed outside the goroutine immediately
due to decisions made by the Go runtime. This change adds a small
timeout for test failure so that the Go runtime has the opportunity to
run the goroutine before the test fails.
Signed-off-by: Samuel Karp <skarp@amazon.com>
(cherry picked from commit fd94bae0b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I've seen this test fail a number of times recently on RS1
Looking at failures, the test is taking a long time ro run (491.77s, which is
more than 8 minutes), so perhaps it's just too slow on RS1, which may be
because we switch to a different base image, or because we're now running
on different machines.
Compared to RS5 (still slow, but a lot faster);
```
--- PASS: Test/DockerSuite/TestAPIImagesSaveAndLoad (146.25s)
```
```
--- FAIL: Test/DockerSuite/TestAPIImagesSaveAndLoad (491.77s)
cli.go:45: assertion failed:
Command: d:\CI-5\CI-93d2cf881\binary\docker.exe inspect --format {{.Id}} sha256:69e7c1ff23be5648c494294a3808c0ea3f78616fad67bfe3b10d3a7e2be5ff02
ExitCode: 1
Error: exit status 1
Stdout:
Stderr: Error: No such object: sha256:69e7c1ff23be5648c494294a3808c0ea3f78616fad67bfe3b10d3a7e2be5ff02
Failures:
ExitCode was 1 expected 0
Expected no error
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5adaf52953)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This `ENV` was added to the Dockerfile in b96093fa56,
when the repository used per-architecture Dockerfiles, and some architectures needed
a different configuration.
Now that we use a multi-arch Dockerfile, and CI uses a Jenkinsfile, we can remove
this `ENV` from the Dockerfile, and set it in CI instead if needed.
Also updated the wording and fixed linting issues in hack/validate/gometalinter
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a464a3d51f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test frequently fails on Windows RS1 (mainly), so skipping it
for now on Windows;
```
ok github.com/docker/docker/daemon/logger 0.525s coverage: 43.0% of statements
time="2019-09-09T20:37:35Z" level=info msg="Trying to get region from EC2 Metadata"
time="2019-09-09T20:37:36Z" level=info msg="Log stream already exists" errorCode=ResourceAlreadyExistsException logGroupName= logStreamName= message= origError="<nil>"
--- FAIL: TestLogBlocking (0.02s)
cloudwatchlogs_test.go:313: Expected to be able to read from stream.messages but was unable to
time="2019-09-09T20:37:36Z" level=error msg=Error
time="2019-09-09T20:37:36Z" level=error msg="Failed to put log events" errorCode=InvalidSequenceTokenException logGroupName=groupName logStreamName=streamName message="use token token" origError="<nil>"
time="2019-09-09T20:37:36Z" level=error msg="Failed to put log events" errorCode=DataAlreadyAcceptedException logGroupName=groupName logStreamName=streamName message="use token token" origError="<nil>"
time="2019-09-09T20:37:36Z" level=info msg="Data already accepted, ignoring error" errorCode=DataAlreadyAcceptedException logGroupName=groupName logStreamName=streamName message="use token token"
FAIL
coverage: 78.2% of statements
FAIL github.com/docker/docker/daemon/logger/awslogs 0.630s
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6c75c86240)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Starting the daemon should not load the busybox image again
in most cases, so add a new `StartNodeWithBusybox` function
to be clear that this one loads the busybox image, and use
`StartNode()` for cases where loading the busybox image is
not needed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ead3f4e7c8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The daemon was already created and started with the busybox
image loaded, so there's no need to load the image again.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8fc23588f1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Noticed this test container not exiting correctly while debugging
another issue. Before this change, signals were being eaten by bash, now
they are hanlded by top. This cuts the test time in half since it
doesn't have to wait for docker to SIGKILL it.
Old:
PASS: docker_cli_swarm_test.go:840: DockerSwarmSuite.TestSwarmServiceTTY 18.997s
New:
PASS: docker_cli_swarm_test.go:840: DockerSwarmSuite.TestSwarmServiceTTY 6.293s
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit e6c5563ae9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of using the locally crafted `serviceContainerCount()` utility
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f874f8b6fd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Windows Dockerfile downloads the Go binaries, which (unlike
the Golang images) do not have a trailing `.0` in their version.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 61450a651b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was an oversight when changing the Dockerfile to use a build-arg;
the Windows Dockerfile downloads the Go binaries, which never have a
trailing `.0`.
This patch makes sure that the trailing zero (if any) is removed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c5bd6e3dc7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
There's already a step in "Nuke Everything", but lets' stop it
after stopping the daemon as well
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e1636ad5fa)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Hyper-V isolated containers do not allow file-operations on a
running container. This test currently uses `docker cp` to verify
that the WORKDIR was automatically created, which cannot be done
while the container is running.
```
FAIL: docker_cli_create_test.go:302: DockerSuite.TestCreateWithWorkdir
assertion failed:
Command: d:\CI-7\CI-f3768a669\binary\docker.exe cp foo:c:\home\foo\bar c:\tmp
ExitCode: 1
Error: exit status 1
Stdout:
Stderr: Error response from daemon: filesystem operations against a running Hyper-V container are not supported
Failures:
ExitCode was 1 expected 0
Expected no error
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ac9ef840ef)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change will cause Jenkins to only build the
latest HEAD of a PR branch, cancelling any
previous builds that may already be in progress.
This will decrease feedback time and help mitigate
resource contention.
Signed-off-by: Peter Salvatore <peter@psftw.com>
(cherry picked from commit 85bcc524ea)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
CI already stores the logs of the test daemon, so we might as well
store them as artifacts
```
[2019-09-03T12:49:39.835Z] INFO: Tidying up at end of run
[2019-09-03T12:49:39.835Z] INFO: Saving daemon under test log (d:\CI-2\CI-3593e7622\dut.out) to C:\windows\TEMP\CIDUT.out
[2019-09-03T12:49:39.835Z] INFO: Saving daemon under test log (d:\CI-2\CI-3593e7622\dut.err) to C:\windows\TEMP\CIDUT.err
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6ee61f5493)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Trying to see if this helps with the cleanup step exiting in CI, but
Jenkins continuing to wait for the script to end afterwards.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8e8c52c4ab)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Seen failing a couple of times:
```
[2019-09-02T08:40:15.796Z] =================================== FAILURES ===================================
[2019-09-02T08:40:15.796Z] __________________ AttachContainerTest.test_attach_no_stream ___________________
[2019-09-02T08:40:15.796Z] tests/integration/api_container_test.py:1250: in test_attach_no_stream
[2019-09-02T08:40:15.796Z] assert output == 'hello\n'.encode(encoding='ascii')
[2019-09-02T08:40:15.796Z] E AssertionError: assert b'' == b'hello\n'
[2019-09-02T08:40:15.796Z] E Right contains more items, first extra item: 104
[2019-09-02T08:40:15.796Z] E Use -v to get the full diff
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ce77a804b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`TestBuildBuildTimeArgEnv` and `TestBuildBuildTimeArgEmptyValVariants` were
using non-standard comparisons.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dbde4786e4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Add quotes to prevent word splitting in `cp` statement (SC2046)
- Replace legacy back tics with `$()`
- Replace `which` with `command -v` (SC2230)
- Fix incorrect (`==`) comparison
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 70d3677825)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of using a `initDispatchTestCases()` function, declare the test-table
inside `TestDispatch` itself, and run the tests as subtests.
```
[2019-08-27T15:14:51.072Z] builder/dockerfile/evaluator_test.go:18:2: `name` is unused (structcheck)
[2019-08-27T15:14:51.072Z] name, expectedError string
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a3f9cb5b63)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Some integration tests are known to fail if Windows Defender is
enabled. On the machines that run our CI, defender is disabled
for that reason.
Contributors likely will have defender enabled, and because of
that are currently not able to run the integration tests.
This patch changes the ERROR into a WARNING, so that contributors
can still run (a limited set of) the integration tests, but get
informed that some may fail.
We should make this requirement more specific, and only skip
tests that are known to require defender to be disabled, but
while that's not yet in place, let's print a warning instead.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 31885181fc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- use subtests to make it clearer what the individual test-cases
are, and to prevent tests from depending on values set by the
previous test(s).
- remove redundant messages in assert (gotest.tools already prints
a useful message if assertions fail).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 556d26c07d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
TestBuildMulitStageResetScratch testcase was actually meant to be
TestBuildMulitStageResetScratch
Signed-off-by: Vitaly Ostrosablin <tmp6154@yandex.ru>
(cherry picked from commit c266d8fe56)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
New features
- build CRIU with Android NDK
- C/R of
- IP RAW sockets
- lsm: dump and restore any SELinux process label
- support restoring ghost files on readonly mounts
Bugfixes
- Do not lock network if running in the host network namespace
- Fix RPC configuration file handling
- util: don't leak file descriptors to third-party tools
- small fixes here and there
Improvements
- travis: switch to the Ubuntu Xenial
- travis-ci: Enable ia32 tests
- Many improvements and bug fixes in the libcriu
- Changes in the API and ABI (SONAME increased from 1 to 2)
full diff: https://github.com/checkpoint-restore/criu/compare/v3.11...v3.12
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 00ad0222ce)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Generate more unique names, based on architecture and test-suite name.
Clean up the path to this integration test to create a useful package name.
"$dir" can be either absolute (/go/src/github.com/docker/docker/integration/foo)
or relative (./integration/foo). To account for both, first we strip the
absolute path, then any leading periods and slashes.
For the package-name, we use periods as separator instead of slashes, to be more
in-line with Java package names (which is what junit.xml was originally designed
for).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f007b0150a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Without these options set, test2json does not include a `Time`
field in the generated JSON;
{"Action":"run","Test":"TestCgroupNamespacesBuild"}
{"Action":"output","Test":"TestCgroupNamespacesBuild","Output":"=== RUN TestCgroupNamespacesBuild\n"}
{"Action":"output","Test":"TestCgroupNamespacesBuild","Output":"--- PASS: TestCgroupNamespacesBuild (1.70s)\n"}
...
{"Action":"pass","Test":"TestCgroupNamespacesBuild"}
As a result, `gotestsum` was not able to calculate test-duration, and
reported `time="0.000000"` for all tests;
<testcase classname="amd64.integration.build" name="TestCgroupNamespacesBuild" time="0.000000"></testcase>
With this patch applied:
{"Time":"2019-08-23T22:42:41.644361357Z","Action":"run","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild"}
{"Time":"2019-08-23T22:42:41.644367647Z","Action":"output","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Output":"=== RUN TestCgroupNamespacesBuild\n"}
{"Time":"2019-08-23T22:42:44.926933252Z","Action":"output","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Output":"--- PASS: TestCgroupNamespacesBuild (3.28s)\n"}
...
{"Time":"2019-08-23T22:42:44.927003836Z","Action":"pass","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Elapsed":3.28}
Which now correctly reports the test's duration:
<testcase classname="amd64.integration.build" name="TestCgroupNamespacesBuild" time="3.280000"></testcase>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d2e00d62e2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
'Namespace' parallel runs by bind-mounting a different directory
in the container, instead of making the tests running inside
the container aware of the namespaced location.
This makes it transparent to the tests, and slightly reduces
complexity.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3262a69be6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The inContainer check isn't really useful anymore.
Even though it was said that we shouldn't rely on its existence back in
2016, we're now in 2019 and this thing still exists so we should just
rely on it now to check whether or not we're in a container.
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
(cherry picked from commit f5cd8fdd44)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Call helper for starting swarm agents and add some logging with daemon
id's when joining the swarm.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit b0fe0dff7a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `DockerDaemonSuite.SetUpTest` already checks for Linux and a local daemon;
```
func (s *DockerDaemonSuite) SetUpTest(c *check.C) {
testRequires(c, DaemonIsLinux, testEnv.IsLocalDaemon)
s.d = daemon.New(c, dockerBinary, dockerdBinary, testdaemon.WithEnvironment(testEnv.Execution))
}
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7f37d99ef5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Code retrying service update operations when receiving "update out of
sequence" errors was removed because of a misunderstanding, which has
made tests flaky. This re-adds the "CmdRetryOutOfSequence" method, and
uses it in TestSwarmPublishAdd to avoid flaky behavior.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit 1de914695b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The session endpoint is no longer experimental since
01c9e7082e, so we don't
need to start an experimental daemon.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit becd29c665)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
A bit too quick on the trigger on some text completion I think...
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 5d818213ff)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is just noise due to timing. I picked `> 2` just based on
logs from tests I've seen there's always 1 or 2.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 15675e28f1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The directory used for storage was either changed or new directories
were added.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6a64a4deec)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This allows our tests, which all share a containerd instance, to be a
bit more isolated by setting the containerd namespaces to the generated
daemon ID's rather than the default namespaces.
This came about because I found in some cases we had test daemons
failing to start (really very slow to start) because it was (seemingly)
processing events from other tests.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 24ad2f486d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Removes some test functions that were unused:
- bridgeNfIP6tables
- ambientCapabilities (added to support #26979, which was reverted in #27737)
- overlay2Supported
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c887b09abc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Both Linux and Windows now use busybox, so no need to keep a
constant for this.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 27f432ca57)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
All current versions of Docker support exec, so no need
to check for this.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7204341950)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The daemon requires kernel 3.10 or up to start, so there's no need
to check if the daemon is kernel 3.8 or up.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 691eb14256)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
IPVLAN no longer is experimental since 3ab093d567,
so there's no need to set this option.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dae9bac675)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These tests require a local daemon, and are not built on Windows
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 1e4bd2623a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Their inclusion is no longer defensible.
closes#39981
Signed-off-by: Harald Niesche <harald@niesche.de>
(cherry picked from commit 77d3c68f97)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case jsonlogfile is used with max-file=1 and max-size set,
the log rotation is not perfomed; instead, the log file is closed
and re-open with O_TRUNC.
This situation is not handled by the log reader in follow mode,
leading to an issue of log reader being stuck forever.
This situation (file close/reopen) could be handled in waitRead(),
but fsnotify library chose to not listen to or deliver this event
(IN_CLOSE_WRITE in inotify lingo).
So, we have to handle this by checking the file size upon receiving
io.EOF from the log reader, and comparing the size with the one received
earlier. In case the new size is less than the old one, the file was
truncated and we need to seek to its beginning.
Fixes#39235.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 9cd24ba605)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
(cherry picked from commit f3a3ea0d3c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Looks like TestServiceWithDefaultAddressPoolInit is failing
randomly in CI. I am not able to reproduce the issue locally
but this has been reported few times. So I tried to modify
code and see if I can fix the random failure.
Signed-off-by: selansen <elango.siva@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 88578aa9e9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Line 441: warning: context.Context should be the first parameter of a function (golint)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dac5710b68)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Line 30: warning: context.Context should be the first parameter of a function (golint)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 123e29f44a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Line 59: warning: context.Context should be the first parameter of a function (golint)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9f9b4290b9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
```
Line 25: warning: context.Context should be the first parameter of a function (golint)
Line 44: warning: context.Context should be the first parameter of a function (golint)
Line 52: warning: context.Context should be the first parameter of a function (golint)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b4c46b0dac)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Line 30: warning: context.Context should be the first parameter of a function (golint)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit caec45a37f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This prevents restarting event processing in a tight loop.
You can see this with the following steps:
```terminal
$ containerd &
$ dockerd --containerd=/run/containerd/containerd.sock &
$ pkill -9 containerd
```
At this point you will be spammed with logs such as:
```
ERRO[2019-07-12T22:29:37.318761400Z] failed to get event error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: connection refused\"" module=libcontainerd namespace=plugins.moby
```
Without this change you can quickly end up with gigabytes of log data.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 1acaf2aabe)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This improves the select logic for the restart tests or starting the
daemon in general. With the way the ticker and select was setup, it was
possible for only the timeout to be displayed and not the wait errors.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit 402433a5e4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Reported by govet linter:
> daemon/monitor.go:57:9: lostcancel: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak (govet)
> ctx, _ := context.WithTimeout(context.Background(), 2*time.Second)
> ^
> daemon/monitor.go:128:9: lostcancel: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak (govet)
> ctx, _ := context.WithTimeout(context.Background(), 2*time.Second)
> ^
Fixes: b5f288 ("Handle blocked I/O of exec'd processes")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 53cbf1797b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is the second part to
https://github.com/containerd/containerd/pull/3361 and will help process
delete not block forever when the process exists but the I/O was
inherited by a subprocess that lives on.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit b5f28865ef)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
By convention, containers spawned by jenkins jobs have the name:
docker-pr${BUILD_NUMBER}
That works fine for jobs with a single container. This commit cleans up
when multiple containers are spawned with the convention that their names
share the same "docker-pr${BUILD_NUMBER}-" prefix.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit f470698c2c)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Essentially, run on merge to target branch which may or may not be
master branch. Could be 19.03 branch, for example.
See: https://jenkins.io/doc/book/pipeline/syntax/
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit e653943e8b)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 039eb05ac8)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
The Blue Ocean UI truncates names, which makes it possible to distinguish
which Windows stage is RS1 or RS5. This patch shortens those names so that they
fit in the Blue Ocean UI.
Other stages and parameters were renamed as well to better reflect what they're running;
Before | After
-------------------|--------------------------------
janky | amd64
power | ppc64le
power-master | ppc64le integration-cli
windowsRS1 | win-RS1
windowsRS5-process | win-RS5
z | s390x
z-master | x390x integration-cli
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
WIP renames
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c18f793f40)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
full diff: bbe341867e...f35d9100f2
changes included:
- docker/swarmkit#2891 [19.03 backport] Remove hardcoded IPAM config subnet value for ingress network
- backport of docker/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network
- fixes [ENGORC-2651] Specifying --default-addr-pool for docker swarm init is not picked up by ingress network
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The warning on pull was incorrectly asking to contact registry admins.
It is kept on push however.
Pulling manifest lists with v2 schema1 manifests will not be supported thus
there is a warning for those, but wording changed to suggest repository author
to upgrade.
Finally, a milder warning on regular pull is kept ONLY for DockerHub users
in order to incite moving away from schema1.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 647dfe99a5)
Signed-off-by: Tibor Vass <tibor@docker.com>
full diff: fcdddc395d...v1.0.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 720b66ee1f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 09cdcc8c0e...92d1fbe1eb
relevant changes included (omitting some changes that were added _and_ reverted in this bump):
- docker/libnetwork#2433 Fix parseIP error when parseIP before get AddressFamily
- fixesdocker/libnetwork#2431 parseIP Error ip=[172 17 0 2 0 0 0 0 0 0 0 0 0 0 0 0]
- https://github.com/docker/libnetwork/issues/2289
- this was a regression introduced in docker/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling
- docker/libnetwork#2440 Bump hashicorp go-sockaddr v1.0.2, go-multierror v1.0.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit bab58c1924)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diffs:
- fc5a7d91d5...62a13ae87c
- b2de5d10e3...v1.0.0
- 604eaf189e...13995c7128ccc8e51e9a6bd2b551020a27180abd
notable changes in libnetwork:
- docker/libnetwork#2366 Bump vishvananda/netlink to 1.0.0
- docker/libnetwork#2339 controller: Check if IPTables is enabled for arrangeUserFilterRule
- addresses docker/libnetwork#2158 dockerd when run with --iptables=false modifies iptables by adding DOCKER-USER
- addresses moby/moby#35777 With iptables=false dockerd still creates DOCKER-USER chain and rules
- addresses docker/for-linux#136 dockerd --iptables=false adds DOCKER-USER chain and modify FORWARD chain anyway
- docker/libnetwork#2394 Make DNS records and queries case-insensitive
- addresses moby/moby#28689 Embedded DNS is case-sensitive
- addresses moby/moby#21169 hostnames with new networking are case-sensitive
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 344b093258)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In situations where the containerd error is consumed directly
and not received over gRPC, errors were not translated.
This patch converts containerd errors to the correct HTTP
status code.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4a516215e2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
While investigating a test failure, I found this in the logs:
```
time="2019-07-04T15:06:32.622506760Z" level=warning msg="Error while setting daemon root propagation, this is not generally critical but may cause some functionality to not work or fallback to less desirable behavior" dir=/go/src/github.com/docker/docker/bundles/test-integration/d1285b8250308/root error="error writing file to signal mount cleanup on shutdown: open /tmp/dxr/d1285b8250308/unmount-on-shutdown: no such file or directory"
```
This path is generated from the daemon's exec-root, which appears to not
exist yet. This change just makes sure it exists before we try to write
a file.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 7725b88edc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fixes#39427
This always sends the exec exit events even when the exec fails to find
the binary. A standard 127 exit status is sent in this situation.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit c08d4da6e5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Trying to start a container that is already running is not an
error condition, so a `304 Not Modified` should be returned instead
of a `409 Conflict`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c030885e7a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The API does not filter images on platform, so searching on
Windows should work as well.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3d1850d10d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `--stars` flag was deprecated, and was replaced by `--filter stars=xx`
Integration tests run with a fixed version of the CLI, and the new
(`--filter`) option is already tested in this test, so there's no need
to verify the old flags.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 85d6fb888c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `--stars` and `--automated` flags have been deprecated, and were
replaced by `--filter stars=xx` and `--filter is-automated=true`.
Integration tests run with a fixed version of the CLI, and the new
(`--filter`) option is already tested in this test, so there's no need
to verify the old flags.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b38c71bfe0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Both `--help` and `--no-trunc` are implemented in the CLI. There's
no need to verify them here because the integration tests use a
fixed version of the CLI.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a78b9a3726)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This reverts commit 21e662c774.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 1be7065e99)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
TestSwarmClusterRotateUnlockKey had been identified as a flaky test. It
turns out that the test code was wrong: where we should have been
checking the string output of a command, we were instead checking the
value of the error. This means that the error case we were expecting was
not being matched, and the test was failing when it should have just
retried.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit b79adac339)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/grpc/grpc-go/compare/v1.20.1...v1.23.0
This update contains security fixes:
- transport: block reading frames when too many transport control frames are queued (grpc/grpc-go#2970)
- Addresses CVE-2019-9512 (Ping Flood), CVE-2019-9514 (Reset Flood), and CVE-2019-9515 (Settings Flood).
Other changes can be found in the release notes:
https://github.com/grpc/grpc-go/releases/tag/v1.23.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f1cd79976a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
kr/pty was moved to creak/pty and the old location was
archived.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0595c01718)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7
> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d5669ec1c6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Trying to link to a non-existing container is not valid, and should return an
"invalid parameter" (400) error. Returning a "not found" error in this situation
would make the client report the container's image could not be found.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 422067ba7b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Ref: #39426
This is a common flaky test that I have seen on multiple PRs. It is not
consistent and should be skipped until it is fixed to be robust. A
simple fix for the swarm tests is not easy as they all poll and have 1
billion timeouts in all the tests so a skip is valid here.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit b94218560e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When mounting overlays which have children, enforce that
the mount is always performed as read only. Newer versions
of the kernel return a device busy error when a lower directory
is in use as an upper directory in another overlay mount.
Adds committed file to indicate when an overlay is being used
as a parent, ensuring it will no longer be mounted with an
upper directory.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
(cherry picked from commit 477bf1e413)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Improve the readability of the connection error displayed to the user on
Windows when running docker commands fails by checking if the client is
privileged. If so then display the actual error wrapped in a generic
error "This error may indicate that the docker daemon is not running."
If not that display the actual error wrapped in a more specific error:
"In the default daemon configuration on Windows, the docker client must
be run with elevated privileges to connect."
Signed-off-by: Nick Adcock <nick.adcock@docker.com>
(cherry picked from commit 1a5dafb31e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `ioctl` interface for the `LOOP_CTL_GET_FREE` request on
`/dev/loop-control` is a little different from what `unix.IoctlGetInt`
expects: the first index is the returned status in `r1`, not an `int`
pointer as the first parameter.
Unfortunately we have to go a little lower level to get the appropriate
loop device index out, using `unix.Syscall` directly to read from
`r1`. Internally, the index is returned as a signed integer to match the
internal `ioctl` expectations of interpreting a negative signed integer
as an error at the userspace ABI boundary, so the direct interface of
`ioctlLoopCtlGetFree` can remain as-is.
[@kolyshkin: it still worked before this fix because of
/dev scan fallback in ioctlLoopCtlGetFree()]
Signed-off-by: Daniel Sweet <danieljsweet@icloud.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit db2bc43017)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Windows RS1 has problems with leaking NdisAdapters during the integration
tests; the windows.ps1 script has a cleanup stesp to remove those
leaked adapters.
For internal testing at Microsoft on internal builds, this cleanup step
was skipped, and only ran on the CI machines in our Jenkins.
Due to the move to our new Jenkins, the names of Windows machines changed,
and because of that, the cleanup step was never executed, resulting in the
leaked adapters not being cleaned up:
```
20:32:23 WARNING: There are 608 NdisAdapters leaked under Psched\Parameters
20:32:23 WARNING: Not cleaning as not a production RS1 server
20:32:24 WARNING: There are 608 NdisAdapters leaked under WFPLWFS\Parameters
20:32:24 WARNING: Not cleaning as not a production RS1 server
```
```
22:01:31 WARNING: There are 1209 NdisAdapters leaked under Psched\Parameters
22:01:31 WARNING: Not cleaning as not a production RS1 server
22:01:31 WARNING: There are 1209 NdisAdapters leaked under WFPLWFS\Parameters
22:01:31 WARNING: Not cleaning as not a production RS1 server
```
This patch removes the check for non-production builds, and unconditionally
cleans up leaked adapters if they are found.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 156ad54fb7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The "new test" detector in test-integration-flaky was a bit flaky since
it would detect function signatures that are not new tests.
In addition, the test calls `return` outside of a function which is not
allowed.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit e2b24490e4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was introduced in a76ff632a4:
+ find bundles -path */root/*overlay2 -prune -o -type f ( -o -name *.log -o -name *.prof ) -print
find: invalid expression; you have used a binary operator '-o' with nothing before it.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ca1e7a3b4a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Failing to archive the bundles should not mark the build as failed.
This can happen if a build is terminated early, or if (to be implemented)
an optional build-stage is skipped / failed;
```
2019-08-24T10:53:09.354Z] + bundleName=janky
[2019-08-24T10:53:09.354Z] + echo Creating janky-bundles.tar.gz
[2019-08-24T10:53:09.354Z] Creating janky-bundles.tar.gz
[2019-08-24T10:53:09.354Z] + xargs tar -czf janky-bundles.tar.gz
[2019-08-24T10:53:09.354Z] + find bundles -path */root/*overlay2 -prune -o -type f ( -name *-report.json -o -name *.log -o -name *.prof -o -name *-report.xml ) -print
[2019-08-24T10:53:09.354Z] find: bundles: No such file or directory
[2019-08-24T10:53:09.354Z] tar: Cowardly refusing to create an empty archive
[2019-08-24T10:53:09.354Z] Try 'tar --help' or 'tar --usage' for more information.
Error when executing always post condition:
hudson.AbortException: script returned exit code 123
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:569)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:515)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:461)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a76ff632a4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `libprotobuf-c0-dev` virtual package is no longer available
in Debian Buster, but is provided by `libprotobuf-c-dev`, which
is available.
https://packages.debian.org/stretch/libprotobuf-c0-dev
> Virtual Package: libprotobuf-c0-dev
>
> This is a virtual package. See the Debian policy for a definition of virtual packages.
>
> Packages providing libprotobuf-c0-dev
> libprotobuf-c-dev
> Protocol Buffers C static library and headers (protobuf-c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d185ca78ec)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This symlink was added in d42753485b,
to allow finding the path to the latest built binary, because at the time,
those paths were prefixed with the version or commit (e.g. `bundles/1.5.0-dev`).
Commit bac2447964 removed the version-prefix in
paths, but kept the old symlink for backward compatiblity. However, many
things were moved since then (e.g. paths were renamed to `binary-daemon`,
and various other changes). With the symlink pointing to the symlink's parent
directory, following the symlink may result into an infinite recursion,
which can happen if scripts using wildcards / globbing to find files.
With this symlink no longer serving a real purpose, we can probably safely
remove this symlink now.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dde1fd78c7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This will run the DCO check in a lightweight alpine container, before
running other stages, and before building the development image/container
(which can take a long time).
A Jenkins parameter was added to optionally skip the DCO check (skip_dco)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d6f7909c76)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Using a build-arg so that we don't have to specify it for each
`apt-get install`, and to preserve that the `DEBIAN_FRONTEND` is
preserved in the image itself (which changes the default behavior,
and can be surprising if the image is run interactively).`
With this patch, some (harmless, but possibly confusing) errors
are no longer printed during build, for example:
```patch
Unpacking libgcc1:armhf (1:6.3.0-18+deb9u1) ...
Selecting previously unselected package libc6:armhf.
Preparing to unpack .../04-libc6_2.24-11+deb9u4_armhf.deb ...
-debconf: unable to initialize frontend: Dialog
-debconf: (TERM is not set, so the dialog frontend is not usable.)
-debconf: falling back to frontend: Readline
Unpacking libc6:armhf (2.24-11+deb9u4) ...
Selecting previously unselected package libgcc1:arm64.
Preparing to unpack .../05-libgcc1_1%3a6.3.0-18+deb9u1_arm64.deb ...
Unpacking libgcc1:arm64 (1:6.3.0-18+deb9u1) ...
Selecting previously unselected package libc6:arm64.
Preparing to unpack .../06-libc6_2.24-11+deb9u4_arm64.deb ...
-debconf: unable to initialize frontend: Dialog
-debconf: (TERM is not set, so the dialog frontend is not usable.)
-debconf: falling back to frontend: Readline
```
Looks like some output is now also printed on stdout instead of stderr
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2ff9ac4de5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is basically taking some stuff that make a custom shell function
for.
This takes a test filter, builds the appropriate TESTFLAGS, and sets the
integration API test dirs that match the given filter to avoid building
all test dirs.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 13064b155e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Adds `DOCKER_BINDDIR_MOUNT_OPTS` to easily tweak the BINDDIR mount
options... primarily adding so I can control the caching mode for
osxfs because compiling takes > 1min for me with the default and < 30s
with both `cached` and `delegated`.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit b1e6536ceb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Attempting to fix;
```
21:16:00 Traceback (most recent call last):
21:16:00 File "/usr/local/bin/yamllint", line 11, in <module>
21:16:00 sys.exit(run())
21:16:00 File "/usr/local/lib/python3.5/dist-packages/yamllint/cli.py", line 170, in run
21:16:00 problems = linter.run(f, conf, filepath)
21:16:00 File "/usr/local/lib/python3.5/dist-packages/yamllint/linter.py", line 233, in run
21:16:00 content = input.read()
21:16:00 File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
21:16:00 return codecs.ascii_decode(input, self.errors)[0]
21:16:00 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 123522: ordinal not in range(128)
21:16:00 Build step 'Execute shell' marked build as failure
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b5e5cac0f5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This also makes sure that we can test all functionality of the
daemon, because some features are not available on static binaries.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4ddb40ee8a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- SC2006: use $(...) notation instead of legacy backticked `...`
- SC2086: double quote to prevent globbing and word splitting
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0b3d201892)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Jenkins groups them per stage, so collecting them for all architectures
is possible (without them conflicting or becoming ambiguous)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e2f5b78e78)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This will send the results directly after the tests complete,
and make the stage more atomic.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7f9328ad2e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case a job fails before even generating a report file.
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit 0cfc1ec2bd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The z-master stage will just run the integration-cli tests. The
existing z stage will run the unit tests and the integration
tests. In this way, PR check jobs will be shorter, but all
integration tests will run after PR is merged to master.
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit bdc1c1a02a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The powerpc-master stage will just run the integration-cli tests. The
existing powerpc stage will run the unit tests and the integration
tests. In this way, PR check jobs will be shorter, but all integration
tests will run after PR is merged to master.
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit c2f9d58375)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
There are many errors like this one:
> 01:39:28.750 find: ‘bundles/test-integration/dbc77018d39a5/root/overlay2/f49953a883daceee60a481dd8e1e37b0f806d309258197d6ba0f6871236d3d47/work/work’: Permission denied
(probably caused by bad permissions)
These directories are not to be looked at when we search for logs, so
let's exclude them. It's not super easy to do in find, here is some
kind of an explanation for find arguments
```
PATTERN ACTION OR PATTERN ACTION
-path X -prune -o -type f [AND] (-name A -o name B) -print
```
(here -o means OR, while AND is implicit)
While at it,
- let the find know we're only looking for files, not directories
- remove a subshell and || true
- remove `-name integration.test` (there are no such files)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b283dff3ff)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was inadvertedly removed in 7bfe48cc00,
because it was documented as a dependency for docker-py, but
actually used to validate the swagger file.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b1723b3721)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
and remove `PullImageTest::test_build_invalid_platform` from the list,
which was a copy/paste error in f8cde0b32d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6f5c377ddc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
All tests that require experimental either spin up a separate daemon,
or use the main daemon if experimental is enabled.
This patch
- allows enabling "experimental" for stages through an environment variable
- enables experimental by default on all stages, so that some of these tests
don't have to start a new daemon.
- removes the seaprate "experimental" stage, because it was running exactly
the same tests as the "janky" stage.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e856b46cfb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These tests are fixed upstream, but those fixes are not yet in a
released version.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f8cde0b32d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The .git mount is only needed for the DCO check, and for building
the binaries if `DOCKER_GITCOMMIT` is not set.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 47ac8a97de)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Having this information can help debugging issues in CI (which could
be caused by missing/incorrect configuration of the machines).
We ping to a fixed version of the script, because this script is ran
directly on the host, and we don't want pull-requests modifying this
script to have direct access to the machines.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a2ad56dfad)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Both of these tests are fairly short, and shouldn't interfer with
eachother, so we can combine them and re-use the same dev-image
(so that it'll only be built once).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f51c139792)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This patch removes the manual steps to resolve the Git commit, and
instead, uses the `GIT_COMMIT` that's set by Jenkins's Git plugin.
Behavior changes slightly, because `GIT_PLUGIN` contains the full
commit-sha, not the short one.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit be0e6e9d34)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Container and image names are already unique because they have
the git-sha or build-number, and a single machine won't be running
tests for multiple architectures.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 337d03a5f0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The main Dockerfile is multi-arch now.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 61fd8b7384)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
From the code style guidelines;
https://wiki.jenkins.io/display/JENKINS/Code+Style+Guidelines
> 1. Use spaces. Tabs are banned.
> 2. Java blocks are 4 spaces. JavaScript blocks as for Java. XML nesting is 2 spaces
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a95f16ca28)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
1. Use `go list` to get list of integration dirs to build. This means we
do not need to have a valid `.go` in every subdirectory and also
filters out other dirs like "bundles" which may have been created.
2. Add option to specify custom flags for integration and
integration-cli. This is needed so both suites can be run AND set
custom flags... since the cli suite does not support standard go
flags.
3. Add options to skip an entire integration suite.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit abece9b562)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit c222c5ac6f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
instead of vfs
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit ccfaf1ed92)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Codecov has shown to be flaky, and calculate the wrong diff, in
addition, it doesn't show coverage for integration tests, which
makes the coverage report not useful.
Removing it for now, while we look at alternatives.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit bd5c5373f1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When connecting with the daemon using a UNIX socket, the HTTP hostname was set, based
on the socket location, which was generating some noise in the test-logs.
Given that the actual hostname is not important (the URL just has to be well-formed),
the hostname/address can be cleaned up to reduce the noise.
This patch strips the path from the `addr`, and keeps `<random-id>.sock` as address.
Before:
daemon.go:329: [d15d31ba75501] error pinging daemon on start: Get http://%2Ftmp%2Fdocker-integration%2Fd15d31ba75501.sock/_ping: dial unix /tmp/docker-integration/d15d31ba75501.sock: connect: no such file or directory
After:
daemon.go:329: [d15d31ba75501] error pinging daemon on start: Get http://d15d31ba75501.sock/_ping: dial unix /tmp/docker-integration/d15d31ba75501.sock: connect: no such file or directory
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 92e6e7dd5f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The test-integration/test=integration-cli directory contains
a directory for each daemon that was created during the integration
tests, which makes it a long list to browse through. In addition,
some tests spin up multiple daemons, and when debugging test-failures,
the daemon-logs often have to be looked at together.
This patch organizes the bundl directory to group daemon storage
locationos per test, making it easier to find information about
all the daemons that were used in a specific test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9b5e78888d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
See if networking works if we run it first
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6aafe0fd9e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When building this image docker-in-docker, the DNS in the environment
may not be usable for the build-container, causing resolution to fail:
```
02:35:31 W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release.gpg Temporary failure resolving 'deb.debian.org'
```
This patch detects if we're building from within a container, and if
so, skips creating a networking namespace for the build by using
`--network=host`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3c15cea650)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This removes all the installation steps for docker-py from the
Dockerfile, and instead builds the upstream Dockerfile, and runs
docker-py tests in a container.
To test;
```
make test-docker-py
...
Removing bundles/
---> Making bundle: dynbinary (in bundles/dynbinary)
Building: bundles/dynbinary-daemon/dockerd-dev
Created binary: bundles/dynbinary-daemon/dockerd-dev
---> Making bundle: test-docker-py (in bundles/test-docker-py)
---> Making bundle: .integration-daemon-start (in bundles/test-docker-py)
Using test binary docker
Starting dockerd
INFO: Waiting for daemon to start...
.
INFO: Building docker-sdk-python3:3.7.0...
sha256:686428ae28479e9b5c8fdad1cadc9b7a39b462e66bd13a7e35bd79c6a152a402
INFO: Starting docker-py tests...
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-4.1.0, py-1.8.0, pluggy-0.9.0
rootdir: /src, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
collected 359 items
tests/integration/api_build_test.py .......s....
....
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7bfe48cc00)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
integration-on-swarm had unnecessary complexity and was too hard to
maintain. Also, it didn't support the new non-CLI integration test suite.
I'm now doing some experiments out of the repo using Kubernetes:
https://github.com/AkihiroSuda/kube-moby-integration
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit e7fbe8e457)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test is failing on Windows currently:
```
11:59:47 --- FAIL: TestHealthKillContainer (8.12s)
11:59:47 health_test.go:57: assertion failed: error is not nil: Error response from daemon: Invalid signal: SIGUSR1
``
That test was added recently in https://github.com/moby/moby/pull/39454, but
rewritten in a commit in the same PR:
f8aef6a92f
In that rewrite, there were some changes:
- originally it was skipped on Windows, but the rewritten test doesn't have that skip:
```go
testRequires(c, DaemonIsLinux) // busybox doesn't work on Windows
```
- the original test used `SIGINT`, but the new one uses `SIGUSR1`
Analysis:
- The Error bubbles up from: 8e610b2b55/pkg/signal/signal.go (L29-L44)
- Interestingly; `ContainerKill` should validate if a signal is valid for the given platform, but somehow we don't hit that part; f1b5612f20/daemon/kill.go (L40-L48)
- Windows only looks to support 2 signals currently 8e610b2b55/pkg/signal/signal_windows.go (L17-L26)
- Upstream Golang looks to define `SIGINT` as well; 77f9b2728e/src/runtime/defs_windows.go (L44)
- This looks like the current list of Signals upstream in Go; 3b58ed4ad3/windows/types_windows.go (L52-L67)
```go
const (
// More invented values for signals
SIGHUP = Signal(0x1)
SIGINT = Signal(0x2)
SIGQUIT = Signal(0x3)
SIGILL = Signal(0x4)
SIGTRAP = Signal(0x5)
SIGABRT = Signal(0x6)
SIGBUS = Signal(0x7)
SIGFPE = Signal(0x8)
SIGKILL = Signal(0x9)
SIGSEGV = Signal(0xb)
SIGPIPE = Signal(0xd)
SIGALRM = Signal(0xe)
SIGTERM = Signal(0xf)
)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit eeaa0b30d4)
Signed-off-by: Dani Louca <dani.louca@docker.com>
Docker daemon always stops healthcheck before sending signal to a
container now. However, when we use "docker kill" to send signals
other than SIGTERM or SIGKILL to a container, such as SIGINT,
daemon still stops container health check though container process
handles the signal normally and continues to work.
Signed-off-by: Ruilin Li <liruilin4@huawei.com>
(cherry picked from commit da574f9343)
Signed-off-by: Dani Louca <dani.louca@docker.com>
go1.12.8 (released 2019/08/13) includes security fixes to the net/http and net/url packages.
See the Go 1.12.8 milestone on our issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.12.8
- net/http: Denial of Service vulnerabilities in the HTTP/2 implementation
net/http and golang.org/x/net/http2 servers that accept direct connections from untrusted
clients could be remotely made to allocate an unlimited amount of memory, until the program
crashes. Servers will now close connections if the send queue accumulates too many control
messages.
The issues are CVE-2019-9512 and CVE-2019-9514, and Go issue golang.org/issue/33606.
Thanks to Jonathan Looney from Netflix for discovering and reporting these issues.
This is also fixed in version v0.0.0-20190813141303-74dc4d7220e7 of golang.org/x/net/http2.
net/url: parsing validation issue
- url.Parse would accept URLs with malformed hosts, such that the Host field could have arbitrary
suffixes that would appear in neither Hostname() nor Port(), allowing authorization bypasses
in certain applications. Note that URLs with invalid, not numeric ports will now return an error
from url.Parse.
The issue is CVE-2019-14809 and Go issue golang.org/issue/29098.
Thanks to Julian Hector and Nikolai Krein from Cure53, and Adi Cohen (adico.me) for discovering
and reporting this issue.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 73b0e4c589)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
1. Use "in-place" variables for if statements to limit their scope to
the respectful `if` block.
2. Report the error returned from sd_journal_* by using CErr().
3. Use errors.New() instead of fmt.Errorf().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 20a0e58a79)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
From the first glance, `docker logs --tail 0` does not make sense,
as it is supposed to produce no output, but `tail -n 0` from GNU
coreutils is working like that, plus there is even a test case
(`TestLogsTail` in integration-cli/docker_cli_logs_test.go).
Now, something like `docker logs --follow --tail 0` makes total
sense, so let's make it work.
(NOTE if --tail is not used, config.Tail is set to -1)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit dd4bfe30a8)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
If we take a long time to process log messages, and during that time
journal file rotation occurs, the journald client library will keep
those rotated files open until sd_journal_process() is called.
By periodically calling sd_journal_process() during the processing
loop we shrink the window of time a client instance has open file
descriptors for rotated (deleted) journal files.
This code is modelled after that of journalctl [1]; the above explanation
as well as the value of 1024 is taken from there.
[v2: fix CErr() argument]
[1] https://github.com/systemd/systemd/blob/dc16327c48d/src/journal/journalctl.c#L2676
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b73fb8fd5d)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
TL;DR: simplify the code, fix --follow hanging indefinitely
Do the following to simplify the followJournal() code:
1. Use Go-native select instead of C-native polling.
2. Use Watch{Producer,Consumer}Gone(), eliminating the need
to have journald.closed variable, and an extra goroutine.
3. Use sd_journal_wait(). In the words of its own man page:
> A synchronous alternative for using sd_journal_get_fd(),
> sd_journal_get_events(), sd_journal_get_timeout() and
> sd_journal_process() is sd_journal_wait().
Unfortunately, the logic is still not as simple as it
could be; the reason being, once the container has exited,
journald might still be writing some logs from its internal
buffers onto journal file(s), and there is no way to
figure out whether it's done so we are guaranteed to
read all of it back. This bug can be reproduced with
something like
> $ ID=$(docker run -d busybox seq 1 150000); docker logs --follow $ID
> ...
> 128123
> $
(The last expected output line should be `150000`).
To avoid exiting from followJournal() early, add the
following logic: once the container is gone, keep trying
to drain the journal until there's no new data for at
least `waitTimeout` time period.
Should fix https://github.com/docker/for-linux/issues/575
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f091febc94)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. The journald client library initializes inotify watch(es)
during the first call to sd_journal_get_fd(), and it make sense
to open it earlier in order to not lose any journal file rotation
events.
2. It only makes sense to call this if we're going to use it
later on -- so add a check for config.Follow.
3. Remove the redundant call to sd_journal_get_fd().
NOTE that any subsequent calls to sd_journal_get_fd() return
the same file descriptor, so there's no real need to save it
for later use in wait_for_data_cancelable().
Based on earlier patch by Nalin Dahyabhai <nalin@redhat.com>.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 981c01665b)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case the LogConsumer is gone, the code that sends the message can
stuck forever. Wrap the code in select case, as all other loggers do.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 79039720c8)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case Tail=N parameter is requested, we need to show N lines.
It does not make sense to walk backwards one by one if we can
do it at once. Now, if Since=T is also provided, make sure we
haven't jumped too far (before T), and if we did, move forward.
The primary motivation for this was to make the code simpler.
This also fixes a tiny bug in the "since" implementation.
Before this commit:
> $ docker logs -t --tail=6000 --since="2019-03-10T03:54:25.00" $ID | head
> 2019-03-10T03:54:24.999821000Z 95981
After:
> $ docker logs -t --tail=6000 --since="2019-03-10T03:54:25.00" $ID | head
> 2019-03-10T03:54:25.000013000Z 95982
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ff3cd167ea)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Minor code simplification.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit e8f6166791)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Clean up a deferred function call in the journal reading logic.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
(cherry picked from commit 1ada3e85bf)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There are a few more places, apparently, that List operations against
Swarm exist, besides just in the List methods. This increases the max
received message size in those places.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit a84a78e976)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
3029e765e2 broke compilation on
non-Linux/Windows systems.
This change fixes that.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 34d5b8867f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is needed to provide fixes for ETW on ARM. The updated ETW package will
no-op on ARM, rather than crashing. Further changes are needed to Go itself to
allow ETW on ARM to work properly.
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
(cherry picked from commit e1f0f77bf4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 961ec3a56b...4fb9e961ab
included:
- docker/swarmkit#2873 [19.03 backport] Only update non-terminal tasks on node removal
- backport of docker/swarmkit#2867 Only update non-terminal tasks on node removal
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fixes a regression introduced in 6d87f19142,
causing `COPY --from` to fail if the target directory does not exist:
```
FROM mcr.microsoft.com/windows/servercore:ltsc2019 as s1
RUN echo "Hello World" > /hello
FROM mcr.microsoft.com/windows/servercore:ltsc2019
COPY --from=s1 /hello /hello/another/world
```
Would produce an error:
```
Step 4/4 : COPY --from=s1 /hello /hello/another/world
failed to copy files: mkdir \\?: The filename, directory name, or volume label syntax is incorrect.
```
The cause for this was that Go's `os.MkdirAll()` does not support/detect volume GUID paths
(`\\?\Volume{dae8d3ac-b9a1-11e9-88eb-e8554b2ba1db}\hello\another}`), and as a result
attempted to create the volume as a directory (`\\?`), causing it to fail.
This patch replaces `os.MkdirAll()` with our own `system.MkdirAll()` function, which
is capable of detecting GUID volumes.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5858a99267)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This function changed to the correct working directory before starting the tests
(which is the same as on Linux), however the `ProcessStartInfo` process does
not inherit this working directory, which caused Windows tests to be running
with a different working directory as Linux (causing files used in tests to not
be found).
From the documentation; https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.workingdirectory?view=netframework-4.8
> When `UseShellExecute` is `true`, the fully qualified name of the directory that contains
> the process to be started. When the `UseShellExecute` property is `false`, the working
> directory for the process to be started. The default is an empty string (`""`).
This patch sets the `ProcessStartInfo.WorkingDirectory` to the correct working
directory before starting the process.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6ae46aeabf)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Golang base images switch to buster, which causes some breakage
in networking and packages that are no longer available; (`btrfs-tools`
is now an empty package, and `libprotobuf-c0-dev` is gone).
Some of out tests also start faiilng on stretch, and will have to be
investigated further;
```
15:13:06 --- FAIL: TestRenameAnonymousContainer (3.37s)
15:13:06 rename_test.go:168: assertion failed: 0 (int) != 1 (inspect.State.ExitCode int): container a7fe866d588d65f353f42ffc5ea5288e52700384e1d90850e9c3d4dce8657666 exited with the wrong exitcode:
15:13:38 --- FAIL: TestHostnameDnsResolution (2.23s)
15:13:38 run_linux_test.go:128: assertion failed:
15:13:38 --- ←
15:13:38 +++ →
15:13:38 @@ -1 +1,2 @@
15:13:38 +ping: bad address 'foobar'
15:13:38
15:13:38
15:13:38 run_linux_test.go:129: assertion failed: 0 (int) != 1 (res.ExitCode int)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ed672bb523)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This check was used to make sure we don't bump Go versions independently
(Linux/Windows). The Dockerfile switched to using a build-arg to allow
overriding the Go version, which rendered this check non-functional.
It also fails if Linux versions use a specific variant of the image;
08:41:31 ERROR: Failed 'ERROR: Mismatched GO versions between Dockerfile and Dockerfile.windows. Update your PR to ensure that both files are updated and in sync. ${GO_VERSION}-stretch ${GO_VERSION}' at 07/20/2019 08:41:31
08:41:31 At C:\gopath\src\github.com\docker\docker\hack\ci\windows.ps1:448 char:9
08:41:31 + Throw "ERROR: Mismatched GO versions between Dockerfile and D ...
08:41:31 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the check by looking for the value of `GO_VERSION` instead
of looking at the `FROM` line (which is harder to parse).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4fa57a8191)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This allows overriding the version of Go without making modifications in the
source code, which can be useful to test against multiple versions.
For example:
make GO_VERSION=1.13beta1 shell
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c6281bc438)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The fastly cdn mirror we're using also mirrors the debian security
repository;
```
Welcome to deb.debian.org (fastly instance)!
This is deb.debian.org. This service provides mirrors for the following Debian archive repositories:
/debian/
/debian-debug/
/debian-ports/
/debian-security/
The server deb.debian.org does not have packages itself, but the name has SRV records in DNS that let apt in stretch and later find places.
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c8f43b5f6f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Protect access to q.quotas map, and lock around changing nextProjectID.
Techinically, the lock in findNextProjectID() is not needed as it is
only called during initialization, but one can never be too careful.
Fixes: 52897d1c09 ("projectquota: utility class for project quota controls")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 1ac0a66a64)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
environment not in the chroot from untrusted files.
See also OpenVZ a3f732ef75/src/enter.c (L227-L234)
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
(cherry picked from commit cea6dca993c2b4cfa99b1e7a19ca134c8ebc236b)
Signed-off-by: Tibor Vass <tibor@docker.com>
Path-specific rules were removed, so this is no longer used.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 530e63c1a61b105a6f7fc143c5acb9b5cd87f958)
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit f8a0f26843)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 77b8465d7e added a secret update
endpoint to allow updating labels on existing secrets. However, when
implementing the endpoint, the DebugRequestMiddleware was not updated
to scrub the Data field (as is being done when creating a secret).
When updating a secret (to set labels), the Data field should be either
`nil` (not set), or contain the same value as the existing secret. In
situations where the Data field is set, and the `dockerd` daemon is
running with debugging enabled / log-level debug, the base64-encoded
value of the secret is printed to the daemon logs.
The docker cli does not have a `docker secret update` command, but
when using `docker stack deploy`, the docker cli sends the secret
data both when _creating_ a stack, and when _updating_ a stack, thus
leaking the secret data if the daemon runs with debug enabled:
1. Start the daemon in debug-mode
dockerd --debug
2. Initialize swarm
docker swarm init
3. Create a file containing a secret
echo secret > my_secret.txt
4. Create a docker-compose file using that secret
cat > docker-compose.yml <<'EOF'
version: "3.3"
services:
web:
image: nginx:alpine
secrets:
- my_secret
secrets:
my_secret:
file: ./my_secret.txt
EOF
5. Deploy the stack
docker stack deploy -c docker-compose.yml test
6. Verify that the secret is scrubbed in the daemon logs
DEBU[2019-07-01T22:36:08.170617400Z] Calling POST /v1.30/secrets/create
DEBU[2019-07-01T22:36:08.171364900Z] form data: {"Data":"*****","Labels":{"com.docker.stack.namespace":"test"},"Name":"test_my_secret"}
7. Re-deploy the stack to trigger an "update"
docker stack deploy -c docker-compose.yml test
8. Notice that this time, the Data field is not scrubbed, and the base64-encoded secret is logged
DEBU[2019-07-01T22:37:35.828819400Z] Calling POST /v1.30/secrets/w3hgvwpzl8yooq5ctnyp71v52/update?version=34
DEBU[2019-07-01T22:37:35.829993700Z] form data: {"Data":"c2VjcmV0Cg==","Labels":{"com.docker.stack.namespace":"test"},"Name":"test_my_secret"}
This patch modifies `maskSecretKeys` to unconditionally scrub `Data` fields.
Currently, only the `secrets` and `configs` endpoints use a field with this
name, and no other POST API endpoints use a data field, so scrubbing this
field unconditionally will only scrub requests for those endpoints.
If a new endpoint is added in future where this field should not be scrubbed,
we can re-introduce more fine-grained (path-specific) handling.
This patch introduces some change in behavior:
- In addition to secrets, requests to create or update _configs_ will
now have their `Data` field scrubbed. Generally, the actual data should
not be interesting for debugging, so likely will not be problematic.
In addition, scrubbing this data for configs may actually be desirable,
because (even though they are not explicitely designed for this purpose)
configs may contain sensitive data (credentials inside a configuration
file, e.g.).
- Requests that send key/value pairs as a "map" and that contain a
key named "data", will see the value of that field scrubbed. This
means that (e.g.) setting a `label` named `data` on a config, will
scrub/mask the value of that label.
- Note that this is already the case for any label named `jointoken`,
`password`, `secret`, `signingcakey`, or `unlockkey`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c7ce4be93ae8edd2da62a588e01c67313a4aba0c)
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 73db8c77bf)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 32d70c7e21631224674cd60021d3ec908c2d888c)
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit ebb542b3f8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add tests for
- case-insensitive matching of fields
- recursive masking
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit db5f811216e70bcb4a10e477c1558d6c68f618c5)
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 18dac2cf32)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Co-Authored-By: Jim Ehrismann <jim-docker@users.noreply.github.com>
Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com>
Signed-off-by: Jim Ehrismann <jim.ehrismann@docker.com>
(cherry picked from commit a77e147d32)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This will add a warning log in the daemon, and will send the message
to be displayed by the CLI.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit d35f8f4329)
Signed-off-by: Tibor Vass <tibor@docker.com>
This reverts commit 98fc09128b in order to
keep registry v2 schema1 handling and libtrust-key-based engine ID.
Because registry v2 schema1 was not officially deprecated and
registries are still relying on it, this patch puts its logic back.
However, registry v1 relics are not added back since v1 logic has been
removed a while ago.
This also fixes an engine upgrade issue in a swarm cluster. It was relying
on the Engine ID to be the same upon upgrade, but the mentioned commit
modified the logic to use UUID and from a different file.
Since the libtrust key is always needed to support v2 schema1 pushes,
that the old engine ID is based on the libtrust key, and that the engine ID
needs to be conserved across upgrades, adding a UUID-based engine ID logic
seems to add more complexity than it solves the problems.
Hence reverting the engine ID changes as well.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit f695e98cb7)
Signed-off-by: Tibor Vass <tibor@docker.com>
This reverts commit 13b7d11be1.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit f23a51a860)
Signed-off-by: Tibor Vass <tibor@docker.com>
Previously, getWalkRoot("/", "foo") would return "//foo"
Now it returns "/foo"
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 7410f1a859)
Signed-off-by: Tibor Vass <tibor@docker.com>
Before 7a7357da, archive.TarResourceRebase was being used to copy files
and folders from the container. That function splits the source path
into a dirname + basename pair to support copying a file:
if you wanted to tar `dir/file` it would tar from `dir` the file `file`
(as part of the IncludedFiles option).
However, that path splitting logic was kept for folders as well, which
resulted in weird inputs to archive.TarWithOptions:
if you wanted to tar `dir1/dir2` it would tar from `dir1` the directory
`dir2` (as part of IncludedFiles option).
Although it was weird, it worked fine until we started chrooting into
the container rootfs when doing a `docker cp` with container source set
to `/` (cf 3029e765).
The fix is to only do the path splitting logic if the source is a file.
Unfortunately, 7a7357da added support for LCOW by duplicating some of
this subtle logic. Ideally we would need to do more refactoring of the
archive codebase to properly encapsulate these behaviors behind well-
documented APIs.
This fix does not do that. Instead, it fixes the issue inline.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 171538c190)
Signed-off-by: Tibor Vass <tibor@docker.com>
CID=$(docker create alpine)
docker cp $CID:/ out
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 6db9f1c3d6)
Signed-off-by: Tibor Vass <tibor@docker.com>
retries to attach to a network, it is already connected to
Fixes - https://github.com/docker/for-linux/issues/632
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
(cherry picked from commit 871acb1c86)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test runs on a daemon also used by other tests
so make sure we don't get failures if another test
doesn't cleanup or is running in parallel.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 915acffdb4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This adds some logs, handles timers better, and sets a request timeout
for the ping request.
I'm not sure the ticker in that loop is what we really want since the
ticker keeps ticking while we are (attempting) to make a request... but
I opted to not change that for now.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 20ea8942b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Moby currently sorts uid and gid ranges in id maps. This causes subuid
and subgid files to be interpreted wrongly.
The subuid file
```
> cat /etc/subuid
jonas:100000:1000
jonas:1000:1
```
configures that the container uids 0-999 are mapped to the host uids
100000-100999 and uid 1000 in the container is mapped to uid 1000 on the
host. The expected uid_map is:
```
> docker run ubuntu cat /proc/self/uid_map
0 100000 1000
1000 1000 1
```
Moby currently sorts the ranges by the first id in the range. Therefore
with the subuid file above the uid 0 in the container is mapped to uid
100000 on host and the uids 1-1000 in container are mapped to the uids
1-1000 on the host. The resulting uid_map is:
```
> docker run ubuntu cat /proc/self/uid_map
0 1000 1
1 100000 1000
```
The ordering was implemented to work around a limitation in Linux 3.8.
This is fixed since Linux 3.9 as stated on the user namespaces manpage
[1]:
> In the initial implementation (Linux 3.8), this requirement was
> satisfied by a simplistic implementation that imposed the further
> requirement that the values in both field 1 and field 2 of successive
> lines must be in ascending numerical order, which prevented some
> otherwise valid maps from being created. Linux 3.9 and later fix this
> limitation, allowing any valid set of nonoverlapping maps.
This fix changes the interpretation of subuid and subgid files which do
not have the ids of in the numerical order for each individual user.
This breaks users that rely on the current behaviour.
The desired mapping above - map low user ids in the container to high
user ids on the host and some higher user ids in the container to lower
user on host - can unfortunately not archived with the current
behaviour.
[1] http://man7.org/linux/man-pages/man7/user_namespaces.7.html
Signed-off-by: Jonas Dohse <jonas@dohse.ch>
(cherry picked from commit c4628d79d2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Running a bundled aufs benchmark sometimes results in this warning:
> WARN[0001] Couldn't run auplink before unmount /tmp/aufs-tests/aufs/mnt/XXXXX error="exit status 22" storage-driver=aufs
If we take a look at what aulink utility produces on stderr, we'll see:
> auplink:proc_mnt.c:96: /tmp/aufs-tests/aufs/mnt/XXXXX: Invalid argument
and auplink exits with exit code of 22 (EINVAL).
Looking into auplink source code, what happens is it tries to find a
record in /proc/self/mounts corresponding to the mount point (by using
setmntent()/getmntent_r() glibc functions), and it fails.
Some manual testing, as well as runtime testing with lots of printf
added on mount/unmount, as well as calls to check the superblock fs
magic on mount point (as in graphdriver.Mounted(graphdriver.FsMagicAufs, target)
confirmed that this record is in fact there, but sometimes auplink
can't find it. I was also able to reproduce the same error (inability
to find a mount in /proc/self/mounts that should definitely be there)
using a small C program, mocking what `auplink` does:
```c
#include <stdio.h>
#include <err.h>
#include <mntent.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char **argv)
{
FILE *fp;
struct mntent m, *p;
char a[4096];
char buf[4096 + 1024];
int found =0, lines = 0;
if (argc != 2) {
fprintf(stderr, "Usage: %s <mountpoint>\n", argv[0]);
exit(1);
}
fp = setmntent("/proc/self/mounts", "r");
if (!fp) {
err(1, "setmntent");
}
setvbuf(fp, a, _IOLBF, sizeof(a));
while ((p = getmntent_r(fp, &m, buf, sizeof(buf)))) {
lines++;
if (!strcmp(p->mnt_dir, argv[1])) {
found++;
}
}
printf("found %d entries for %s (%d lines seen)\n", found, argv[1], lines);
return !found;
}
```
I have also wrote a few other C proggies -- one that reads
/proc/self/mounts directly, one that reads /proc/self/mountinfo instead.
They are also prone to the same occasional error.
It is not perfectly clear why this happens, but so far my best theory
is when a lot of mounts/unmounts happen in parallel with reading
contents of /proc/self/mounts, sometimes the kernel fails to provide
continuity (i.e. it skips some part of file or mixes it up in some
other way). In other words, this is a kernel bug (which is probably
hard to fix unless some other interface to get a mount entry is added).
Now, there is no real fix, and a workaround I was able to come up
with is to retry when we got EINVAL. It usually works on the second
attempt, although I've once seen it took two attempts to go through.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ae431b10a9)
Do not use filepath.Walk() as there's no requirement to recursively
go into every directory under mnt -- a (non-recursive) list of
directories in mnt is sufficient.
With filepath.Walk(), in case some container will fail to unmount,
it'll go through the whole container filesystem which is both
excessive and useless.
This is similar to commit f1a4592297 ("devmapper.shutdown:
optimize")
While at it, raise the priority of "unmount error" message from debug
to a warning. Note we don't have to explicitly add `m` as unmount error (from
pkg/mount) will have it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 8fda12c607)
In case there are a big number of layers, so that mount data won't fit
into a single memory page (4096 bytes on most platforms, which is good
enough for about 40 layers, depending on how long graphdriver root path
is), we supply additional layers with O_REMOUNT, as described in aufs
documentation.
Problem is, the current implementation does that one layer at a time
(i.e. there is one mount syscall per each additional layer).
Optimize the code to supply as many layers as we can fit in one page
(basically reusing the same code as for the original mount).
Note, per aufs docs, "[a]t remount-time, the options are interpreted
in the given order, e.g. left to right" so we should be good.
Tested on an image with ~100 layers.
Before (35 syscalls):
> [pid 22756] 1556919088.686955 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", "aufs", 0, "br:/mnt/volume_sfo2_09/docker-au"...) = 0 <0.000504>
> [pid 22756] 1556919088.687643 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c451b0, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000105>
> [pid 22756] 1556919088.687851 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c451ba, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000098>
> ..... (~30 lines skipped for clarity)
> [pid 22756] 1556919088.696182 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c45310, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000266>
After (2 syscalls):
> [pid 24352] 1556919361.799889 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/8e7ba189e347a834e99eea4ed568f95b86cec809c227516afdc7c70286ff9a20", "aufs", 0, "br:/mnt/volume_sfo2_09/docker-au"...) = 0 <0.001717>
> [pid 24352] 1556919361.801761 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/8e7ba189e347a834e99eea4ed568f95b86cec809c227516afdc7c70286ff9a20", 0xc000dbecb0, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.001358>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit d58c434bff)
Apparently there is some kind of race in aufs kernel module code,
which leads to the errors like:
[98221.158606] aufs au_xino_create2:186:dockerd[25801]: aufs.xino create err -17
[98221.162128] aufs au_xino_set:1229:dockerd[25801]: I/O Error, failed creating xino(-17).
[98362.239085] aufs au_xino_create2:186:dockerd[6348]: aufs.xino create err -17
[98362.243860] aufs au_xino_set:1229:dockerd[6348]: I/O Error, failed creating xino(-17).
[98373.775380] aufs au_xino_create:767:dockerd[27435]: open /dev/shm/aufs.xino(-17)
[98389.015640] aufs au_xino_create2:186:dockerd[26753]: aufs.xino create err -17
[98389.018776] aufs au_xino_set:1229:dockerd[26753]: I/O Error, failed creating xino(-17).
[98424.117584] aufs au_xino_create:767:dockerd[27105]: open /dev/shm/aufs.xino(-17)
So, we have to have a lock around mount syscall.
While at it, don't call the whole Unmount() on an error path, as
it leads to bogus error from auplink flush.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 5cd62852fa)
1. Use mount.Unmount() which ignores EINVAL ("not mounted") error,
and provides better error diagnostics (so we don't have to explicitly
add target to error messages).
2. Since we're ignoring "not mounted" error, we can call
multiple unmounts without any locking -- but since "auplink flush"
is still involved and can produce an error in logs, let's keep
the check for fs being mounted (it's just a statfs so should be fast).
2. While at it, improve the "can't unmount" error message in Put().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 4beee98026)
Both mount and unmount calls are already protected by fine-grained
(per id) locks in Get()/Put() introduced in commit fc1cf1911b
("Add more locking to storage drivers"), so there's no point in
having a global lock in mount/unmount.
The only place from which unmount is called without any locking
is Cleanup() -- this is to be addressed in the next patch.
This reverts commit 824c24e680.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f93750b2c4)
Increases the max recieved gRPC message size for Node and Secret list
operations. This has already been done for the other swarm types, but
was not done for these.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit a0903e1fa3)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit ca5aab19b4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously `docker info` had reported "cgroupfs" as the cgroup driver
but the driver wasn't actually used at all.
This PR reports "none" as the cgroup driver so as to avoid confusion.
e.g. kubeadm/kubelet will detect cgroupless-ness by checking this docker
info field. https://github.com/rootless-containers/usernetes/pull/97
Note that user still cannot specify `native.cgroupdriver=none` manually.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 153466ba0a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously only unpack operations were supported with chroot.
This adds chroot support for packing operations.
This prevents potential breakouts when copying data from a container.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 3029e765e2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is useful for preventing CVE-2018-15664 where a malicious container
process can take advantage of a race on symlink resolution/sanitization.
Before this change chrootarchive would chroot to the destination
directory which is attacker controlled. With this patch we always chroot
to the container's root which is not attacker controlled.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit d089b63937)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Reverts the change to swarmkit that made all updates set UpdateStatus to
Completed
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit c7d9599e3d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As pointed out by Tonis, there's a race between ReleaseRWLayer()
and GetRWLayer():
```
----- goroutine 1 ----- ----- goroutine 2 -----
ReleaseRWLayer()
m := ls.mounts[l.Name()]
...
m.deleteReference(l)
m.hasReferences()
... GetRWLayer()
... mount := ls.mounts[id]
ls.driver.Remove(m.mountID)
ls.store.RemoveMount(m.name) return mount.getReference()
delete(ls.mounts, m.Name())
----------------------- -----------------------
```
When something like this happens, GetRWLayer will return
an RWLayer without a storage. Oops.
There might be more races like this, and it seems the best
solution is to lock by layer id/name by using pkg/locker.
With this in place, name collision could not happen, so remove
the part of previous commit that protected against it in
CreateRWLayer (temporary nil assigmment and associated rollback).
So, now we have
* layerStore.mountL sync.Mutex to protect layerStore.mount map[]
(against concurrent access);
* mountedLayer's embedded `sync.Mutex` to protect its references map[];
* layerStore.layerL (which I haven't touched);
* per-id locker, to avoid name conflicts and concurrent operations
on the same rw layer.
The whole rig seems to look more readable now (mutexes use is
straightforward, no nested locks).
Reported-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit af433dd200)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is an additon to commit 1fea38856a ("Remove v1.10 migrator")
aka PR #38265. Since that one, CreateRWLayerByGraphID() is not
used anywhere, so let's drop it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b4e9b50765)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Goroutine stack analisys shown some lock contention
while doing massively (100 instances of `docker rm`)
parallel image removal, with many goroutines waiting
for the mountL mutex. Optimize it.
With this commit, the above operation is about 3x
faster, with no noticeable change to container
creation times (tested on aufs and overlay2).
kolyshkin@:
- squashed commits
- added description
- protected CreateRWLayer against name collisions by
temporary assiging nil to ls.mounts[name], and treating
nil as "non-existent" in all the other functions.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 05250a4f00)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a mutex to protect concurrent access to mountedLayer.references map.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f73b5cb4e8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Currently the API spec would allow `"443/tcp": [null]`, but what should
be allowed is `"443/tcp": null`
Signed-off-by: Dominic Tubach <dominic.tubach@to.com>
(cherry picked from commit 32b5d296ea)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Make sure adapter.removeNetworks executes during task Remove
adapter.removeNetworks was being skipped for cases when
isUnknownContainer(err) was true after adapter.remove was executed
This fix eliminates the nil return case forcing the function
to continue executing unless there is a true error
Fixes https://github.com/moby/moby/issues/39225
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
(cherry picked from commit 70fa7b6a3f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test is dependent on the search results returned by Docker Hub, which
can change at any moment, and causes this test to be unpredictable.
Removing this test instead of trying to catch up with Docker Hub any time
the results change, because it's effectively testing Docker Hub, and not
the daemon.
Unit tests are already in place to test the core functionality of the daemon,
so it should be safe to remove this test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 21e662c774)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Errors were being ignored and always telling the user that the path
doesn't exist even if it was some other problem, such as a permission
error.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit ebcef28834)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`chmod` is a legacy syscall, and not present on arm64, which
caused this test to fail.
Add `fchmodat` to the profile so that this test can run both
on x64 and arm64.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4bd8964b23)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 9ff9b57c34...5ac07abef4
brings in:
- docker/libnetwork#2376 Forcing a nil IP specified in PortBindings to IPv4zero (0.0.0.0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a66ddd8ab8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Sometimes this test fails (allegedly due to problems with Docker Hub),
but it fails later than it should, for example:
> 01:20:34.845 assertion failed: expression is false: strings.Count(outSearchCmdStars, "[OK]") <= strings.Count(outSearchCmd, "[OK]"): The quantity of images with stars should be less than that of all images: <...>
This, with non-empty list of images following, means that the initial
`docker search busybox` command returned not enough results. So, add
a check that `docker search busybox` returns something.
While at it,
* raise the number of stars to 10;
* simplify check for number of lines (no need to count [OK]'s);
* improve error message.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 4f80a1953d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Today `$ docker service create --limit-cpu` configures a containers
`CpuPeriod` and `CpuQuota` variables, this commit switches this to
configure a containers `NanoCpu` variable instead.
Signed-off-by: Olly Pomeroy <olly@docker.com>
(cherry picked from commit 8a60a1e14a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Killing the shell script process does not kill the forked process.
This commit switches to `exec` so that the executed process can be
easily killed.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 34cc5c24d0)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Now the child process is killed when the parent dies (rootless-containers/rootlesskit#66)
e92d5e7...27a0c7a
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 00c92a6719)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
no local changes, just syncing with containerd
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d6d2b30fd2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 08a7655d27...v1.2.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5d51ac544b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 8818c67cff...v0.5.0
- moby/buildkit#909 exporter: support unpack opt for image exporter
- moby/buildkit#961 dockerfile: allow subdirs for remote contexts
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3e4723cf33)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As of Go 1.12, the `testing` package panics if a goroutine logs to a
`testing.T` after the relevant test has completed. This was not
documented as a change at all; see the commit
95d06ab6c982f58b127b14a52c3325acf0bd3926 in the Go repository for the
relevant change.
At any point in the integration tests, tests could panic with the
message "Log in goroutine after TEST_FUNCTION has completed". This was
exacerbated by less direct logging I/O, e.g. running `make test` with
its output piped instead of attached to a TTY.
The most common cause of panics was that there was a race condition
between an exit logging goroutine and the `StopWithError` method:
`StopWithError` could return, causing the calling test method to return,
causing the `testing.T` to be marked as finished, before the goroutine
could log that the test daemon had exited. The fix is simple: capture
the result of `cmd.Wait()`, _then_ log, _then_ send the captured
result over the `Wait` channel. This ensures that the message is
logged before `StopWithError` can return, blocking the test method
so that the target `testing.T` is not marked as finished.
Signed-off-by: Daniel Sweet <danieljsweet@icloud.com>
(cherry picked from commit 7546322e99)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
make sure the LB sandbox is removed when a service is updated
with a --network-rm option
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
(cherry picked from commit 680d0ba4ab)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 029124da7a...425e105d5a
- opencontainers/runc#2043 Vendor in latest selinux code for keycreate errors
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4bc310c11b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 029124da7a...425e105d5a
- opencontainers/runc#2043 Vendor in latest selinux code for keycreate errors
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6df6fe6020)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `--rootless` flag had a couple of issues:
* #38702: euid=0, $USER="root" but no access to cgroup ("rootful" Docker in rootless Docker)
* #39009: euid=0 but $USER="docker" (rootful boot2docker)
To fix#38702, XDG dirs are ignored as in rootful Docker, unless the
dockerd is directly running under RootlessKit namespaces.
RootlessKit detection is implemented by checking whether `$ROOTLESSKIT_STATE_DIR` is set.
To fix#39009, the non-robust `$USER` check is now completely removed.
The entire logic can be illustrated as follows:
```
withRootlessKit := getenv("ROOTLESSKIT_STATE_DIR")
rootlessMode := withRootlessKit || cliFlag("--rootless")
honorXDG := withRootlessKit
useRootlessKitDockerProxy := withRootlessKit
removeCgroupSpec := rootlessMode
adjustOOMScoreAdj := rootlessMode
```
Close#39024Fix#38702#39009
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 3518383ed9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
lxc-user-nic can eliminate slirp overhead but needs /etc/lxc/lxc-usernet to be configured for the current user.
To use lxc-user-nic, $DOCKERD_ROOTLESS_ROOTLESSKIT_NET=lxc-user-nic also needs to be set.
This commit also bumps up RootlessKit from v0.3.0 to v0.4.0:
70e0502f32...e92d5e772e
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 63a66b0eb0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
1. There is no need to persist DOCKER_GITCOMMIT,
as it's not needed for runtime, only for build.
So, remove ENV.
2. In case $GITCOMMIT is not defined during build time
(and it happens if .git directory is not present),
we still need to have some value set, so set it to
`undefined`. Otherwise we'll have something like
> => ERROR [builder 2/3] RUN hack/make.sh build-integration-test-binary
> ------
> > [builder 2/3] RUN hack/make.sh build-integration-test-binary:
> #32 0.488
> #32 0.505 error: .git directory missing and DOCKER_GITCOMMIT not specified
> #32 0.505 Please either build with the .git directory accessible, or specify the
> #32 0.505 exact (--short) commit hash you are building using DOCKER_GITCOMMIT for
> #32 0.505 future accountability in diagnosing build issues. Thanks!
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit c3b24944ca)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Package "gotest.tools/assert" uses source introspection to
print more info in case of assertion failure. When source code
is not available, it prints an error instead.
In other words, before this commit:
> --- SKIP: TestCgroupDriverSystemdMemoryLimit (0.00s)
> cgroupdriver_systemd_test.go:32: failed to parse source file: /go/src/github.com/docker/docker/integration/system/cgroupdriver_systemd_test.go: open /go/src/github.com/docker/docker/integration/system/cgroupdriver_systemd_test.go: no such file or directory
> cgroupdriver_systemd_test.go:32:
and after:
> --- SKIP: TestCgroupDriverSystemdMemoryLimit (0.09s)
> cgroupdriver_systemd_test.go:32: !hasSystemd()
This increases the resulting image size by about 2 MB
on my system (from 758 to 760 MB).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 0deb18ab42)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test case requires not just daemon >= 1.40, but also
client API >= 1.40. In case older client is used, we'll
get failure from the very first check:
> ipcmode_linux_test.go:313: assertion failed: shareable (string) != private (string)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 1ada1c8391)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Older versions did not use an UUID as ID
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 05bd9958f2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The daemon may already have other volumes, so filter out those
when running the test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 566eea13e6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
TestNISDomainname in the integration suite covers this
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2b5880c2eb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Older versions of the daemon would concatenate hostname and
domainname, so hostname "foobar" and domainname "baz.cyphar.com"
would produce `foobar.baz.cyphar.com` as hostname.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c91c3776ea)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Dockerfile missed some fixtures, which caused this test
fail when running from this image.
I also noticed some other fixtures missing in integration-cli,
where the image had symlinks to some certificates, but the
original files were not included;
```
|-- integration-cli
|-- fixtures
| |-- auth
| | `-- docker-credential-shell-test
| |-- credentialspecs
| | `-- valid.json
| |-- https
| | |-- ca.pem -> ../../../integration/testdata/https/ca.pem
| | |-- client-cert.pem -> ../../../integration/testdata/https/client-cert.pem
| | |-- client-key.pem -> ../../../integration/testdata/https/client-key.pem
| | |-- client-rogue-cert.pem
| | |-- client-rogue-key.pem
| | |-- server-cert.pem -> ../../../integration/testdata/https/server-cert.pem
| | |-- server-key.pem -> ../../../integration/testdata/https/server-key.pem
| | |-- server-rogue-cert.pem
| | `-- server-rogue-key.pem
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 48fd0e921c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When we see an `ECONNREFUSED` (or equivalent) from an attempted `HEAD` on the
`/_ping` endpoint there is no point in trying again with `GET` since the server
is not responding/available at all.
Once vendored into the cli this will partially mitigate https://github.com/docker/cli/issues/1739
("Docker commands take 1 minute to timeout if context endpoint is unreachable")
by cutting the effective timeout in half.
Signed-off-by: Ian Campbell <ijc@docker.com>
(cherry picked from commit 8c8457b0f2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-05-13 15:17:51 -07:00
10277 changed files with 725259 additions and 1448855 deletions
# removes docker service that is currently installed on the runner. we
# could use Uninstall-Package but not yet available on Windows runners.
# more info: https://github.com/actions/virtual-environments/blob/d3a5bad25f3b4326c5666bab0011ac7f1beec95e/images/win/scripts/Installers/Install-Docker.ps1#L11
name:Removing current daemon
run:|
if (Get-Service docker -ErrorAction SilentlyContinue) {
$dockerVersion = (docker version -f "{{.Server.Version}}")
run:echo "::error::PR title suggests targetting the ${{ steps.title_branch.outputs.branch }} branch, but is opened against ${{ github.event.pull_request.base.ref }}" && exit 1
DOCKER_GITCOMMIT:=$(shell git rev-parse --short HEAD ||echo unsupported)
exportDOCKER_GITCOMMIT
# allow overriding the repository and branch that validation scripts are running
@ -16,9 +20,6 @@ export VALIDATE_REPO
exportVALIDATE_BRANCH
exportVALIDATE_ORIGIN_BRANCH
exportPAGER
exportGIT_PAGER
# env vars passed through directly to Docker's build scripts
# to allow things like `make KEEPBUNDLE=1 binary` easily
# `project/PACKAGERS.md` have some limited documentation of some of these
@ -27,9 +28,11 @@ export GIT_PAGER
# option of "go build". For example, a built-in graphdriver priority list
# can be changed during build time like this:
#
# make DOCKER_LDFLAGS="-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,zfs" dynbinary
# make DOCKER_LDFLAGS="-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper" dynbinary
#
DOCKER_ENVS:=\
-e DOCKER_CROSSPLATFORMS \
-e BUILD_APT_MIRROR \
-e BUILDFLAGS \
-e KEEPBUNDLE \
-e DOCKER_BUILD_ARGS \
@ -39,10 +42,6 @@ DOCKER_ENVS := \
-e DOCKER_BUILDKIT \
-e DOCKER_BASH_COMPLETION_PATH \
-e DOCKER_CLI_PATH \
-e DOCKERCLI_VERSION \
-e DOCKERCLI_REPOSITORY \
-e DOCKERCLI_INTEGRATION_VERSION \
-e DOCKERCLI_INTEGRATION_REPOSITORY \
-e DOCKER_DEBUG \
-e DOCKER_EXPERIMENTAL \
-e DOCKER_GITCOMMIT \
@ -50,21 +49,13 @@ DOCKER_ENVS := \
-e DOCKER_LDFLAGS \
-e DOCKER_PORT \
-e DOCKER_REMAP_ROOT \
-e DOCKER_ROOTLESS \
-e DOCKER_STORAGE_OPTS \
-e DOCKER_TEST_HOST \
-e DOCKER_USERLANDPROXY \
-e DOCKERD_ARGS \
-e DELVE_PORT \
-e GITHUB_ACTIONS \
-e TEST_FORCE_VALIDATE \
-e TEST_INTEGRATION_DIR \
-e TEST_INTEGRATION_USE_SNAPSHOTTER \
-e TEST_INTEGRATION_FAIL_FAST \
-e TEST_SKIP_INTEGRATION \
-e TEST_SKIP_INTEGRATION_CLI \
-e TEST_IGNORE_CGROUP_CHECK \
-e TESTCOVERAGE \
-e TESTDEBUG \
-e TESTDIRS \
-e TESTFLAGS \
@ -75,16 +66,16 @@ DOCKER_ENVS := \
-e VALIDATE_REPO \
-e VALIDATE_BRANCH \
-e VALIDATE_ORIGIN_BRANCH \
-e HTTP_PROXY \
-e HTTPS_PROXY \
-e NO_PROXY \
-e http_proxy \
-e https_proxy \
-e no_proxy \
-e VERSION \
-e PLATFORM \
-e DEFAULT_PRODUCT_LICENSE \
-e PRODUCT \
-e PACKAGER_NAME \
-e PAGER \
-e GIT_PAGER \
-e OTEL_EXPORTER_OTLP_ENDPOINT \
-e OTEL_EXPORTER_OTLP_PROTOCOL \
-e OTEL_SERVICE_NAME
-e PRODUCT
# note: we _cannot_ add "-e DOCKER_BUILDTAGS" here because even if it's unset in the shell, that would shadow the "ENV DOCKER_BUILDTAGS" set in our Dockerfile, which is very important for our official builds
# to allow `make BIND_DIR=. shell` or `make BIND_DIR= test`
# Note that `BIND_DIR` will already be set to `bundles` if `DOCKER_HOST` is not set (see above BIND_DIR line), in such case this will do nothing since `DOCKER_MOUNT` will already be set.
$(warning The docker client CLI has moved to github.com/docker/cli. For a dev-test cycle involving the CLI, run:${\n}DOCKER_CLI_PATH=/host/path/to/cli/binary make shell ${\n}then change the cli and compile into a binary at the same location.${\n})
@ -14,7 +14,7 @@ Moby is an open project guided by strong principles, aiming to be modular, flexi
It is open to the community to help set its direction.
- Modular: the project includes lots of components that have well-defined functions and APIs that work together.
- Batteries included but swappable: Moby includes enough components to build fully featured container systems, but its modular architecture ensures that most of the components can be swapped by different implementations.
- Batteries included but swappable: Moby includes enough components to build fully featured container system, but its modular architecture ensures that most of the components can be swapped by different implementations.
- Usable security: Moby provides secure defaults without compromising usability.
- Developer focused: The APIs are intended to be functional and useful to build powerful tools.
They are not necessarily intended as end user tools but as components aimed at developers.
The Moby maintainers take security seriously. If you discover a security issue, please bring it to their attention right away!
### Reporting a Vulnerability
Please **DO NOT** file a public issue, instead send your report privately to security@docker.com.
Security reports are greatly appreciated and we will publicly thank you for it, although we keep your name confidential if you request it. We also like to send gifts—if you're into schwag, make sure to let us know. We currently do not offer a paid security bounty program, but are not ruling it out in the future.
@ -37,6 +37,6 @@ There is hopefully enough example material in the file for you to copy a similar
When you make edits to `swagger.yaml`, you may want to check the generated API documentation to ensure it renders correctly.
Run `make swagger-docs` and a preview will be running at `http://localhost:9000`. Some of the styling may be incorrect, but you'll be able to ensure that it is generating the correct documentation.
Run `make swagger-docs` and a preview will be running at `http://localhost`. Some of the styling may be incorrect, but you'll be able to ensure that it is generating the correct documentation.
The production documentation is generated by vendoring `swagger.yaml` into [docker/docker.github.io](https://github.com/docker/docker.github.io).
expectedErr:fmt.Sprintf("invalid API version: the minimum API version (%s) is higher than the default version (%s)",api.DefaultVersion,api.MinSupportedAPIVersion),
},
{
doc:"invalid default too low",
defaultVersion:"0.1",
minVersion:api.MinSupportedAPIVersion,
expectedErr:fmt.Sprintf("invalid default API version (0.1): must be between %s and %s",api.MinSupportedAPIVersion,api.DefaultVersion),
},
{
doc:"invalid default too high",
defaultVersion:"9999.9999",
minVersion:api.DefaultVersion,
expectedErr:fmt.Sprintf("invalid default API version (9999.9999): must be between %s and %s",api.MinSupportedAPIVersion,api.DefaultVersion),
},
{
doc:"invalid minimum too low",
defaultVersion:api.MinSupportedAPIVersion,
minVersion:"0.1",
expectedErr:fmt.Sprintf("invalid minimum API version (0.1): must be between %s and %s",api.MinSupportedAPIVersion,api.DefaultVersion),
},
{
doc:"invalid minimum too high",
defaultVersion:api.DefaultVersion,
minVersion:"9999.9999",
expectedErr:fmt.Sprintf("invalid minimum API version (9999.9999): must be between %s and %s",api.MinSupportedAPIVersion,api.DefaultVersion),
errString:fmt.Sprintf("client version 0.1 is too old. Minimum supported API version is %s, please upgrade your client to a newer version",api.MinSupportedAPIVersion),
errString:"client version 0.1 is too old. Minimum supported API version is 1.2.0, please upgrade your client to a newer version",
},
{
reqVersion:"9999.9999",
errString:fmt.Sprintf("client version 9999.9999 is too new. Maximum supported API version is %s",api.DefaultVersion),
errString:"client version 9999.9999 is too new. Maximum supported API version is 1.10.0",
// There is existing endpoint config - if it's not indexed by NetworkMode.Name(), we
// can't tell which network the container-wide settings was intended for. NetworkMode,
// the keys in EndpointsConfig and the NetworkID in EndpointsConfig may mix network
// name/id/short-id. It's not safe to create EndpointsConfig under the NetworkMode
// name to store the container-wide MAC address, because that may result in two sets
// of EndpointsConfig for the same network and one set will be discarded later. So,
// reject the request ...
ep,ok:=networkingConfig.EndpointsConfig[nwName]
if!ok{
return"",errdefs.InvalidParameter(errors.New("if a container-wide MAC address is supplied, HostConfig.NetworkMode must match the identity of a network in NetworkSettings.Networks"))
}
// ep is the endpoint that needs the container-wide MAC address; migrate the address
// to it, or bail out if there's a mismatch.
ifep.MacAddress==""{
ep.MacAddress=deprecatedMacAddress
}elseifep.MacAddress!=deprecatedMacAddress{
return"",errdefs.InvalidParameter(errors.New("the container-wide MAC address must match the endpoint-specific MAC address for the main network, or be left empty"))
}
}
}
warning="The container-wide MacAddress field is now deprecated. It should be specified in EndpointsConfig instead."
config.MacAddress=""//nolint:staticcheck // ignore SA1019: field is deprecated, but still used on API < v1.44.
stream:=grpc.StreamInterceptor(grpc_middleware.ChainStreamServer(otelgrpc.StreamServerInterceptor(),grpcerrors.StreamServerInterceptor))//nolint:staticcheck // TODO(thaJeztah): ignore SA1019 for deprecated options: see https://github.com/moby/moby/issues/47437
withTrace:=otelgrpc.UnaryServerInterceptor()//nolint:staticcheck // TODO(thaJeztah): ignore SA1019 for deprecated options: see https://github.com/moby/moby/issues/47437
returnerrdefs.Unavailable(fmt.Errorf("this node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again"))
}elseif!c.manager{
returnerrdefs.Unavailable(fmt.Errorf("this node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager"))
