Commit graph

39250 commits

Author SHA1 Message Date
Sebastiaan van Stijn
2b41e84719
pkg/system: fix compile on darwin (macOS)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-10 20:49:09 +01:00
Sebastiaan van Stijn
b4b902b88f
Jenkinsfile: remove e2e image stage
The image that's built is not pushed anywhere, and is just
building the same as the main image already builds, so didn't
add value.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-10 18:42:13 +01:00
Sebastiaan van Stijn
6d243cdf27
testutil: remove remaining use of deprecated pkg/mount
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-10 14:46:59 +01:00
Sebastiaan van Stijn
0e8023ddea
Merge pull request #41638 from TBBle/37352-fix-Windows-CI-pipeline
Reinstate Windows CI Pipeline usefulness
2020-11-10 02:52:54 +01:00
Brian Goff
470ae8422f
Merge pull request #41652 from thaJeztah/cgroupv2_detection
use containerd/cgroups to detect cgroups v2
2020-11-09 10:03:30 -08:00
Brian Goff
e11cabe558
Merge pull request #41554 from thaJeztah/bump_cobra
vendor: github.com/spf13/cobra v1.1.1
2020-11-09 09:37:31 -08:00
Akihiro Suda
985592f6b7
Merge pull request #41653 from kaydxh/master
[DEL] remove useless assert
2020-11-10 02:08:29 +09:00
kaydxh
43d61d2bcf [DEL] remove useless assert
Signed-off-by: xiaohua ding <xiao_hua_ding@sina.cn>
2020-11-09 23:38:45 +08:00
Sebastiaan van Stijn
6458f750e1
use containerd/cgroups to detect cgroups v2
libcontainer does not guarantee a stable API, and is not intended
for external consumers.

this patch replaces some uses of libcontainer/cgroups with
containerd/cgroups.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-09 15:00:32 +01:00
Paul "TBBle" Hampson
7ba05f2b2b Rewrite VolumesMountedAsShared/Slave as Integration tests
This moves the two tests from integration-CLI to integration.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-08 23:15:15 +11:00
Paul "TBBle" Hampson
7bb729e928 Break out non-Windows sys/mount usage into helper
Rather than bifurcate the test completely, this lets us keep the test
intact with a small function wrapper to allow the compiler to build the
code that'll never be called on Windows, on Windows.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-08 23:15:15 +11:00
Paul "TBBle" Hampson
3e3f3d7168 Move tests using sys/mount to not build on Windows
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-08 23:15:15 +11:00
Paul "TBBle" Hampson
a158b53d86 Separate non-Windows mount code from common code
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-08 23:15:15 +11:00
Paul "TBBle" Hampson
080d9df63e Don't clear the exit code from the Finally block
It turns out that the Finally block does not see the exit code from the
`exit` call that triggered it, but from an earlier state. And it seems
that actions take in the Finally block other than `exit` will not affect
the $LastErrorCode set by the `exit` that triggered the Finally block.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-08 23:15:15 +11:00
Akihiro Suda
c7109494fe
Merge pull request #41617 from thaJeztah/unify_consts
opts: unify host-options, and use consts
2020-11-08 19:31:07 +09:00
Brian Goff
b5ea9abf25
Merge pull request #41648 from thaJeztah/fix_stream_detection
builder: fix detection of experimental --stream option (deprecated)
2020-11-06 14:13:25 -08:00
Sebastiaan van Stijn
4ee8688c4a
Merge pull request #41640 from tianon/containerd-namespace-flags
Unhide containerd-namespace flags
2020-11-06 22:12:30 +01:00
Sebastiaan van Stijn
1bb3491601
builder: fix detection of experimental --stream option (deprecated)
Commit 6ca3ec88ae deprecated the experimental
"--stream" option for the legacy builder, adding an error message is a client
attempted to use this feature.

However, the detection used the session-ID (`session=xxx` query parameter),
which happens to be set automatically by the CLI if it detects that the daemon
has session support. Because of this, builds fail when trying to perform them
on a daemon with the `--experimental` flag set.

This patch changes the detection to look for the `remote` query parameter, which
is set to "client-session" when using the `--stream` option with the classic
(non-Buildkit) builder.

Before this change, running `docker build` with an older (19.03 or older) cli
against a daemon with `--experimental` enabled caused an error:

    $ dockerd --experimental &
    $ docker pull docker:18.09
    $ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker:18.09 sh -c 'echo "FROM scratch" | docker build -'

    Sending build context to Docker daemon  2.048kB
    Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead

    docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -w /foo docker:18.09 sh -c 'echo "FROM scratch" > Dockerfile && docker build --stream .'
    Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead

With this patch, the error only occurs when trying to use the experimental
`--stream` option:

    $ dockerd --experimental &
    $ docker pull docker:18.09
    $ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker:18.09 sh -c 'echo "FROM scratch" | docker build -'

    Step 1/1 : FROM scratch
     --->
    No image was generated. Is your Dockerfile empty?

    $ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -w /foo docker:18.09 sh -c 'echo "FROM scratch" > Dockerfile && docker build --stream .'

    Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-06 21:13:28 +01:00
Tianon Gravi
49e809fbfe
Merge pull request #41641 from cpuguy83/fix_checkpoint
Don't set image on containerd container.
2020-11-06 16:43:51 +00:00
Akihiro Suda
3b14605f41
Merge pull request #41605 from thaJeztah/login2_cleanup
registry.loginV2(): minor cleanup
2020-11-06 14:35:46 +09:00
Brian Goff
51f5b1279d Don't set image on containerd container.
We aren't using containerd's image store, so we shouldn't be setting
this value.

This fixes container checkpoints, where containerd attempts to
checkpoint the image since one is set, but the image does not exist in
containerd.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-11-06 04:55:03 +00:00
Tianon Gravi
d99e8d2c3b Unhide containerd-namespace flags
The daemon expects to manage/handle everything going into the containerd namespace it uses, and things break when it shares that namespace with a second instance, so we should expose these flags and expect users that want two daemons sharing a containerd instance to use separate namespaces for them.

Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2020-11-05 12:14:01 -08:00
Tianon Gravi
761066faf1
Merge pull request #41100 from cpuguy83/windows_fix_rename_open_file
Fix windows log file rotation with readers
2020-11-05 20:02:58 +00:00
Brian Goff
9ca3bb632e Store image manifests in containerd content store
This allows us to cache manifests and avoid extra round trips to the
registry for content we already know about.

dockerd currently does not support containerd on Windows, so this does
not store manifests on Windows, yet.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-11-05 20:02:18 +00:00
Tianon Gravi
12d4466d87
Merge pull request #41612 from sparrc/task-delete-add-container-id
Added container ID to containerd task delete event messages
2020-11-05 19:57:54 +00:00
Tianon Gravi
3e955cd531
Merge pull request #41634 from AkihiroSuda/rootlesskit-0.11.0
bump up rootlesskit to v0.11.0
2020-11-05 19:12:53 +00:00
Benjamin Böhmke
66459cc623 Added ip6tables config option
Signed-off-by: Benjamin Böhmke <benjamin@boehmke.net>
2020-11-05 16:18:23 +01:00
Sebastiaan van Stijn
c90671b4cd
Merge pull request #41604 from arkodg/bump-libnetwork-ipv6nat
Vendor in Libnetwork changes
2020-11-05 15:57:32 +01:00
Akihiro Suda
c6accc67f2
bump up rootlesskit to v0.11.0
Important fix: Lock state dir for preventing automatic clean-up by systemd-tmpfiles
(https://github.com/rootless-containers/rootlesskit/pull/188)

Full changes:https://github.com/rootless-containers/rootlesskit/compare/v0.10.0...v0.11.0

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-11-05 16:49:32 +09:00
Arko Dasgupta
dfc2d770e4 Update iptable.Exists API in integration-cli
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2020-11-03 15:03:41 -08:00
Tibor Vass
42855ebc7f
Merge pull request #41606 from thaJeztah/moby_sys_symlink
replace pkg/symlink with github.com/moby/sys/symlink
2020-11-03 09:58:08 -08:00
Sebastiaan van Stijn
53727ce2f0
vendor: github.com/creack/pty v1.1.11
full diff: https://github.com/creack/pty/compare/v1.1.9...v1.1.11

- v1.1.11: Add arm support for OpenBSD
- v1.1.10: Fix CTTY to work with go1.15

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-03 13:38:27 +01:00
Sebastiaan van Stijn
dc3c382b34
replace pkg/symlink with github.com/moby/sys/symlink
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-03 11:17:12 +01:00
Akihiro Suda
0b93c6e131
Merge pull request #41458 from thaJeztah/bump_mountinfo
vendor: moby/sys mountinfo/v0.4.0
2020-11-03 16:39:18 +09:00
Cuong Manh Le
c208f03fbd
pkg/devicemapper: fix invalid usage of reflect.SliceHeader
The current usage of reflect.SliceHeader violates rule 6th of
unsafe.Pointer conversion. In short, reflect.SliceHeader could not be
used as plain struct.

See https://golang.org/pkg/unsafe/#Pointer

Signed-off-by: Cuong Manh Le <cuong.manhle.vn@gmail.com>
2020-11-03 14:20:52 +07:00
Tibor Vass
7bb1944edb
Merge pull request #41573 from thaJeztah/bump_sys
vendor: golang.org/x/sys eeed37f84f13f52d35e095e8023ba65671ff86a1
2020-11-02 09:44:25 -08:00
Sebastiaan van Stijn
6a5393636e
opts: unify host-options, and use consts
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-31 16:44:21 +01:00
Cam
80a5df9c49
Added container ID to containerd task delete event messages
Signed-off-by: Cam <gh@sparr.email>
2020-10-30 20:58:57 -07:00
Sebastiaan van Stijn
c2cc352355
Merge pull request #41611 from thaJeztah/move_const
Move HostGatewayName const to opts, and change vars to consts
2020-10-31 00:29:32 +01:00
Sebastiaan van Stijn
053962af92
Move HostGatewayName const to opts, and change vars to consts
This prevents consumers of the opts package to also having to
depend on daemon/network, and everything related.

We can probably change some of the other constants to strings,
for easier concatenating, and need to review the windows-specific
"127.0.0.1" (instead of "localhost"), which may no longer be
needed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-30 21:17:34 +01:00
Brian Goff
e4cf1c7336 pull: use tag service for pulling tagged reference
The tag service does a `HEAD` request to get the manifest digest, where
we can then do a `GET` against the digest.

The `GET` by tag is not cacheable, but the `GET` against the digest is.
This allows proxies to work way better.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-10-30 01:57:11 +00:00
Sebastiaan van Stijn
e766361271
pkg/mount: update deprecated wrappers
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-29 23:04:20 +01:00
Sebastiaan van Stijn
32d506b394
vendor: moby/sys mountinfo/v0.4.0
full diff: https://github.com/moby/sys/compare/mountinfo/v0.1.3...mountinfo/v0.4.0

> Note that this dependency uses submodules, providing "github.com/moby/sys/mount"
> and "github.com/moby/sys/mountinfo". Our vendoring tool (vndr) currently doesn't
> support submodules, so we vendor the top-level moby/sys repository (which contains
> both) and pick the most recent tag, which could be either `mountinfo/vXXX` or
> `mount/vXXX`.

github.com/moby/sys/mountinfo v0.4.0
--------------------------------------------------------------------------------

Breaking changes:

- `PidMountInfo` is now deprecated and will be removed before v1.0; users should switch to `GetMountsFromReader`

Fixes and improvements:

- run filter after all fields are parsed
- correct handling errors from bufio.Scan
- documentation formatting fixes

github.com/moby/sys/mountinfo v0.3.1
--------------------------------------------------------------------------------

- mount: use MNT_* flags from golang.org/x/sys/unix on freebsd
- various godoc and CI fixes
- mountinfo: make GetMountinfoFromReader Linux-specific
- Add support for OpenBSD in addition to FreeBSD
- mountinfo: use idiomatic naming for fields

github.com/moby/sys/mountinfo v0.2.0
--------------------------------------------------------------------------------

Bug fixes:

- Fix path unescaping for paths with double quotes

Improvements:

- Mounted: speed up by adding fast paths using openat2 (Linux-only) and stat
- Mounted: relax path requirements (allow relative, non-cleaned paths, symlinks)
- Unescape fstype and source fields
- Documentation improvements

Testing/CI:

- Unit tests: exclude darwin
- CI: run tests under Fedora 32 to test openat2
- TestGetMounts: fix for Ubuntu build system
- Makefile: fix ignoring test failures
- CI: add cross build

github.com/moby/sys/mount v0.1.1
--------------------------------------------------------------------------------

https://github.com/moby/sys/releases/tag/mount%2Fv0.1.1

Improvements:

- RecursiveUnmount: add a fast path (#26)
- Unmount: improve doc
- fix CI linter warning on Windows

Testing/CI:

- Unit tests: exclude darwin
- Makefile: fix ignoring test failures
- CI: add cross build

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-29 23:02:28 +01:00
Sebastiaan van Stijn
182795cff6
Do not call mount.RecursiveUnmount() on Windows
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-29 23:00:16 +01:00
Arko Dasgupta
1623e6b222 Vendor in Libnetwork changes
Bring in changes from https://github.com/moby/libnetwork/pull/2572 to moby

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2020-10-29 14:32:04 -07:00
Sebastiaan van Stijn
ac84f1b55f
registry.loginV2(): minor cleanup
just some minor cleaning up; construct the endpoint-path once
instead of twice, and grouping all variable declarations at the
start.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-29 22:26:57 +01:00
=
b102d4637c Fix windows log file rotation with readers
This fixes the case where log rotation fails on Windows while there are
clients reading container logs.

Evicts readers if there is an error during rotation and try rotation again.
This is needed for Windows with this scenario:

1. `docker logs -f` is called
2. Log rotation occurs (log.txt -> log.txt.1, truncate and re-open
   log.txt)
3. Log rotation occurs again (rm log.txt.1, log.txt -> log.txt.1)

On step 3, before this change, the log rotation will fail with `Access
is denied`.
In this case, what we have is a reader holding a file handle to the
primary log file. The log file is then rotated, but the reader still has
a the handle open. `FILE_SHARE_DELETE` allows this to happen... but then
we try to do it again for the next rotation and it blows up.
So when it blows up we force all the readers to disconnect, close the
log file, and try rotation again, which will succeed based on the added
tests.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-10-29 18:38:30 +00:00
Tibor Vass
bb23f1bf61
Merge pull request #41599 from thaJeztah/cleanup_some_v1_stuff
Cleanup v1 registry code that is not used for searching
2020-10-28 15:00:07 -07:00
Brian Goff
bcc993b494 Fix logfile to open all files with custom openFile
This makes sure, on Windows, that all files are opened with
FILE_SHARE_DELETE.

On non-Windows this just calls the same `os.Open()`.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-10-28 20:36:32 +00:00
Brian Goff
56ba96b6c1 Clean up some logfile implementation
- Ignore some pointless errors (like not exist on remove)
- Consolidate error handling/logging
- Fix race condition reading last log timestamp in the compression
  goroutine. This needs to be done while holding the write lock, which
  is not (or may not be) locked while compressing a rotated log file.
- Remove some indentation and consolidate mutex unlocking in
  `compressFile`

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-10-28 20:36:32 +00:00