Commit graph

1233 commits

Author SHA1 Message Date
Sebastiaan van Stijn
f266f13965
Merge pull request #41636 from TBBle/37352-test-and-fix
Set 127GB default sandbox size for WCOW, and ensure storage-opts is honoured on all paths under WCOW and LCOW
2021-01-25 14:34:34 +01:00
Akihiro Suda
ec416c51b7
TestCgroupNamespacesRunOlderClient: support cgroup v2
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-01-22 16:17:47 +09:00
Sebastiaan van Stijn
d5612a0ef8
Merge pull request #41854 from cpuguy83/for-linux-1169-plugins-custom-runtime-panic
Add shim config for custom runtimes for plugins
2021-01-21 16:26:36 +01:00
Brian Goff
2903863a1d Add shim config for custom runtimes for plugins
This fixes a panic when an admin specifies a custom default runtime,
when a plugin is started the shim config is nil.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2021-01-14 19:28:28 +00:00
Sebastiaan van Stijn
41e5d459f1
Merge pull request #41865 from thaJeztah/load_frozen_images
integration: fix TestBuildUserNamespaceValidateCapabilitiesAreV2 not using frozen image
2021-01-13 18:50:22 +01:00
Akihiro Suda
327daef5c1
Merge pull request #41724 from EricMountain/dishonoured-capabilities
Use v2 capabilities in layer archives
2021-01-13 15:54:29 +09:00
Sebastiaan van Stijn
46dfc31342
integration: fix TestBuildUserNamespaceValidateCapabilitiesAreV2 not using frozen image
Commit f2f5106c92 added this test to verify loading
of images that were built with user-namespaces enabled.

However, because this test spins up a new daemon, not the daemon that's set up by
the test-suite's `TestMain()` (which loads the frozen images).

As a result, the `debian:bullseye` image was pulled from Docker Hub when running
the test;

    Calling POST /v1.41/images/load?quiet=1
    Applying tar in /go/src/github.com/docker/docker/bundles/test-integration/TestBuildUserNamespaceValidateCapabilitiesAreV2/d4d366b15997b/root/165536.165536/overlay2/3f7f9375197667acaf7bc810b34689c21f8fed9c52c6765c032497092ca023d6/diff" storage-driver=overlay
    Applied tar sha256:845f0e5159140e9dbcad00c0326c2a506fbe375aa1c229c43f082867d283149c to 3f7f9375197667acaf7bc810b34689c21f8fed9c52c6765c032497092ca023d6, size: 5922359
    Calling POST /v1.41/build?buildargs=null&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&t=capabilities%3A1.0&target=&ulimits=null&version=
    Trying to pull debian from https://registry-1.docker.io v2
    Fetching manifest from remote" digest="sha256:f169dbadc9021fc0b08e371d50a772809286a167f62a8b6ae86e4745878d283d" error="<nil>" remote="docker.io/library/debian:bullseye
    Pulling ref from V2 registry: debian:bullseye
    ...

This patch updates `TestBuildUserNamespaceValidateCapabilitiesAreV2` to load the
frozen image. `StartWithBusybox` is also changed to `Start`, because the test
is not using the busybox image, so there's no need to load it.

In a followup, we should probably add some utilities to make this easier to set up
(and to allow passing the list frozen images that we want to load, without having
to "hard-code" the image name to load).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-01-08 13:25:21 +01:00
Brian Goff
e6591a9c7a Make test work with rootless mode
Using `d.Kill()` with rootless mode causes the restarted daemon to not
be able to start containerd (it times out).

Originally this was SIGKILLing the daemon because we were hoping to not
have to manipulate on disk state, but since we need to anyway we can
shut it down normally.

I also tested this to ensure the test fails correctly without the fix
that the test was added to check for.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2021-01-05 00:25:33 +00:00
Tibor Vass
ffc4dc9aec
Merge pull request #41817 from simonferquel/desktop-startup-hang
Fix a potential hang when starting after a non-clean shutdown
2020-12-23 23:22:00 -08:00
Brian Goff
4a175fd050 Cleanup container shutdown check and add test
Adds a test case for the case where dockerd gets stuck on startup due to
hanging `daemon.shutdownContainer`

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-12-23 16:59:03 +00:00
Eric Mountain
95eb490780
Use v2 capabilities in layer archives
When building images in a user-namespaced container, v3 capabilities are
stored including the root UID of the creator of the user-namespace.

This UID does not make sense outside the build environment however. If
the image is run in a non-user-namespaced runtime, or if a user-namespaced
runtime uses a different UID, the capabilities requested by the effective
bit will not be honoured by `execve(2)` due to this mismatch.

Instead, we convert v3 capabilities to v2, dropping the root UID on the
fly.

Signed-off-by: Eric Mountain <eric.mountain@datadoghq.com>
2020-12-23 14:16:41 +01:00
Eric Mountain
f2f5106c92
Add a test for #41723
Capabilities are serialised in VFS_CAP_REVISION_3 when an image is
built in a user-namespaced daemon, instead of VFS_CAP_REVISION_2.

This adds a test for this, though it's currently wired to fail if
the capabilities are serialised in VFS_CAP_REVISION_2 instead in this
situation, since this is unexpected.

Signed-off-by: Eric Mountain <eric.mountain@datadoghq.com>
2020-12-18 07:51:44 +01:00
Sebastiaan van Stijn
2680a0fe2c
Networking: add regression test for networks endpoint with trailing slash
This adds a regression test for https://github.com/moby/moby/issues/24595

    make DOCKER_GRAPHDRIVER=vfs TEST_FILTER='TestNetworkList' test-integration

    INFO: Testing against a local daemon
    === RUN   TestNetworkList
    === RUN   TestNetworkList//networks
    === PAUSE TestNetworkList//networks
    === RUN   TestNetworkList//networks/
    === PAUSE TestNetworkList//networks/
    === CONT  TestNetworkList//networks
    === CONT  TestNetworkList//networks/
    --- PASS: TestNetworkList (0.05s)
        --- PASS: TestNetworkList//networks/ (0.01s)
        --- PASS: TestNetworkList//networks (0.01s)
    PASS

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-30 12:34:24 +01:00
Paul "TBBle" Hampson
1571e9331b Use specific APIs for shared mount-point behaviour
Thanks to @cpuguy83 for pointing these APIs out in #41638.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-10 20:00:48 +11:00
Paul "TBBle" Hampson
695b151a18 Work around small disk on Windows-RS5 CI nodes
The free disk space on the Windows RS5 CI nodes appears to be just the
right size that the TestBuildWCOWSandboxSize test can generate 21GB of
layers, and then a 21GB sandbox inside a container, and then runs out of
space while committing the layer.

Helpfully, this failure is distinguishable in the logs from a failure
when the sandbox is too small, so we can do that.

TODO: Revert this if-and-when the Windows-RS5 CI nodes have more free
space.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-10 19:51:46 +11:00
Paul "TBBle" Hampson
142b2b785b Add TestBuildWCOWSandboxSize integration test
This test validates that `RUN` and `COPY` both target a read-write
sandbox on Windows that is configured according to the daemon's
`storage-opts` setting.

Sadly, this is a slow test, so we need to bump the timeout to 60 minutes
from the default of 10 minutes.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-10 19:51:46 +11:00
Paul "TBBle" Hampson
7ba05f2b2b Rewrite VolumesMountedAsShared/Slave as Integration tests
This moves the two tests from integration-CLI to integration.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-08 23:15:15 +11:00
Brian Goff
88c0271605 Don't set default platform on container create
This fixes a regression based on expectations of the runtime:

```
docker pull arm32v7/alpine
docker run arm32v7/alpine
```

Without this change, the `docker run` will fail due to platform
matching on non-arm32v7 systems, even though the image could run
(assuming the system is setup correctly).

This also emits a warning to make sure that the user is aware that a
platform that does not match the default platform of the system is being
run, for the cases like:

```
docker pull --platform armhf busybox
docker run busybox
```

Not typically an issue if the requests are done together like that, but
if the image was already there and someone did `docker run` without an
explicit `--platform`, they may very well be expecting to run a native
version of the image instead of the armhf one.

This warning does add some extra noise in the case of platform specific
images being run, such as `arm32v7/alpine`, but this can be supressed by
explicitly setting the platform.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-10-20 20:17:23 +00:00
Tibor Vass
29b149ebb1
Merge pull request #41430 from TBBle/40444-update-gowinio-for-8gB-file-fix
Revendor Microsoft/go-winio for 8gB file fix when importing or committing image layers
2020-09-24 11:16:10 -07:00
Olli Janatuinen
66a37b460c Windows: Enable more integration tests
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2020-09-20 22:09:27 +03:00
Paul "TBBle" Hampson
2dd9faaa16 Enable TestBuildWithHugeFile on Windows
This reproduces #40444, based on a suggestion from GitHub user @marosset

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-09-19 23:13:44 +10:00
limeidan
2344b3a9e1 fix integration test case TestExternalGraphDriver failed in mips arch
Signed-off-by: limeidan <limeidan@loongson.cn>
2020-09-15 10:04:00 +08:00
Brian Goff
bf6f0d82bc
Merge pull request #41360 from lzhfromustc/GL_5
Builder: Add 1 buffer to two channels to avoid blocking goroutine
2020-09-12 21:43:05 -07:00
Sebastiaan van Stijn
5ca758199d
replace pkg/locker with github.com/moby/locker
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-09-10 22:15:40 +02:00
lzhfromustc
d1c2f953c5 Test: Add buffer to two channels to avoid blocking goroutine
Signed-off-by: Ziheng Liu <lzhfromustc@gmail.com>
2020-08-27 18:53:43 -04:00
Brian Goff
24f173a003 Replace service "Capabilities" w/ add/drop API
After dicussing with maintainers, it was decided putting the burden of
providing the full cap list on the client is not a good design.
Instead we decided to follow along with the container API and use cap
add/drop.

This brings in the changes already merged into swarmkit.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-07-27 10:09:42 -07:00
Sebastiaan van Stijn
06832a702d
Merge pull request #41065 from AkihiroSuda/cgroup2-ci
test-integration: support cgroup2
2020-07-16 16:55:47 +02:00
wanghuaiqing
228d74842f Upgrading the versions of images in Dockerfile.
In order to run tests at mips64el device.
Now official-images has supported the following images for mips64el.
buildpack-deps:stretch
buildpack-deps:buster
debian:stretch
debian:buster

But official-images does not support the following images for mips64el.
debian:jessie
buildpack-deps:jessie

Signed-off-by: wanghuaiqing <wanghuaiqing@loongson.cn>
2020-06-30 12:24:06 +08:00
Sebastiaan van Stijn
3258d565cf
Fix status code for missing --volumes-from container
If the container specified in `--volumes-from` did not exist, the
API returned a 404 status, which was interpreted by the CLI as the
specified _image_ to be missing (even if that was not the case).

This patch changes these error to return a 400 (bad request);

Before this change:

    # make sure the image is present
    docker pull busybox
    docker create --volumes-from=nosuchcontainer busybox
    # Unable to find image 'busybox:latest' locally
    # latest: Pulling from library/busybox
    # Digest: sha256:95cf004f559831017cdf4628aaf1bb30133677be8702a8c5f2994629f637a209
    # Status: Image is up to date for busybox:latest
    # Error response from daemon: No such container: nosuchcontainer

After this change:

    # make sure the image is present
    docker pull busybox
    docker create --volumes-from=nosuchcontainer busybox
    # Error response from daemon: No such container: nosuchcontainer

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-06-29 13:28:14 +02:00
Akihiro Suda
ed89041433
test-integration: support cgroup2
Usage: DOCKER_BUILD_ARGS="--build-arg CONTAINERD_COMMIT=master --build-arg RUNC_COMMIT=master" DOCKER_EXPERIMENTAL=1 TEST_SKIP_INTEGRATION_CLI=1 make test-integration

Depends on containerd master (v1.4) and runc master (v1.0.0-rc91).

Currently `TEST_SKIP_INTEGRATION_CLI=1` must be specified.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-06-22 14:17:45 +09:00
Sebastiaan van Stijn
87b8947580
TestDaemonDNSFallback: increase timeout to reduce flakiness
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-06-18 10:10:30 +02:00
Brian Goff
7fa2026620
Merge pull request #40938 from thaJeztah/move_pidslimit
API: swarm: move PidsLimit to TaskTemplate.Resources
2020-06-11 12:04:44 -07:00
Sebastiaan van Stijn
687bdc7c71
API: swarm: move PidsLimit to TaskTemplate.Resources
The initial implementation followed the Swarm API, where
PidsLimit is located in ContainerSpec. This is not the
desired place for this property, so moving the field to
TaskTemplate.Resources in our API.

A similar change should be made in the SwarmKit API (likely
keeping the old field for backward compatibility, because
it was merged some releases back)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-06-05 12:50:38 +02:00
Kir Kolyshkin
9b77370213 int/build/TestBuildWithEmptyLayers: prevent panic
The test case panics like that:

> build_test.go:381: assertion failed: 3 (int) != 1 (int)
> panic: runtime error: index out of range [2] with length 1 [recovered]
> panic: runtime error: index out of range [2] with length 1

The fix is trivial.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2020-06-04 11:22:04 -07:00
Tibor Vass
9a78e916ad integration: Add TestDaemonDNSFallback
Signed-off-by: Tibor Vass <tibor@docker.com>
2020-05-28 04:30:51 +00:00
Tibor Vass
5c10ea6ae8
Merge pull request #40725 from cpuguy83/check_img_platform
Accept platform spec on container create
2020-05-21 11:33:27 -07:00
Cristina Yenyxe Gonzalez Garcia
0154dc7a23 Using test names instead of hardcoded ones in integration/image directory
Signed-off-by: Cristina Yenyxe Gonzalez Garcia <cristina.yenyxe@gmail.com>
2020-05-18 16:37:09 +01:00
Cristina Yenyxe Gonzalez Garcia
4cabad6bae Using test names instead of hardcoded ones in integration/container directory
Signed-off-by: Cristina Yenyxe Gonzalez Garcia <cristina.yenyxe@gmail.com>
2020-05-18 16:37:04 +01:00
Cristina Yenyxe Gonzalez Garcia
6bc112abde Fixed small documentation typo
Signed-off-by: Cristina Yenyxe Gonzalez Garcia <cristina.yenyxe@gmail.com>
2020-05-18 16:36:53 +01:00
Cristina Yenyxe Gonzalez Garcia
ebd025b63a Using test names instead of hardcoded ones in integration/build directory
Signed-off-by: Cristina Yenyxe Gonzalez Garcia <cristina.yenyxe@gmail.com>
2020-05-18 16:36:23 +01:00
Sebastiaan van Stijn
0fc914de5c
Merge pull request #40595 from sfzhu93/GL_2_25
[integration] add buffer to prevent goroutine leak
2020-05-08 16:00:52 +02:00
Akihiro Suda
33ee7941d4 support --privileged --cgroupns=private on cgroup v1
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-04-21 23:11:32 +09:00
Akihiro Suda
ba8129b28a
Merge pull request #39946 from cpuguy83/plugins_use_containerd_dist
Replace plugin distribution code w/ containerd
2020-04-21 10:03:16 +09:00
Sebastiaan van Stijn
c8e31dc2f2
Merge pull request #39882 from thaJeztah/swarm_pids_limit
Add API support for PidsLimit on services
2020-04-16 21:02:30 +02:00
Sebastiaan van Stijn
54d88a7cd3
Merge pull request #40478 from cpuguy83/dont-prime-the-stats
Add stats options to not prime the stats
2020-04-16 20:57:06 +02:00
Sebastiaan van Stijn
157c53c8e0
Add API support for PidsLimit on services
Support for PidsLimit was added to SwarmKit in docker/swarmkit/pull/2415,
but never exposed through the Docker remove API.

This patch exposes the feature in the repote API.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-15 22:37:42 +02:00
Brian Goff
eac618caaa
Merge pull request #39771 from thaJeztah/cleanup_TestTemplatedConfig
Integration: fix various issues in "config" and "secret" tests
2020-04-15 10:35:33 -07:00
Sebastiaan van Stijn
cdc39fa29c
integration/config: add check for removing non-existing config
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:38 +02:00
Sebastiaan van Stijn
1a7ffe4fe4
integration/secret: add check for empty list not producing an error
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:36 +02:00
Sebastiaan van Stijn
632cc7019a
integration/config: remove check that was already done in TestConfigInspect
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:34 +02:00
Sebastiaan van Stijn
26f2eddaa3
integration/config: combine TestConfigCreateWithLabels with TestConfigsCreateAndDelete
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:29 +02:00
Sebastiaan van Stijn
3c4b258f21
integration/secret: don't string-match errors
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:26 +02:00
Sebastiaan van Stijn
18cae89436
integration/config: don't string-match errors
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:24 +02:00
Sebastiaan van Stijn
348f412d85
integration: sync minor changes between config and secret tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:22 +02:00
Sebastiaan van Stijn
a45c89ecd1
integration/secret: fix TestSecretInspect not actually checking response
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:21 +02:00
Sebastiaan van Stijn
25424cf772
integration/config: move functions to be in line with "secret" tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:19 +02:00
Sebastiaan van Stijn
3c6f018f94
integration/secret: TestTemplatedSecret: simplify task code
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:17 +02:00
Sebastiaan van Stijn
de78663181
integration/secret: rename variables that collided with imported package name
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:15 +02:00
Sebastiaan van Stijn
dd67b4794e
integration/config: TestTemplatedConfig: simplify task code
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:13 +02:00
Sebastiaan van Stijn
ec78112edd
integration/config: rename variables that collided with imported package name
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-14 18:31:11 +02:00
Brian Goff
ced91bee4b On startup, actually shutdown the container.
When a container is left running after the daemon exits (e.g. the daemon
is SIGKILL'd or crashes), it should stop any running containers when the
daemon starts back up.

What actually happens is the daemon only sends the container's
configured stop signal and does not check if it has exited.
If the container does not actually exit then it is left running.

This fixes this unexpected behavior by calling the same function to shut
down the container that the daemon shutdown process does.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-04-13 14:20:12 -07:00
Brian Goff
d7ba1f85ef Use containerd dist libs for plugin pull/pull
This removes the use of the old distribution code in the plugin packages
and replaces it with containerd libraries for plugin pushes and pulls.

Additionally it uses a content store from containerd which seems like
it's compatible with the old "basicBlobStore" in the plugin package.
This is being used locally isntead of through the containerd client for
now.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-04-02 15:03:06 -07:00
Sebastiaan van Stijn
af0415257e
Merge pull request #40694 from kolyshkin/moby-sys-mount-part-II
switch to moby/sys/{mount,mountinfo} part II
2020-04-02 21:52:21 +02:00
Brian Goff
7a9cb29fb9 Accept platform spec on container create
This enables image lookup when creating a container to fail when the
reference exists but it is for the wrong platform. This prevents trying
to run an image for the wrong platform, as can be the case with, for
example binfmt_misc+qemu.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-03-20 16:10:36 -07:00
Kir Kolyshkin
39048cf656 Really switch to moby/sys/mount*
Switch to moby/sys/mount and mountinfo. Keep the pkg/mount for potential
outside users.

This commit was generated by the following bash script:

```
set -e -u -o pipefail

for file in $(git grep -l 'docker/docker/pkg/mount"' | grep -v ^pkg/mount); do
	sed -i -e 's#/docker/docker/pkg/mount"#/moby/sys/mount"#' \
		-e 's#mount\.\(GetMounts\|Mounted\|Info\|[A-Za-z]*Filter\)#mountinfo.\1#g' \
		$file
	goimports -w $file
done
```

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2020-03-20 09:46:25 -07:00
Akihiro Suda
5e1b246b9a test-integration: support more rootless tests
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-03-16 17:37:38 +09:00
Akihiro Suda
5bf33adba2 hack: support $DOCKER_ROOTLESS for testing rootless
```
$ DOCKER_EXPERIMENTAL=1 DOCKER_ROOTLESS=1 TEST_SKIP_INTEGRATION_CLI=1 \
 make test-integration
```

test-integration-cli is unsupported currently.
Also, tests that spawn custom daemon (testutil/daemon) are skipped.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-03-10 05:49:45 +09:00
Arko Dasgupta
2e0762ae44
Add TC for com.docker.network.host_ipv4 label
This PR adds a testcase for the com.docker.network.host_ipv4
label commited via https://github.com/docker/libnetwork/pull/2454

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2020-03-09 18:51:19 +01:00
Sebastiaan van Stijn
e1710b42d3
vendor: bump libnetwork bf2bd42abc0a3734f12b5ec724e571434e42c669
full diff: 264bffcb88...bf2bd42abc

relevant changes:

- docker/libnetwork#2407 Macvlan internal network should not change default gateway
    - fixes docker/libnetwork#2406 Internal macvlan network overrides default gateway
- vendor godbus/dbus v5
- Fix InhibitIPv4 nil panic
- Cleanup VFP during overlay network removal
    - fixes VFP leak in windows overlay network deletion

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-03-07 00:35:47 +01:00
Akihiro Suda
3fbc41fe36 TestCgroupDriverSystemdMemoryLimit: fix failure
The test was failing:

  --- FAIL: TestCgroupDriverSystemdMemoryLimit (1.76s)
      cgroupdriver_systemd_test.go:54: assertion failed: 67108864 (s.HostConfig.Memory int64) != 67108864 (mem int)

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-03-03 11:30:55 +09:00
Brian Goff
ce1ceeb257 Add stats options to not prime the stats
Metrics collectors generally don't need the daemon to prime the stats
with something to compare since they already have something to compare
with.
Before this change, the API does 2 collection cycles (which takes
roughly 2s) in order to provide comparison for CPU usage over 1s. This
was primarily added so that `docker stats --no-stream` had something to
compare against.

Really the CLI should have just made a 2nd call and done the comparison
itself rather than forcing it on all API consumers.
That ship has long sailed, though.

With this change, clients can set an option to just pull a single stat,
which is *at least* a full second faster:

Old:
```
time curl --unix-socket
/go/src/github.com/docker/docker/bundles/test-integration-shell/docker.sock
http://./containers/test/stats?stream=false\&one-shot=false > /dev/null
2>&1

real0m1.864s
user0m0.005s
sys0m0.007s

time curl --unix-socket
/go/src/github.com/docker/docker/bundles/test-integration-shell/docker.sock
http://./containers/test/stats?stream=false\&one-shot=false > /dev/null
2>&1

real0m1.173s
user0m0.010s
sys0m0.006s
```

New:
```
time curl --unix-socket
/go/src/github.com/docker/docker/bundles/test-integration-shell/docker.sock
http://./containers/test/stats?stream=false\&one-shot=true > /dev/null
2>&1
real0m0.680s
user0m0.008s
sys0m0.004s

time curl --unix-socket
/go/src/github.com/docker/docker/bundles/test-integration-shell/docker.sock
http://./containers/test/stats?stream=false\&one-shot=true > /dev/null
2>&1

real0m0.156s
user0m0.007s
sys0m0.007s
```

This fixes issues with downstreams ability to use the stats API to
collect metrics.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-02-28 09:54:37 -08:00
Ziheng Liu
c322af8019 test: add buffer to prevent goroutine leak
Signed-off-by: Ziheng Liu <lzhfromustc@gmail.com>
2020-02-27 21:37:26 -05:00
Sebastiaan van Stijn
39679991f4
Merge pull request #40543 from SamWhited/upstream_logging
Upstream logging changes from Enterprise Edition
2020-02-27 13:54:14 +01:00
Akihiro Suda
26e6708ccf
Merge pull request #40475 from cpuguy83/fix_client_exec_type_id
Exec inspect field should be "ID" not "ExecID"
2020-02-26 19:11:33 +09:00
Sam Whited
a6f99232f5 Always execute tests and skip in tests themselves
See issue #40193

Signed-off-by: Sam Whited <sam@samwhited.com>
2020-02-21 14:31:01 -05:00
Brian Goff
750f0d1648 Support configuration of log cacher.
Configuration over the API per container is intentionally left out for
the time being, but is supported to configure the default from the
daemon config.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit cbecf48bc352e680a5390a7ca9cff53098cd16d7)
Signed-off-by: Madhu Venugopal <madhu@docker.com>
2020-02-19 17:02:34 -05:00
Brian Goff
e2ceb83a53 Support reads for all log drivers.
This supplements any log driver which does not support reads with a
custom read implementation that uses a local file cache.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit d675e2bf2b75865915c7a4552e00802feeb0847f)
Signed-off-by: Madhu Venugopal <madhu@docker.com>
2020-02-19 17:01:44 -05:00
Sebastiaan van Stijn
58c2615208
Merge pull request #40497 from arkodg/fix-bip-subnet-config
Set the bip network value as the subnet
2020-02-12 12:41:29 +01:00
Sebastiaan van Stijn
562880b276
Fix more goimports
```
daemon/logger/splunk/splunk_test.go:33: File is not `goimports`-ed (goimports)
        envKey:      "a",
        envRegexKey: "^foo",
        labelsKey:   "b",
        tagKey:      "c",
integration/build/build_test.go:41: File is not `goimports`-ed (goimports)
            rm:      false,
            forceRm: false,
integration/image/remove_unix_test.go:49: File is not `goimports`-ed (goimports)
        Root: d.Root,
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-11 18:56:25 +01:00
Arko Dasgupta
f800d5f786 Set the bip network value as the subnet
Dont assign the --bip value directly to the subnet
for the default bridge. Instead use the network value
from the ParseCIDR output

Addresses: https://github.com/moby/moby/issues/40392

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2020-02-10 17:38:54 -08:00
Sebastiaan van Stijn
9f0b3f5609
bump gotest.tools v3.0.1 for compatibility with Go 1.14
full diff: https://github.com/gotestyourself/gotest.tools/compare/v2.3.0...v3.0.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-11 00:06:42 +01:00
Brian Goff
cc993a9cbf Exec inspect field should be "ID" not "ExecID"
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-02-10 14:08:54 -08:00
Sebastiaan van Stijn
e55bff541e
TestLoginFailsWithBadCredentials: prepare for Go 1.14 changes
Go 1.14 adds quotes around the url in the error returned:

    === FAIL: arm64.integration.system TestLoginFailsWithBadCredentials (0.27s)
        TestLoginFailsWithBadCredentials: login_test.go:27: assertion failed: expected error "Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password", got "Error response from daemon: Get \"https://registry-1.docker.io/v2/\": unauthorized: incorrect username or password"
            Error response from daemon: Get "https://registry-1.docker.io/v2/": unauthorized: incorrect username or password

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-10 19:05:12 +01:00
Brian Goff
f464c31668 Check tmpfs mounts before create anon volume
This makes sure that things like `--tmpfs` mounts over an anonymous
volume don't create volumes uneccessarily.
One method only checks mountpoints, the other checks both mountpoints
and tmpfs... the usage of these should likely be consolidated.

Ideally, processing for `--tmpfs` mounts would get merged in with the
rest of the mount parsing. I opted not to do that for this change so the
fix is minimal and can potentially be backported with fewer changes of
breaking things.
Merging the mount processing for tmpfs can be handled in a followup.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-02-04 10:12:05 -08:00
Sebastiaan van Stijn
ca20bc4214
Merge pull request #40007 from arkodg/add-host-docker-internal
Support host.docker.internal in dockerd on Linux
2020-01-27 13:42:26 +01:00
Arko Dasgupta
92e809a680 Support host.docker.internal in dockerd on Linux
Docker Desktop (on MAC and Windows hosts) allows containers
running inside a Linux VM to connect to the host using
the host.docker.internal DNS name, which is implemented by
VPNkit (DNS proxy on the host)

This PR allows containers to connect to Linux hosts
by appending a special string "host-gateway" to --add-host
e.g. "--add-host=host.docker.internal:host-gateway" which adds
host.docker.internal DNS entry in /etc/hosts and maps it to host-gateway-ip

This PR also add a daemon flag call host-gateway-ip which defaults to
the default bridge IP
Docker Desktop will need to set this field to the Host Proxy IP
so DNS requests for host.docker.internal can be routed to VPNkit

Addresses: https://github.com/docker/for-linux/issues/264

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2020-01-22 13:30:00 -08:00
Drew Erny
30d9fe30b1 Add swarm jobs
Adds support for ReplicatedJob and GlobalJob service modes. These modes
allow running service which execute tasks that exit upon success,
instead of daemon-type tasks.

Signed-off-by: Drew Erny <drew.erny@docker.com>
2020-01-13 13:21:12 -06:00
Sebastiaan van Stijn
d64156970f
Merge pull request #40309 from dperny/bump-swarmkit
Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2
2020-01-09 20:16:33 +01:00
Drew Erny
07efe6a0a7 Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2
Bumps the vendoring of github.com/docker/swarmkit to the above commit,
which is the current master at commit time.

Most notably, this includes a change making the ingress network respect
the default address pool. Because of this change, a change to network
integration tests was needed.

Signed-off-by: Drew Erny <drew.erny@docker.com>
2020-01-07 09:43:22 -06:00
Akihiro Suda
19baeaca26 cgroup2: enable cgroup namespace by default
For cgroup v1, we were unable to change the default because of
compatibility issue.

For cgroup v2, we should change the default right now because switching
to cgroup v2 is already breaking change.

See also containers/libpod#4363 containers/libpod#4374

Privileged containers also use cgroupns=private by default.
https://github.com/containers/libpod/pull/4374#issuecomment-549776387

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-01-01 02:58:40 +09:00
Brian Goff
5f83f15ecb
Merge pull request #40193 from vikramhh/volumes_plugins_test_failure
Volume plugin tests [integration/plugin/volumes] should always be executed - checks for OS Type should happen within the test
2019-12-19 11:29:46 -08:00
Brian Goff
f9c8717d87
Merge pull request #40268 from olljanat/garbage-collector-remove-layerdb-values
Make sure that layers are removed from layerdb after succefull layer …
2019-12-18 09:49:22 -08:00
Akihiro Suda
0a4d980d71
Merge pull request #40142 from jmartin84/unique-names-intergration-service-inspect-test
refactored integration/service/instead_test.go to ues unique resource…
2019-12-13 18:25:08 +09:00
Brian Goff
b95fad8e51
Merge pull request #40263 from thaJeztah/normalize_comments
Normalize comment formatting
2019-12-12 12:06:22 -08:00
Jintao Zhang
56c5668068 Testcase: Change time comparison logic.
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
2019-12-10 20:10:28 +08:00
Olli Janatuinen
446fe7f263 Make sure that layers are removed from layerdb after succefull layer removal
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2019-11-28 20:06:09 +02:00
Sebastiaan van Stijn
92ad849327
integration: normalize comment formatting
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-11-27 15:37:59 +01:00
vikrambirsingh
83783af08e Disable tests missed by PR 40155
The tests starts a new daemon, but attempts to run it with overlay2,
and using a unix:// socket, which doesn't really work on Windows.

40155 tried to disable such tests but missed two of them.
They are being disabled with this change.

Signed-off-by: vikrambirsingh <vikrambir.singh@docker.com>
2019-11-11 18:54:21 +00:00
Justen Martin
36f6cc11a4
refactored integration/service/instead_test.go to ues unique resource names
Signed-off-by: Justen Martin <jmart@the-coder.com>
2019-11-08 16:39:20 -06:00
vikrambirsingh
25c5a55261 Check for OS Type and skip within the test
Premature check for OS type means that the test
will never even get to run on other OS types. This
will cause it to be always flagged as a failure on
such OS types.

Signed-off-by: vikrambirsingh <vikrambir.singh@docker.com>
2019-11-07 22:42:20 +00:00
Kirill Kolyshkin
154cf042fd
Merge pull request #40144 from lzhfromustc/GL_outputDone
integration/internal/container: fix a goroutine leak bug
2019-11-07 12:02:47 -08:00
Sebastiaan van Stijn
a0a5ec4c6d
Integration: skip TestInfoDebug on Windows
The test starts a new daemon, but attempts to run it with overlay2,
and using a unix:// socket, which doesn't really work on Windows.

```
00:14:14.623  --- FAIL: TestInfoDebug (0.01s)
00:14:14.623      info_test.go:75: [dbe75bf7729f3] failed to start daemon with arguments [--containerd /var/run/docker/containerd/containerd.sock --data-root D:\gopath\src\github.com\docker\docker\bundles\tmp\TestInfoDebug\dbe75bf7729f3\root --exec-root C:\windows\TEMP\dxr\dbe75bf7729f3 --pidfile D:\gopath\src\github.com\docker\docker\bundles\tmp\TestInfoDebug\dbe75bf7729f3\docker.pid --userland-proxy=true --containerd-namespace dbe75bf7729f3 --containerd-plugins-namespace dbe75bf7729f3p --host unix://C:\windows\TEMP\docker-integration\dbe75bf7729f3.sock --storage-driver overlay2 --debug] : protocol not available
00:14:14.623  === RUN   TestInfoInsecureRegistries
00:14:14.623  --- FAIL: TestInfoInsecureRegistries (0.00s)
00:14:14.623      info_test.go💯 [d3c745c16a39c] failed to start daemon with arguments [--containerd /var/run/docker/containerd/containerd.sock --data-root D:\gopath\src\github.com\docker\docker\bundles\tmp\TestInfoInsecureRegistries\d3c745c16a39c\root --exec-root C:\windows\TEMP\dxr\d3c745c16a39c --pidfile D:\gopath\src\github.com\docker\docker\bundles\tmp\TestInfoInsecureRegistries\d3c745c16a39c\docker.pid --userland-proxy=true --containerd-namespace d3c745c16a39c --containerd-plugins-namespace d3c745c16a39cp --host unix://C:\windows\TEMP\docker-integration\d3c745c16a39c.sock --debug --storage-driver overlay2 --insecure-registry=192.168.1.0/24 --insecure-registry=insecurehost.com:5000] : protocol not available
00:14:14.623  === RUN   TestInfoRegistryMirrors
00:14:14.623  --- FAIL: TestInfoRegistryMirrors (0.01s)
00:14:14.623      info_test.go:124: [d277126ad0515] failed to start daemon with arguments [--containerd /var/run/docker/containerd/containerd.sock --data-root D:\gopath\src\github.com\docker\docker\bundles\tmp\TestInfoRegistryMirrors\d277126ad0515\root --exec-root C:\windows\TEMP\dxr\d277126ad0515 --pidfile D:\gopath\src\github.com\docker\docker\bundles\tmp\TestInfoRegistryMirrors\d277126ad0515\docker.pid --userland-proxy=true --containerd-namespace d277126ad0515 --containerd-plugins-namespace d277126ad0515p --host unix://C:\windows\TEMP\docker-integration\d277126ad0515.sock --debug --storage-driver overlay2 --registry-mirror=https://192.168.1.2 --registry-mirror=http://registry.mirror.com:5000] : protocol not available
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-30 17:42:52 +01:00
Ziheng Liu
6233217a31 integration/internal/container: fix a goroutine leak bug by adding 1 buffer
Signed-off-by: Ziheng Liu <lzhfromustc@gmail.com>
2019-10-27 20:22:52 -04:00
Sebastiaan van Stijn
ec4a34ae2f
integration-cli: migrate some "info" tests to integration
- Updated TestInfoSecurityOptions to not rely on CLI output. Note that this
  test should be migrated to the integration suite, but that suite does not yet
  have checks for "Seccomp" and "AppArmor"
- TestInfoAPIWarnings: don't start with busybox because we're not running containers in this test
- Migrate TestInfoDebug to integration suite
- Migrate TestInsecureRegistries to integration suite (renamed to TestInfoInsecureRegistries)
- Migrate TestRegistryMirrors to integration suite (renamed to TestInfoRegistryMirrors)
- Migrate TestInfoDiscoveryBackend to integration suite
- Migrate TestInfoDiscoveryInvalidAdvertise to integration suite
- Migrate TestInfoDiscoveryAdvertiseInterfaceName to integration suite
- Remove TestInfoFormat, which is testing the CLI functionality, and there is an
  existing test in docker/cli (TestFormatInfo) covering this

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-22 00:11:52 +02:00
Justin Cormack
f681590a25
Merge pull request #40081 from thaJeztah/http_constants
Use http constants for HTTP methods and status codes
2019-10-17 11:30:26 -07:00
Drew Erny
f36042d259 Add support for sending down service Running and Desired task counts
Adds a new ServiceStatus field to the Service object, which includes the
running and desired task counts. This new field is gated behind a
"status" query parameter.

Signed-off-by: Drew Erny <drew.erny@docker.com>
2019-10-14 10:43:00 -05:00
Sebastiaan van Stijn
9ed58987ce
integration: use constants for http methods
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-13 17:30:13 +02:00
Tibor Vass
6ca3ec88ae builder: remove legacy build's session handling
This feature was used by docker build --stream and it was kept experimental.

Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-10-02 20:29:15 +00:00
Sebastiaan van Stijn
f60d6ee4bc
testutil: update WithInitsignature to be a daemon.Option
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-30 17:38:26 +02:00
Sebastiaan van Stijn
554d9cec25
testutil: update WithExperimental signature to be a daemon.Option
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-30 17:38:24 +02:00
Sam Whited
ae0a878b86 testutil, integration: untangle image dependency
Signed-off-by: Sam Whited <sam@samwhited.com>
2019-09-24 14:50:59 -05:00
Sebastiaan van Stijn
41ee87c681
Merge pull request #39942 from SamWhited/daemon_ops_type
testutil/daemon: group options under type
2019-09-19 17:44:49 +02:00
Sam Whited
41adef29f5 testutil/daemon: group options under type
Signed-off-by: Sam Whited <sam@samwhited.com>
2019-09-18 09:14:50 -05:00
Sebastiaan van Stijn
5358c95a76
TestGraphdriverPluginV2: SA5001: should check returned error before deferring (staticcheck)
```
integration/plugin/graphdriver/external_test.go:427:2: SA5001: should check returned error before deferring responseReader.Close() (staticcheck)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:57:55 +02:00
Sebastiaan van Stijn
101ff26eb5
misspell: fix some spelling issues
```
internal/test/environment/environment.go:37:23: `useing` is a misspelling of `using`(misspell)
integration/container/wait_test.go:49:9: `waitres` is a misspelling of `waiters`(misspell)
integration/container/wait_test.go:95:9: `waitres` is a misspelling of `waiters`(misspell)
integration-cli/docker_api_containers_test.go:1042:7: `waitres` is a misspelling of `waiters`(misspell)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:57:48 +02:00
Sebastiaan van Stijn
d948306255
integration: remove unused constants and fields (unused)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:57:10 +02:00
Sebastiaan van Stijn
4b13f69882
integration/network: remove unuset setupTest()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:57:08 +02:00
Sebastiaan van Stijn
4a37dcab48
Fix and re-enable TestDockerNetworkMacvlan/MultiSubnet
The linter marked an issue because testMacvlanMultiSubnet was not used.
Re-enabling the test showed that there was a typo in the assert, causing the
test to fail:

```
--- FAIL: TestDockerNetworkMacvlan/MultiSubnet (4.74s)
    macvlan_test.go:243: assertion failed: 2001:db8:abc4::254 (c3.NetworkSettings.Networks["dualstackbridge"].IPv6Gateway string) != 2001:db8.abc4::254 (string)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:57:07 +02:00
Michael Crosby
9f08135e02
Merge pull request #39844 from thaJeztah/fix_test_bashisms
Fix some bashisms/non-standard comparisons in integration tests
2019-09-12 14:49:56 -04:00
Justen Martin
548623b758 Use unique names in integration/service/plugin_test.go
Signed-off-by: Justen Martin <jmart@the-coder.com>
2019-09-11 19:09:54 -05:00
Sebastiaan van Stijn
32f1c65162
TestBuildSquashParent: fix non-standard comparisson
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-12 00:53:59 +02:00
Sam Whited
b37c214e3c testutil: make testing packages public
This was done with something along the lines of:

```
mv internal/test testutil
pushd testutil/; grep -IRl "package test" | xargs -I '{}' sed -i -e 's|package test|package testutil|g' {}; popd
mv internal/testutil/*.go testutil/ && rm -rf internal/
grep -IRl "github.com\/docker\/docker\/internal\/test" | xargs -I '{}' sed -i -e 's|github.com/docker/docker/internal/test|github.com/docker/docker/test|g' {}
goimports .
```

I also modified the basic plugin path in testutil/fixtures/plugin.

Signed-off-by: Sam Whited <sam@samwhited.com>
2019-09-11 07:47:23 -05:00
Sebastiaan van Stijn
cd9e4ec240
Merge pull request #39852 from thaJeztah/detect_invalid_linked_container
Return "invalid parameter" when linking to non-existing container
2019-09-11 09:51:10 +02:00
Tibor Vass
15c25714e8
Merge pull request #39881 from thaJeztah/inspect_test_use_RunningTasksCount
integration: TestInspect(): use swarm.RunningTasksCount
2019-09-10 14:18:33 -07:00
Sebastiaan van Stijn
422067ba7b
Return "invalid parameter" when linking to non-existing container
Trying to link to a non-existing container is not valid, and should return an
"invalid parameter" (400) error. Returning a "not found" error in this situation
would make the client report the container's image could not be found.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-10 23:06:56 +02:00
Tibor Vass
000b563006
Merge pull request #39671 from arkodg/fix-flaky-addr-pool-init-test
Fix flaky TestServiceWithDefaultAddressPoolInit
2019-09-09 15:01:11 -07:00
Sebastiaan van Stijn
f874f8b6fd
integration: TestInspect(): use swarm.RunningTasksCount
Instead of using the locally crafted `serviceContainerCount()` utility

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-09 17:48:12 +02:00
Rob Gulewich
96f6c81ab7 Change TestCgroupNamespacesRunPrivileged requirement
This test requires cgroup namespaces to be enabled, not the other way
around.

Signed-off-by: Rob Gulewich <rgulewich@netflix.com>
2019-08-23 11:27:27 -07:00
Vikram bir Singh
b146b2382c Fixups for 39695 - edit comments, redundant lines
1. Modify comments added in 5858a99267
Windows Volume GUID path format is: \\?\Volume{<GUID Value>}\<path>
Rewrote the example given in comments to conform to the format..

2. Remove two redundant asserts[assert.NilError]. They are redundant
because the last statement will not change the value of err.

Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>
2019-08-15 21:13:46 +00:00
Arko Dasgupta
a65dee30fc Move defer method to the top right after New is called
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2019-08-15 14:09:47 -07:00
Arko Dasgupta
f3a3ea0d3c Fix flaky TestServiceWithDefaultAddressPoolInit
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not

This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm

2. Make sure client connections are closed

3. Make sure every service and network name is unique

4. Make sure services and networks are cleaned up

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2019-08-14 08:02:15 -07:00
Sebastiaan van Stijn
5858a99267
Builder: fix "COPY --from" to non-existing directory on Windows
This fixes a regression introduced in 6d87f19142,
causing `COPY --from` to fail if the target directory does not exist:

```
FROM mcr.microsoft.com/windows/servercore:ltsc2019 as s1
RUN echo "Hello World" > /hello

FROM mcr.microsoft.com/windows/servercore:ltsc2019
COPY --from=s1 /hello /hello/another/world
```

Would produce an error:

```
Step 4/4 : COPY --from=s1 /hello /hello/another/world
failed to copy files: mkdir \\?: The filename, directory name, or volume label syntax is incorrect.
```

The cause for this was that Go's `os.MkdirAll()` does not support/detect volume GUID paths
(`\\?\Volume{dae8d3ac-b9a1-11e9-88eb-e8554b2ba1db}\hello\another}`), and as a result
attempted to create the volume as a directory (`\\?`), causing it to fail.

This patch replaces `os.MkdirAll()` with our own `system.MkdirAll()` function, which
is capable of detecting GUID volumes.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-08 16:45:10 +02:00
Brian Goff
abece9b562 Improvements to the test runners
1. Use `go list` to get list of integration dirs to build. This means we
   do not need to have a valid `.go` in every subdirectory and also
   filters out other dirs like "bundles" which may have been created.
2. Add option to specify custom flags for integration and
   integration-cli. This is needed so both suites can be run AND set
   custom flags... since the cli suite does not support standard go
   flags.
3. Add options to skip an entire integration suite.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-07-31 16:37:17 -07:00
Michael Crosby
d6cbeee470 Add extra permission check in removal test
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-07-22 20:39:22 +00:00
Sebastiaan van Stijn
eeaa0b30d4
Skip TestHealthKillContainer on Windows
This test is failing on Windows currently:

```
11:59:47 --- FAIL: TestHealthKillContainer (8.12s)
11:59:47     health_test.go:57: assertion failed: error is not nil: Error response from daemon: Invalid signal: SIGUSR1
``

That test was added recently in https://github.com/moby/moby/pull/39454, but
rewritten in a commit in the same PR:
f8aef6a92f

In that rewrite, there were some changes:

- originally it was skipped on Windows, but the rewritten test doesn't have that skip:

    ```go
    testRequires(c, DaemonIsLinux) // busybox doesn't work on Windows
    ```

- the original test used `SIGINT`, but the new one uses `SIGUSR1`

Analysis:

- The Error bubbles up from: 8e610b2b55/pkg/signal/signal.go (L29-L44)
- Interestingly; `ContainerKill` should validate if a signal is valid for the given platform, but somehow we don't hit that part; f1b5612f20/daemon/kill.go (L40-L48)
- Windows only looks to support 2 signals currently 8e610b2b55/pkg/signal/signal_windows.go (L17-L26)
- Upstream Golang looks to define `SIGINT` as well; 77f9b2728e/src/runtime/defs_windows.go (L44)
- This looks like the current list of Signals upstream in Go; 3b58ed4ad3/windows/types_windows.go (L52-L67)

```go
const (
	// More invented values for signals
	SIGHUP  = Signal(0x1)
	SIGINT  = Signal(0x2)
	SIGQUIT = Signal(0x3)
	SIGILL  = Signal(0x4)
	SIGTRAP = Signal(0x5)
	SIGABRT = Signal(0x6)
	SIGBUS  = Signal(0x7)
	SIGFPE  = Signal(0x8)
	SIGKILL = Signal(0x9)
	SIGSEGV = Signal(0xb)
	SIGPIPE = Signal(0xd)
	SIGALRM = Signal(0xe)
	SIGTERM = Signal(0xf)
)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-19 16:09:50 +02:00
Sebastiaan van Stijn
becd29c665
integration: run build session tests on non-experimental
The session endpoint is no longer experimental since
01c9e7082e, so we don't
need to start an experimental daemon.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-17 23:56:04 +02:00
Sebastiaan van Stijn
9d1b4f5fc3
Add regression tests for invalid platform status codes
Before we handled containerd errors, using an invalid platform produced a 500 status:

```bash
curl -v \
  -X POST \
  --unix-socket /var/run/docker.sock \
  "http://localhost:2375/v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest" \
  -H "Content-Type: application/json"
```

```
* Connected to localhost (docker.sock) port 80 (#0)
> POST /v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest HTTP/1.1
> Host: localhost:2375
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.40
< Content-Length: 85
< Content-Type: application/json
< Date: Mon, 15 Jul 2019 15:25:44 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/19.03.0-rc2 (linux)
<
{"message":"\"foobar\": unknown operating system or architecture: invalid argument"}
```

That problem is now fixed, and the API correctly returns a 4xx status:

```bash
curl -v \
  -X POST \
  --unix-socket /var/run/docker.sock \
  "http://localhost:2375/v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest" \
  -H "Content-Type: application/json"
```

```
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> POST /v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest HTTP/1.1
> Host: localhost:2375
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 400 Bad Request
< Api-Version: 1.41
< Content-Type: application/json
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Mon, 15 Jul 2019 15:13:42 GMT
< Content-Length: 85
<
{"message":"\"foobar\": unknown operating system or architecture: invalid argument"}
* Curl_http_done: called premature == 0
```

This patch adds tests to validate the behaviour

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-15 20:37:00 +02:00
Sebastiaan van Stijn
6359da4afa
Merge pull request #39431 from thaJeztah/macvlan_redundant_checks
Integration: remove redundant kernel version check for MACVlan
2019-07-15 11:33:15 +02:00
Sebastiaan van Stijn
5ce4394486
Merge pull request #37836 from selansen/test_initpool
TestServiceWithDefaultAddressPoolInit
2019-07-15 11:25:01 +02:00
selansen
88578aa9e9
TestServiceWithDefaultAddressPoolInit
Looks like TestServiceWithDefaultAddressPoolInit is failing
randomly in CI. I am not able to reproduce the issue locally
but this has been reported few times.  So I tried to modify
code and see if I can fix the random failure.

Signed-off-by: selansen <elango.siva@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-14 20:02:46 +02:00
Brian Goff
f8aef6a92f
Move kill health test to integration
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-07-14 11:53:21 +02:00
Sebastiaan van Stijn
316e16618f
Integration: MACVlan add missing import comment and build-tag
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-13 13:35:21 +02:00
Sebastiaan van Stijn
691eb14256
Integration: remove redundant kernel version check for MACVlan
The daemon requires kernel 3.10 or up to start, so there's no need
to check if the daemon is kernel 3.8 or up.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-13 13:35:18 +02:00
Sebastiaan van Stijn
4060a7026c
Integration: exclude IPVlan test-suite on Windows
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-11 10:39:04 +02:00
Sebastiaan van Stijn
93b28677bf
Integration: IPVlan add missing import comment
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-11 10:09:52 +02:00
Sebastiaan van Stijn
dae9bac675
Integration: remove "experimental" option for IPVLAN test-daemons
IPVLAN no longer is experimental since 3ab093d567,
so there's no need to set this option.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-11 10:04:04 +02:00
Sebastiaan van Stijn
1e4bd2623a
Integration: remove unneeded platform check for IPVLAN tests
These tests require a local daemon, and are not built on Windows

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-11 10:04:00 +02:00
Michael Crosby
fb459f6671
Merge pull request #38441 from sirlatrom/swarm_plugin_env
Allow specifying environment variables when installing an engine plugin as a Swarm service
2019-07-08 15:26:55 -04:00
Michael Crosby
b94218560e Skip TestServiceRemoveKeepsIngressNetwork
Ref: #39426

This is a common flaky test that I have seen on multiple PRs.  It is not
consistent and should be skipped until it is fixed to be robust.  A
simple fix for the swarm tests is not easy as they all poll and have 1
billion timeouts in all the tests so a skip is valid here.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-07-02 16:33:25 -04:00
Kirill Kolyshkin
67be90dcec
Merge pull request #39358 from jim-docker/testForIpvlan
For ipvlan tests, check that the ipvlan module is there, rather than just testing the kernel version.
2019-06-24 15:33:19 -07:00
Tibor Vass
f923321aae Add test for keeping same daemon ID on upgrade
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-18 01:40:25 +00:00
Tibor Vass
f23a51a860 Revert "Remove Schema1 integration test suite"
This reverts commit 13b7d11be1.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-18 00:23:04 +00:00
Kir Kolyshkin
a77e147d32 For ipvlan tests check that the ipvlan module is enabled (instead of just ensuring the kernel version is greater than 4.2)
Co-Authored-By: Jim Ehrismann <jim-docker@users.noreply.github.com>
Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com>
Signed-off-by: Jim Ehrismann <jim.ehrismann@docker.com>
2019-06-14 10:49:25 -04:00
Tibor Vass
02f1eb89a4 add more tests
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-13 06:31:05 +00:00
Brian Goff
6db9f1c3d6 Add test for copying entire container rootfs
CID=$(docker create alpine)
docker cp $CID:/ out

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-12 22:17:18 +00:00
Tibor Vass
a281289515 integration: get tests to compile again
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-12 20:41:36 +00:00
Sebastiaan van Stijn
04ff4a2ba4
Merge pull request #39137 from arkodg/attach-to-existing-network-error
Handle the error case when a container reattaches to the same network
2019-06-12 19:58:04 +02:00
Brian Goff
02dceeaa7b
Merge pull request #39240 from olljanat/enable-new-integration-tests-for-win
Enable integrations API tests for Windows CI
2019-06-11 14:24:03 -07:00
Sebastiaan van Stijn
e511b3be89
Merge pull request #39336 from justincormack/entropy-cannot-be-saved
Entropy cannot be saved
2019-06-11 18:40:19 +02:00
Olli Janatuinen
2f22247cad Enable integrations API tests for Windows CI
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2019-06-10 21:32:36 +03:00
Yong Tang
b10cd5c0e6
Merge pull request #39332 from thaJeztah/fix_golint_again
Integration: change signatures to fix golint warnings
2019-06-07 06:46:58 -07:00
Sebastiaan van Stijn
66f8f2b87c
Merge pull request #39193 from olljanat/38488-layer-garbage-collector
Added garbage collector for image layers
2019-06-07 14:08:04 +02:00
Sebastiaan van Stijn
dac5710b68
integration: change testGraphDriver signature to fix linting
Line 441: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-07 13:05:51 +02:00
Sebastiaan van Stijn
123e29f44a
integration: change createAmbiguousNetworks signature to fix linting
Line 30: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-07 13:05:43 +02:00
Sebastiaan van Stijn
9f9b4290b9
integration: change container.Run signature to fix linting
Line 59: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-07 13:05:26 +02:00
Sebastiaan van Stijn
b4c46b0dac
integration: change container.Create signature to fix linting
```
Line 25: warning: context.Context should be the first parameter of a function (golint)
Line 44: warning: context.Context should be the first parameter of a function (golint)
Line 52: warning: context.Context should be the first parameter of a function (golint)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-07 13:04:44 +02:00
Sebastiaan van Stijn
caec45a37f
integration: change network.CreateNoError signature to fix linting
Line 30: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-07 13:03:24 +02:00
Justin Cormack
2df693e533
Entropy cannot be saved
Remove non cryptographic randomness.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2019-06-07 11:54:45 +01:00
Kirill Kolyshkin
1d5748d975
Merge pull request #39173 from olljanat/25885-capabilities-swarm
Add support for capabilities options in services
2019-06-06 15:03:46 -07:00
Olli Janatuinen
213681b66a First step to implement full garbage collector for image layers
Refactored exiting logic on way that layers are first marked to be under
removal so if actual removal fails they can be found from disk and
cleaned up.

Full garbage collector will be implemented as part of containerd
migration.

Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2019-06-05 22:02:42 +03:00
Tibor Vass
54043d2f6f
Merge pull request #39281 from dperny/vendor-swarmkit
Revert docker/swarmkit#2804
2019-06-04 10:28:42 -07:00
Drew Erny
c7d9599e3d Revert docker/swarmkit#2804
Reverts the change to swarmkit that made all updates set UpdateStatus to
Completed

Signed-off-by: Drew Erny <drew.erny@docker.com>
2019-05-29 12:54:39 -05:00
Olli Janatuinen
f787b235de Add support capabilities list on services
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2019-05-28 19:52:36 +03:00
Olli Janatuinen
a3fcd4b82a Add alias for hostname if hostname != container
name which happens if user manually specify hostname

Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2019-05-22 20:20:43 +03:00
Yong Tang
3042254a87
Merge pull request #38377 from rgulewich/38332-cgroup-ns
Start containers in their own cgroup namespaces
2019-05-11 20:18:31 -07:00
Rob Gulewich
072400fc4b Make cgroup namespaces configurable
This adds both a daemon-wide flag and a container creation property:
- Set the `CgroupnsMode: "host|private"` HostConfig property at
  container creation time to control what cgroup namespace the container
  is created in
- Set the `--default-cgroupns-mode=host|private` daemon flag to control
  what cgroup namespace containers are created in by default
- Set the default if the daemon flag is unset to "host", for backward
  compatibility
- Default to CgroupnsMode: "host" for client versions < 1.40

Signed-off-by: Rob Gulewich <rgulewich@netflix.com>
2019-05-07 10:22:16 -07:00
Rob Gulewich
256eb04d69 Start containers in their own cgroup namespaces
This is enabled for all containers that are not run with --privileged,
if the kernel supports it.

Fixes #38332

Signed-off-by: Rob Gulewich <rgulewich@netflix.com>
2019-05-07 10:22:16 -07:00
Arko Dasgupta
680d0ba4ab Remove a network during task SHUTDOWN instead of REMOVE to
make sure the LB sandbox is removed when a service is updated
with a --network-rm option

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2019-05-06 20:26:59 -07:00
Arko Dasgupta
871acb1c86 Gracefully take care of the error case when a container
retries to attach to a network, it is already connected to

Fixes - https://github.com/docker/for-linux/issues/632

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
2019-04-26 15:58:58 -07:00
Kir Kolyshkin
1ada1c8391 TestIpcModeOlderClient: skip if client < 1.40
This test case requires not just daemon >= 1.40, but also
client API >= 1.40. In case older client is used, we'll
get failure from the very first check:

> ipcmode_linux_test.go:313: assertion failed: shareable (string) != private (string)

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2019-04-25 19:19:44 -07:00
Sebastiaan van Stijn
83ac2b4c13
Skip TestImagesFilterMultiReference on API < v1.40
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-04-24 17:52:21 -07:00
Sebastiaan van Stijn
05bd9958f2
Skip TestUUIDGeneration on API < v1.40
Older versions did not use an UUID as ID

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-04-24 17:52:13 -07:00
Sebastiaan van Stijn
d080a866cc
Skip TestPingCacheHeaders on API < v1.40
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-04-24 17:52:10 -07:00
Sebastiaan van Stijn
0e7b46aafe
Skip TestBuildWithEmptyDockerfile on API < v1.40
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-04-24 17:52:07 -07:00
Sebastiaan van Stijn
566eea13e6
Fix TestVolumesCreateAndList when running against a shared daemon
The daemon may already have other volumes, so filter out those
when running the test.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-04-24 17:52:03 -07:00
Sebastiaan van Stijn
c91c3776ea
Skip TestNISDomainname on API < 1.40
Older versions of the daemon would concatenate hostname and
domainname, so hostname "foobar" and domainname "baz.cyphar.com"
would produce `foobar.baz.cyphar.com` as hostname.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-04-24 17:51:57 -07:00
Sune Keller
fca5ee3bd5 Support environment vars in Swarm plugins services
Allow specifying environment variables when installing an engine plugin
as a Swarm service. Invalid environment variable entries (without an
equals (`=`) char) will be ignored.

Signed-off-by: Sune Keller <absukl@almbrand.dk>
2019-04-07 09:48:19 +02:00
Tõnis Tiigi
7a337ec3c4
Merge pull request #38990 from tiborvass/http-grpc
Add undocumented /grpc endpoint and register BuildKit's controller
2019-04-02 18:13:13 -07:00
Tibor Vass
e8382ece65 api: add undocumented /grpc endpoint to talk to GRPC services
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-04-02 19:57:59 +00:00
Sebastiaan van Stijn
23fec5025d
Temporarily disable TestCheckpoint
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-04-01 22:07:48 +02:00
Yong Tang
5635c248cd
Merge pull request #38871 from crosbymichael/exec-spec
Use original process spec for execs
2019-03-25 11:58:33 +08:00
Tibor Vass
29de017df7
Merge pull request #38599 from thaJeztah/builder_fix_copy_permissions
builder: fix `COPY --from` should preserve ownership
2019-03-22 09:38:13 -07:00
Kirill Kolyshkin
3764bd9e98
Merge pull request #38182 from huangqg/fix_log
integration/network: fix log message
2019-03-21 14:36:21 -07:00
Tõnis Tiigi
25661a3a04
Merge pull request #38793 from thaJeztah/pids_limit_improvements
Some refactoring on PidsLimit
2019-03-21 13:44:05 -07:00
Michael Crosby
7603c22c73 Use original process spec for execs
Fixes #38865

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-03-21 15:41:53 -04:00
Sebastiaan van Stijn
ae875d4069
Update more tests to use new errors
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-03-16 00:41:02 +01:00
Sebastiaan van Stijn
ca0b64ee3b
Merge pull request #35621 from kolyshkin/ipc-private
daemon: use 'private' ipc mode by default
2019-03-14 19:27:30 +01:00
Tibor Vass
5b6771345a
Merge pull request #38875 from thaJeztah/revert_38518-docker_info_builder_version
Revert "Adding builder version"
2019-03-14 10:55:44 -07:00
Sebastiaan van Stijn
f73dd5fdad
Revert "Adding builder version"
This reverts commit f821f002e5.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-03-14 00:18:46 +01:00
Justin Cormack
c435551ccc
Switch to google/uuid
pborman/uuid and google/uuid used to be different versions of
the same package, but now pborman/uuid is a compatibility wrapper
around google/uuid, maintained by the same person.

Clean up some of the usage as the functions differ slightly.

Not yet removed some uses of pborman/uuid in vendored code but
I have PRs in process for these.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2019-03-13 14:13:58 +00:00