Commit graph

2443 commits

Author SHA1 Message Date
Tibor Vass
e554fb23c8 Jenkinsfile: reduce time of integration tests by dividing tests into 3 parallel runs
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-08-12 20:41:03 +00:00
Tibor Vass
13df617d4c hack: unmount leftover daemon root folders
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-08-12 20:41:03 +00:00
Sebastiaan van Stijn
6f5c377ddc
docker-py: skip PullImageTest::test_pull_invalid_platform
and remove `PullImageTest::test_build_invalid_platform` from the list,
which was a copy/paste error in f8cde0b32d

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-10 13:18:13 +02:00
Sebastiaan van Stijn
a43123cab1
Consistently use DOCKER_EXPERIMENTAL=1 instead or =y
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-09 20:24:00 +02:00
Sebastiaan van Stijn
f8cde0b32d
docker-py: deselect broken experimental tests
These tests are fixed upstream, but those fixes are not yet in a
released version.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-09 18:55:08 +02:00
Sebastiaan van Stijn
6ae46aeabf
make.ps1: Run-IntegrationTests(): set working directory for test suite
This function changed to the correct working directory before starting the tests
(which is the same as on Linux), however the `ProcessStartInfo` process does
not inherit this working directory, which caused Windows tests to be running
with a different working directory as Linux (causing files used in tests to not
be found).

From the documentation; https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.workingdirectory?view=netframework-4.8

> When `UseShellExecute` is `true`, the fully qualified name of the directory that contains
> the process to be started. When the `UseShellExecute` property is `false`, the working
> directory for the process to be started. The default is an empty string (`""`).

This patch sets the `ProcessStartInfo.WorkingDirectory` to the correct working
directory before starting the process.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-08 13:20:04 +02:00
Brian Goff
ae0c0cdffd
Merge pull request #39684 from cpuguy83/test_detector_fix
Improve integration test detector
2019-08-07 13:17:37 -07:00
Kirill Kolyshkin
7a057e7fd2
Merge pull request #39669 from thaJeztah/fix_yamllint_locale
Set locale to fix yamlllint
2019-08-07 12:40:41 -07:00
Brian Goff
e2b24490e4 Improve integration test detecetor
The "new test" detector in test-integration-flaky was a bit flaky since
it would detect function signatures that are not new tests.

In addition, the test calls `return` outside of a function which is not
allowed.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-08-07 08:43:43 -07:00
Eli Uriegas
e665263b10
daemon: Remove btrfs_noversion build flag
btrfs_noversion was added in d7c37b5a28
for distributions that did not have the `btrfs/version.h` header file.

Seeing how all of the distributions we currently support do have the
`btrfs/version.h` file we should probably just remove this build flag
altogether.

Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
2019-08-06 22:55:29 +00:00
Sebastiaan van Stijn
b5e5cac0f5
Set locale to fix yamlllint
Attempting to fix;

```
21:16:00 Traceback (most recent call last):
21:16:00 File "/usr/local/bin/yamllint", line 11, in <module>
21:16:00 sys.exit(run())
21:16:00 File "/usr/local/lib/python3.5/dist-packages/yamllint/cli.py", line 170, in run
21:16:00 problems = linter.run(f, conf, filepath)
21:16:00 File "/usr/local/lib/python3.5/dist-packages/yamllint/linter.py", line 233, in run
21:16:00 content = input.read()
21:16:00 File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
21:16:00 return codecs.ascii_decode(input, self.errors)[0]
21:16:00 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 123522: ordinal not in range(128)
21:16:00 Build step 'Execute shell' marked build as failure
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-06 01:35:35 +02:00
Sebastiaan van Stijn
7449ca395c
Merge pull request #39657 from cpuguy83/test_filter
Add support for setting a test filter
2019-08-03 23:34:01 +02:00
Brian Goff
13064b155e Add support for setting a test filter
This is basically taking some stuff that make a custom shell function
for.
This takes a test filter, builds the appropriate TESTFLAGS, and sets the
integration API test dirs that match the given filter to avoid building
all test dirs.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-08-02 15:56:55 -07:00
Sebastiaan van Stijn
2cffe9be3d
hack: fix mixed tabs/spaces for indentation
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-02 15:58:33 +02:00
Vincent Demeester
4e83c90ae8
Merge pull request #39638 from tiborvass/unit-junit
added entry for running unit tests with junit report
2019-08-01 23:03:16 -04:00
Brian Goff
4fb5e9e696
Merge pull request #39628 from cpuguy83/test_setup_improvements
Improvements to the test runners
2019-08-01 15:45:16 -07:00
Andrew Hsu
42f0a0db75 run unit tests and generate junit report
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2019-08-01 06:08:35 +00:00
Brian Goff
abece9b562 Improvements to the test runners
1. Use `go list` to get list of integration dirs to build. This means we
   do not need to have a valid `.go` in every subdirectory and also
   filters out other dirs like "bundles" which may have been created.
2. Add option to specify custom flags for integration and
   integration-cli. This is needed so both suites can be run AND set
   custom flags... since the cli suite does not support standard go
   flags.
3. Add options to skip an entire integration suite.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-07-31 16:37:17 -07:00
Sebastiaan van Stijn
6f234db9fe
bump libnetwork to 09cdcc8c0eab3946c2d70e8f6225b05baf1e90d1
full diff: 83d30db536...09cdcc8c0e

changes included:

- docker/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling
- docker/libnetwork#2411 Macvlan network handles netlabel.Internal wrong
  - fixes docker/libnetwork#2410 Macvlan network handles netlabel.Internal wrong
- docker/libnetwork#2414 Allow network with --config-from to be --internal
  - fixes docker/libnetwork#2413 Network with --config-from does not honor --internal
- docker/libnetwork#2351 Use fewer modprobes
  - relates to moby/moby#38930 Use fewer modprobes
- docker/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks
  - carry of docker/libnetwork#2295 phantom ip/mac vlan network after a powercycle
  - fixes docker/libnetwork#1743 Phantom docker network

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-30 10:37:48 -07:00
Michael Zhao
790da6c223 Set TIMEOUT according to os/arch.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
2019-07-23 10:44:50 +08:00
Andrew Hsu
aac6e62209
added hack/ci/master as entry point for master codeline checks
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2019-07-22 20:00:07 +02:00
Sebastiaan van Stijn
4fa57a8191
Windows: fix Golang version checks for GO_VERSION build-arg
This check was used to make sure we don't bump Go versions independently
(Linux/Windows). The Dockerfile switched to using a build-arg to allow
overriding the Go version, which rendered this check non-functional.

It also fails if Linux versions use a specific variant of the image;

08:41:31 ERROR: Failed 'ERROR: Mismatched GO versions between Dockerfile and Dockerfile.windows. Update your PR to ensure that both files are updated and in sync. ${GO_VERSION}-stretch ${GO_VERSION}' at 07/20/2019 08:41:31
08:41:31 At C:\gopath\src\github.com\docker\docker\hack\ci\windows.ps1:448 char:9
08:41:31 +         Throw "ERROR: Mismatched GO versions between Dockerfile and D ...
08:41:31 +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This patch fixes the check by looking for the value of `GO_VERSION` instead
of looking at the `FROM` line (which is harder to parse).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-20 12:54:50 +02:00
Sebastiaan van Stijn
c6281bc438 Dockerfile: use GO_VERSION build-arg for overriding Go version
This allows overriding the version of Go without making modifications in the
source code, which can be useful to test against multiple versions.

For example:

    make GO_VERSION=1.13beta1 shell

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-18 17:36:09 -07:00
Tibor Vass
764160031e
Merge pull request #39068 from thaJeztah/separate_docker_py
Bump docker-py to 4.0.2, and run tests from upstream repository
2019-07-18 11:54:16 -07:00
Tibor Vass
c4aadee761
Merge pull request #39493 from AkihiroSuda/rootlesskit-060
rootless: allow exposing dockerd TCP socket easily
2019-07-17 12:34:42 -07:00
Hongxu Jia
3c701e4db1 imporve hardcoded CC on cross compile
Since commit applied in moby [61a3285 Support cross-compile for arm]
it hardcoded var-CC to support cross-compile for arm

Correct it with "${parameter:-word}" format, it is helpful for user
define toolchains

(Use Default Values.  If parameter is unset or null, the expansion of
word is substituted.  Otherwise, the value of parameter is substituted.)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
2019-07-17 17:42:14 +08:00
Andrew Hsu
c222c5ac6f allow running of single integration test
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-07-16 21:09:25 +00:00
Sebastiaan van Stijn
6aafe0fd9e
WIP Move docker-py tests first again
See if networking works if we run it first

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 19:53:17 +02:00
Sebastiaan van Stijn
980f2813b4
docker-py: skip flaky tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 14:35:00 +02:00
Sebastiaan van Stijn
3c15cea650
docker-py: use host-network for nested build of docker-py
When building this image docker-in-docker, the DNS in the environment
may not be usable for the build-container, causing resolution to fail:

```
02:35:31 W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release.gpg  Temporary failure resolving 'deb.debian.org'
```

This patch detects if we're building from within a container, and if
so, skips creating a networking namespace for the build by using
`--network=host`.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 14:34:54 +02:00
Sebastiaan van Stijn
ba8f4c7994
docker-py: don't build --quiet is TESTDEBUG is set
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 14:34:52 +02:00
Sebastiaan van Stijn
7bfe48cc00
Bump docker-py to 4.0.2, and run tests from upstream repository
This removes all the installation steps for docker-py from the
Dockerfile, and instead builds the upstream Dockerfile, and runs
docker-py tests in a container.

To test;

```
make test-docker-py

...

Removing bundles/

---> Making bundle: dynbinary (in bundles/dynbinary)
Building: bundles/dynbinary-daemon/dockerd-dev
Created binary: bundles/dynbinary-daemon/dockerd-dev

---> Making bundle: test-docker-py (in bundles/test-docker-py)
---> Making bundle: .integration-daemon-start (in bundles/test-docker-py)
Using test binary docker
Starting dockerd
INFO: Waiting for daemon to start...
.
INFO: Building docker-sdk-python3:3.7.0...
sha256:686428ae28479e9b5c8fdad1cadc9b7a39b462e66bd13a7e35bd79c6a152a402
INFO: Starting docker-py tests...
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-4.1.0, py-1.8.0, pluggy-0.9.0
rootdir: /src, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
collected 359 items

tests/integration/api_build_test.py .......s....
....
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 14:34:46 +02:00
Sebastiaan van Stijn
bd5c5373f1
Remove Codecov
Codecov has shown to be flaky, and calculate the wrong diff, in
addition, it doesn't show coverage for integration tests, which
makes the coverage report not useful.

Removing it for now, while we look at alternatives.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-15 20:30:59 +02:00
Sebastiaan van Stijn
f75f34249b
Fix "Removing bundles/" not actually removing bundles
Before:

Running `ls -la bundles/` before, and after removing:

    ls -la bundles/
    total 16
    drwxr-xr-x  7 root root  224 Jul 12 12:25 .
    drwxr-xr-x  1 root root 4096 Jul 12 12:30 ..
    drwxr-xr-x  2 root root   64 Jul 12 10:00 dynbinary
    drwxr-xr-x  6 root root  192 Jul 12 12:25 dynbinary-daemon
    lrwxrwxrwx  1 root root    1 Jul 12 12:25 latest -> .
    drwxr-xr-x 92 root root 2944 Jul 12 12:29 test-integration

    Removing bundles/

    ls -la bundles/
    total 16
    drwxr-xr-x  7 root root  224 Jul 12 12:25 .
    drwxr-xr-x  1 root root 4096 Jul 12 12:30 ..
    drwxr-xr-x  2 root root   64 Jul 12 10:00 dynbinary
    drwxr-xr-x  6 root root  192 Jul 12 12:25 dynbinary-daemon
    lrwxrwxrwx  1 root root    1 Jul 12 12:25 latest -> .
    drwxr-xr-x 92 root root 2944 Jul 12 12:29 test-integration

After:

Running `ls -la bundles/` before, and after removing:

    ls -la bundles/
    total 16
    drwxr-xr-x  7 root root  224 Jul 12 12:25 .
    drwxr-xr-x  1 root root 4096 Jul 12 12:30 ..
    drwxr-xr-x  2 root root   64 Jul 12 10:00 dynbinary
    drwxr-xr-x  6 root root  192 Jul 12 12:25 dynbinary-daemon
    lrwxrwxrwx  1 root root    1 Jul 12 12:25 latest -> .
    drwxr-xr-x 92 root root 2944 Jul 12 12:29 test-integration

    Removing bundles/

    ls -la bundles/
    total 4
    drwxr-xr-x 2 root root   64 Jul 12 12:25 .
    drwxr-xr-x 1 root root 4096 Jul 12 12:30 ..

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-12 14:41:08 +02:00
Akihiro Suda
34f4729bc0 rootless: allow exposing dockerd TCP socket easily
eg.

  $ DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS="-p 0.0.0.0:2376:2376/tcp" \
   dockerd-rootless.sh --experimental \
   -H tcp://0.0.0.0:2376 \
   --tlsverify --tlscacert=ca.pem --tlscert=cert.pem --tlskey=key.pem

This commit bumps up RootlessKit from v0.4.1 to v0.6.0:
27a0c7a248...2fcff6ceae

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-07-11 11:09:29 +09:00
CarlosEDP
9eaab0425b Update modules to support riscv64
Signed-off-by: CarlosEDP <me@carlosedp.com>
2019-06-29 18:45:42 +00:00
Sebastiaan van Stijn
0a3c9b935c bump vndr to f5ab8fc5f, and revendor
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-28 18:13:17 +00:00
Sebastiaan van Stijn
344b093258
bump libnetwork. vishvananda/netlink 1.0, vishvananda/netns
full diffs:

- fc5a7d91d5...62a13ae87c
- b2de5d10e3...v1.0.0
- 604eaf189e...13995c7128ccc8e51e9a6bd2b551020a27180abd

notable changes in libnetwork:

- docker/libnetwork#2366 Bump vishvananda/netlink to 1.0.0
- docker/libnetwork#2339 controller: Check if IPTables is enabled for arrangeUserFilterRule
  - addresses docker/libnetwork#2158 dockerd when run with --iptables=false modifies iptables by adding DOCKER-USER
  - addresses moby/moby#35777 With iptables=false dockerd still creates DOCKER-USER chain and rules
  - addresses docker/for-linux#136 dockerd --iptables=false adds DOCKER-USER chain and modify FORWARD chain anyway
- docker/libnetwork#2394 Make DNS records and queries case-insensitive
  - addresses moby/moby#28689 Embedded DNS is case-sensitive
  - addresses moby/moby#21169 hostnames with new networking are case-sensitive

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-25 14:10:15 +02:00
Tibor Vass
a9026e0e3f Revert "validate: temporarily disable deprecate-integration-cli as part of a revert"
This reverts commit 3f1cdd5364.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-19 00:54:14 +00:00
Tibor Vass
3f1cdd5364 validate: temporarily disable deprecate-integration-cli as part of a revert
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-18 01:40:25 +00:00
Sebastiaan van Stijn
d5669ec1c6
Update containerd to v1.2.7
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-14 13:30:38 +02:00
Brian Goff
349d4dd0c9
Merge pull request #39294 from AkihiroSuda/remove-it-on-swarm
hack: remove integration-cli-on-swarm
2019-06-11 14:24:48 -07:00
Brian Goff
02dceeaa7b
Merge pull request #39240 from olljanat/enable-new-integration-tests-for-win
Enable integrations API tests for Windows CI
2019-06-11 14:24:03 -07:00
Olli Janatuinen
2f22247cad Enable integrations API tests for Windows CI
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2019-06-10 21:32:36 +03:00
Sebastiaan van Stijn
35069de3fd
Re-align proxy commit with libnetwork vendor
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-07 15:57:23 +02:00
Sebastiaan van Stijn
0105613eb2
Merge pull request #39259 from olljanat/fix-win-tmp
Windows CI - Corrected LOCALAPPDATA location
2019-06-03 11:12:42 +02:00
Akihiro Suda
e7fbe8e457 hack: remove integration-cli-on-swarm
integration-on-swarm had unnecessary complexity and was too hard to
maintain. Also, it didn't support the new non-CLI integration test suite.

I'm now doing some experiments out of the repo using Kubernetes:
https://github.com/AkihiroSuda/kube-moby-integration

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-06-01 04:45:26 +09:00
Olli Janatuinen
61815f6763 Windows CI - Corrected LOCALAPPDATA location
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2019-05-29 23:44:40 +03:00
Sebastiaan van Stijn
a66ddd8ab8
bump libnetwork 5ac07abef4eee176423fdc1b870d435258e2d381
full diff: 9ff9b57c34...5ac07abef4

brings in:

- docker/libnetwork#2376 Forcing a nil IP specified in PortBindings to IPv4zero (0.0.0.0)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-05-21 14:46:18 +02:00
Akihiro Suda
00c92a6719 bump up rootlesskit to v0.4.1
Now the child process is killed when the parent dies (rootless-containers/rootlesskit#66)

e92d5e7...27a0c7a

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-05-15 20:51:48 +09:00