Commit graph

3011 commits

Author SHA1 Message Date
Akihiro Suda
e20b7323fb rootless: harden slirp4netns with mount namespace and seccomp
When slirp4netns v0.4.0+ is used, now slirp4netns is hardened using
mount namespace ("sandbox") and seccomp to mitigate potential
vulnerabilities.

bump up rootlesskit: 2fcff6ceae...791ac8cb20

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-09-02 14:58:58 +09:00
Sebastiaan van Stijn
70d3677825
hack/make/binary-daemon: fix some linting issues
- Add quotes to prevent word splitting in `cp` statement (SC2046)
- Replace legacy back tics with `$()`
- Replace `which` with `command -v` (SC2230)
- Fix incorrect (`==`) comparison

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-31 12:12:25 +02:00
Stefan Scherer
52a53e2587
Fix docker inspect for dutimgVersion
Signed-off-by: Stefan Scherer <stefan.scherer@docker.com>
2019-08-30 12:44:20 +02:00
Sebastiaan van Stijn
156ad54fb7
windows.ps1: fix leaked NdisAdapters not being cleaned up on RS1
Windows RS1 has problems with leaking NdisAdapters during the integration
tests; the windows.ps1 script has a cleanup stesp to remove those
leaked adapters.

For internal testing at Microsoft on internal builds, this cleanup step
was skipped, and only ran on the CI machines in our Jenkins.

Due to the move to our new Jenkins, the names of Windows machines changed,
and because of that, the cleanup step was never executed, resulting in the
leaked adapters not being cleaned up:

```
20:32:23  WARNING: There are 608 NdisAdapters leaked under Psched\Parameters
20:32:23  WARNING: Not cleaning as not a production RS1 server
20:32:24  WARNING: There are 608 NdisAdapters leaked under WFPLWFS\Parameters
20:32:24  WARNING: Not cleaning as not a production RS1 server
```

```
22:01:31  WARNING: There are 1209 NdisAdapters leaked under Psched\Parameters
22:01:31  WARNING: Not cleaning as not a production RS1 server
22:01:31  WARNING: There are 1209 NdisAdapters leaked under WFPLWFS\Parameters
22:01:31  WARNING: Not cleaning as not a production RS1 server
```

This patch removes the check for non-production builds, and unconditionally
cleans up leaked adapters if they are found.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-29 23:47:23 +02:00
Sebastiaan van Stijn
31885181fc
integration: windows.ps1: turn defender error into a warning
Some integration tests are known to fail if Windows Defender is
enabled. On the machines that run our CI, defender is disabled
for that reason.

Contributors likely will have defender enabled, and because of
that are currently not able to run the integration tests.

This patch changes the ERROR into a WARNING, so that contributors
can still run (a limited set of) the integration tests, but get
informed that some may fail.

We should make this requirement more specific, and only skip
tests that are known to require defender to be disabled, but
while that's not yet in place, let's print a warning instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-28 16:08:53 +02:00
Tianon Gravi
7a8a9c186b
Merge pull request #39604 from seemethere/in_container_question_mark
hack: Remove inContainer check, it wasn't useful
2019-08-27 16:10:47 -07:00
Sebastiaan van Stijn
001b78bffe
Merge pull request #39790 from thaJeztah/jenkinsfile_linting_and_skipping
integration: fix some linting issues, and don't fail build on artifact uploads
2019-08-27 15:34:04 +02:00
Kirill Kolyshkin
5c0d487416
Merge pull request #39795 from thaJeztah/remove_bundles_latest_symlink
hack/make.sh remove "latest" symlink
2019-08-26 23:14:20 -07:00
Sebastiaan van Stijn
38ab9da003
Merge pull request #39747 from tiborvass/use-new-windows-labels
Jenkinsfile: Use new windows labels
2019-08-26 09:45:03 +02:00
Sebastiaan van Stijn
917b0dcd3d
hack/make: fix some linting issues reported by shellcheck
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-25 17:02:20 +02:00
Sebastiaan van Stijn
dde1fd78c7
hack/make.sh remove "latest" symlink
This symlink was added in d42753485b,
to allow finding the path to the latest built binary, because at the time,
those paths were prefixed with the version or commit (e.g. `bundles/1.5.0-dev`).

Commit bac2447964 removed the version-prefix in
paths, but kept the old symlink for backward compatiblity. However, many
things were moved since then (e.g. paths were renamed to `binary-daemon`,
and various other changes). With the symlink pointing to the symlink's parent
directory, following the symlink may result into an infinite recursion,
which can happen if scripts using wildcards / globbing to find files.

With this symlink no longer serving a real purpose, we can probably safely
remove this symlink now.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-25 16:26:14 +02:00
Jintao Zhang
1264a85303 Update containerd to v1.2.8
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
2019-08-23 11:13:10 +08:00
Stefan Scherer
ca3e230b77
Use new windows labels
Signed-off-by: Stefan Scherer <stefan.scherer@docker.com>
2019-08-21 13:12:52 +02:00
Sebastiaan van Stijn
5969bbee79
docker-py: output junit.xml for test-results
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-15 02:45:02 +02:00
Sebastiaan van Stijn
535e29da05
docker-py: use --mount for bind-mounting docker.sock
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-15 02:45:00 +02:00
Sebastiaan van Stijn
b04cbf1072
docker-py: run without tty to disable color output
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-15 02:44:57 +02:00
Sebastiaan van Stijn
0b3d201892
docker-py: fix linting issues reported by shellcheck
- SC2006: use $(...) notation instead of legacy backticked `...`
- SC2086: double quote to prevent globbing and word splitting

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-15 02:44:54 +02:00
Tibor Vass
e554fb23c8 Jenkinsfile: reduce time of integration tests by dividing tests into 3 parallel runs
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-08-12 20:41:03 +00:00
Tibor Vass
13df617d4c hack: unmount leftover daemon root folders
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-08-12 20:41:03 +00:00
Sebastiaan van Stijn
6f5c377ddc
docker-py: skip PullImageTest::test_pull_invalid_platform
and remove `PullImageTest::test_build_invalid_platform` from the list,
which was a copy/paste error in f8cde0b32d

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-10 13:18:13 +02:00
Sebastiaan van Stijn
a43123cab1
Consistently use DOCKER_EXPERIMENTAL=1 instead or =y
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-09 20:24:00 +02:00
Sebastiaan van Stijn
f8cde0b32d
docker-py: deselect broken experimental tests
These tests are fixed upstream, but those fixes are not yet in a
released version.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-09 18:55:08 +02:00
Sebastiaan van Stijn
6ae46aeabf
make.ps1: Run-IntegrationTests(): set working directory for test suite
This function changed to the correct working directory before starting the tests
(which is the same as on Linux), however the `ProcessStartInfo` process does
not inherit this working directory, which caused Windows tests to be running
with a different working directory as Linux (causing files used in tests to not
be found).

From the documentation; https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.workingdirectory?view=netframework-4.8

> When `UseShellExecute` is `true`, the fully qualified name of the directory that contains
> the process to be started. When the `UseShellExecute` property is `false`, the working
> directory for the process to be started. The default is an empty string (`""`).

This patch sets the `ProcessStartInfo.WorkingDirectory` to the correct working
directory before starting the process.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-08 13:20:04 +02:00
Brian Goff
ae0c0cdffd
Merge pull request #39684 from cpuguy83/test_detector_fix
Improve integration test detector
2019-08-07 13:17:37 -07:00
Kirill Kolyshkin
7a057e7fd2
Merge pull request #39669 from thaJeztah/fix_yamllint_locale
Set locale to fix yamlllint
2019-08-07 12:40:41 -07:00
Brian Goff
e2b24490e4 Improve integration test detecetor
The "new test" detector in test-integration-flaky was a bit flaky since
it would detect function signatures that are not new tests.

In addition, the test calls `return` outside of a function which is not
allowed.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-08-07 08:43:43 -07:00
Eli Uriegas
e665263b10
daemon: Remove btrfs_noversion build flag
btrfs_noversion was added in d7c37b5a28
for distributions that did not have the `btrfs/version.h` header file.

Seeing how all of the distributions we currently support do have the
`btrfs/version.h` file we should probably just remove this build flag
altogether.

Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
2019-08-06 22:55:29 +00:00
Sebastiaan van Stijn
b5e5cac0f5
Set locale to fix yamlllint
Attempting to fix;

```
21:16:00 Traceback (most recent call last):
21:16:00 File "/usr/local/bin/yamllint", line 11, in <module>
21:16:00 sys.exit(run())
21:16:00 File "/usr/local/lib/python3.5/dist-packages/yamllint/cli.py", line 170, in run
21:16:00 problems = linter.run(f, conf, filepath)
21:16:00 File "/usr/local/lib/python3.5/dist-packages/yamllint/linter.py", line 233, in run
21:16:00 content = input.read()
21:16:00 File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
21:16:00 return codecs.ascii_decode(input, self.errors)[0]
21:16:00 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 123522: ordinal not in range(128)
21:16:00 Build step 'Execute shell' marked build as failure
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-06 01:35:35 +02:00
Sebastiaan van Stijn
7449ca395c
Merge pull request #39657 from cpuguy83/test_filter
Add support for setting a test filter
2019-08-03 23:34:01 +02:00
Brian Goff
13064b155e Add support for setting a test filter
This is basically taking some stuff that make a custom shell function
for.
This takes a test filter, builds the appropriate TESTFLAGS, and sets the
integration API test dirs that match the given filter to avoid building
all test dirs.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-08-02 15:56:55 -07:00
Sebastiaan van Stijn
2cffe9be3d
hack: fix mixed tabs/spaces for indentation
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-02 15:58:33 +02:00
Vincent Demeester
4e83c90ae8
Merge pull request #39638 from tiborvass/unit-junit
added entry for running unit tests with junit report
2019-08-01 23:03:16 -04:00
Brian Goff
4fb5e9e696
Merge pull request #39628 from cpuguy83/test_setup_improvements
Improvements to the test runners
2019-08-01 15:45:16 -07:00
Andrew Hsu
42f0a0db75 run unit tests and generate junit report
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2019-08-01 06:08:35 +00:00
Brian Goff
abece9b562 Improvements to the test runners
1. Use `go list` to get list of integration dirs to build. This means we
   do not need to have a valid `.go` in every subdirectory and also
   filters out other dirs like "bundles" which may have been created.
2. Add option to specify custom flags for integration and
   integration-cli. This is needed so both suites can be run AND set
   custom flags... since the cli suite does not support standard go
   flags.
3. Add options to skip an entire integration suite.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-07-31 16:37:17 -07:00
Sebastiaan van Stijn
6f234db9fe
bump libnetwork to 09cdcc8c0eab3946c2d70e8f6225b05baf1e90d1
full diff: 83d30db536...09cdcc8c0e

changes included:

- docker/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling
- docker/libnetwork#2411 Macvlan network handles netlabel.Internal wrong
  - fixes docker/libnetwork#2410 Macvlan network handles netlabel.Internal wrong
- docker/libnetwork#2414 Allow network with --config-from to be --internal
  - fixes docker/libnetwork#2413 Network with --config-from does not honor --internal
- docker/libnetwork#2351 Use fewer modprobes
  - relates to moby/moby#38930 Use fewer modprobes
- docker/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks
  - carry of docker/libnetwork#2295 phantom ip/mac vlan network after a powercycle
  - fixes docker/libnetwork#1743 Phantom docker network

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-30 10:37:48 -07:00
Eli Uriegas
f5cd8fdd44
hack: Remove inContainer check, it wasn't useful
The inContainer check isn't really useful anymore.

Even though it was said that we shouldn't rely on its existence back in
2016, we're now in 2019 and this thing still exists so we should just
rely on it now to check whether or not we're in a container.

Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
2019-07-25 23:12:01 +00:00
Michael Zhao
790da6c223 Set TIMEOUT according to os/arch.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
2019-07-23 10:44:50 +08:00
Andrew Hsu
aac6e62209
added hack/ci/master as entry point for master codeline checks
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2019-07-22 20:00:07 +02:00
Sebastiaan van Stijn
4fa57a8191
Windows: fix Golang version checks for GO_VERSION build-arg
This check was used to make sure we don't bump Go versions independently
(Linux/Windows). The Dockerfile switched to using a build-arg to allow
overriding the Go version, which rendered this check non-functional.

It also fails if Linux versions use a specific variant of the image;

08:41:31 ERROR: Failed 'ERROR: Mismatched GO versions between Dockerfile and Dockerfile.windows. Update your PR to ensure that both files are updated and in sync. ${GO_VERSION}-stretch ${GO_VERSION}' at 07/20/2019 08:41:31
08:41:31 At C:\gopath\src\github.com\docker\docker\hack\ci\windows.ps1:448 char:9
08:41:31 +         Throw "ERROR: Mismatched GO versions between Dockerfile and D ...
08:41:31 +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This patch fixes the check by looking for the value of `GO_VERSION` instead
of looking at the `FROM` line (which is harder to parse).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-20 12:54:50 +02:00
Sebastiaan van Stijn
c6281bc438 Dockerfile: use GO_VERSION build-arg for overriding Go version
This allows overriding the version of Go without making modifications in the
source code, which can be useful to test against multiple versions.

For example:

    make GO_VERSION=1.13beta1 shell

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-18 17:36:09 -07:00
Tibor Vass
764160031e
Merge pull request #39068 from thaJeztah/separate_docker_py
Bump docker-py to 4.0.2, and run tests from upstream repository
2019-07-18 11:54:16 -07:00
Tibor Vass
c4aadee761
Merge pull request #39493 from AkihiroSuda/rootlesskit-060
rootless: allow exposing dockerd TCP socket easily
2019-07-17 12:34:42 -07:00
Hongxu Jia
3c701e4db1 imporve hardcoded CC on cross compile
Since commit applied in moby [61a3285 Support cross-compile for arm]
it hardcoded var-CC to support cross-compile for arm

Correct it with "${parameter:-word}" format, it is helpful for user
define toolchains

(Use Default Values.  If parameter is unset or null, the expansion of
word is substituted.  Otherwise, the value of parameter is substituted.)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
2019-07-17 17:42:14 +08:00
Andrew Hsu
c222c5ac6f allow running of single integration test
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-07-16 21:09:25 +00:00
Sebastiaan van Stijn
6aafe0fd9e
WIP Move docker-py tests first again
See if networking works if we run it first

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 19:53:17 +02:00
Sebastiaan van Stijn
980f2813b4
docker-py: skip flaky tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 14:35:00 +02:00
Sebastiaan van Stijn
3c15cea650
docker-py: use host-network for nested build of docker-py
When building this image docker-in-docker, the DNS in the environment
may not be usable for the build-container, causing resolution to fail:

```
02:35:31 W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release.gpg  Temporary failure resolving 'deb.debian.org'
```

This patch detects if we're building from within a container, and if
so, skips creating a networking namespace for the build by using
`--network=host`.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 14:34:54 +02:00
Sebastiaan van Stijn
ba8f4c7994
docker-py: don't build --quiet is TESTDEBUG is set
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 14:34:52 +02:00
Sebastiaan van Stijn
7bfe48cc00
Bump docker-py to 4.0.2, and run tests from upstream repository
This removes all the installation steps for docker-py from the
Dockerfile, and instead builds the upstream Dockerfile, and runs
docker-py tests in a container.

To test;

```
make test-docker-py

...

Removing bundles/

---> Making bundle: dynbinary (in bundles/dynbinary)
Building: bundles/dynbinary-daemon/dockerd-dev
Created binary: bundles/dynbinary-daemon/dockerd-dev

---> Making bundle: test-docker-py (in bundles/test-docker-py)
---> Making bundle: .integration-daemon-start (in bundles/test-docker-py)
Using test binary docker
Starting dockerd
INFO: Waiting for daemon to start...
.
INFO: Building docker-sdk-python3:3.7.0...
sha256:686428ae28479e9b5c8fdad1cadc9b7a39b462e66bd13a7e35bd79c6a152a402
INFO: Starting docker-py tests...
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-4.1.0, py-1.8.0, pluggy-0.9.0
rootdir: /src, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
collected 359 items

tests/integration/api_build_test.py .......s....
....
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-16 14:34:46 +02:00