Commit graph

2571 commits

Author SHA1 Message Date
Sebastiaan van Stijn
25a1bf53d2
Fix containerutility compilation on gcc-mingw-w64 8.3
The makefile for this binary has version 6.3 hardcoded,
which causes compilation on 8.3 to fail:

```
Building: bundles/cross/windows/amd64/containerutility.exe
In file included from /usr/x86_64-w64-mingw32/include/minwindef.h:163,
                 from /usr/x86_64-w64-mingw32/include/windef.h:8,
                 from /usr/x86_64-w64-mingw32/include/windows.h:69,
                 from containerutility.h:3,
                 from argumentstream.cpp:1:
/usr/x86_64-w64-mingw32/include/winnt.h:1554:11: fatal error: x86intrin.h: No such file or directory
 # include <x86intrin.h>
           ^~~~~~~~~~~~~
compilation terminated.
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-01-09 18:52:09 +01:00
Sebastiaan van Stijn
5dbfae6949
hack/make: ignore failure to stop apparmor
```
 ---> Making bundle: .integration-daemon-stop (in bundles/test-integration)
 ++++ cat bundles/test-integration/docker.pid
 +++ kill 13137
 +++ /etc/init.d/apparmor stop
 Leaving: AppArmorNo profiles have been unloaded.

 Unloading profiles will leave already running processes permanently
 unconfined, which can lead to unexpected situations.

 To set a process to complain mode, use the command line tool
 'aa-complain'. To really tear down all profiles, run 'aa-teardown'."

script returned exit code 255
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-01-09 18:52:03 +01:00
Sebastiaan van Stijn
f2b25e498f
docker-py: skip broken ImageCollectionTest::test_pull_multiple
The ImageCollectionTest.test_pull_multiple test performs a `docker pull` without
a `:tag` specified) to pull all tags of the given repository (image).

After pulling the image, the image(s) pulled are checked to verify if the list
of images contains the `:latest` tag.

However, the test assumes that all tags of the image are tags for the same
version of the image (same digest), and thus a *single* image is returned, which
is not always the case.

Currently, the `hello-world:latest` and `hello-world:linux` tags point to a
different digest, therefore the `client.images.pull()` returns multiple images:
one image for digest, making the test fail:

    =================================== FAILURES ===================================
    ____________________ ImageCollectionTest.test_pull_multiple ____________________
    tests/integration/models_images_test.py:90: in test_pull_multiple
        assert len(images) == 1
    E   AssertionError: assert 2 == 1
    E    +  where 2 = len([<Image: 'hello-world:linux'>, <Image: 'hello-world:latest'>])

This patch temporarily skips the broken test until it is fixed upstream.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-01-03 15:47:39 +01:00
Sebastiaan van Stijn
6bc45b09e7
docker-py: re-enable tests that were fixed in v4.1.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-01-03 15:46:28 +01:00
Sebastiaan van Stijn
bbcabf69c8
Merge pull request #40050 from thaJeztah/ci_force_validate
Jenkinsfile: force validate on master, and some fixes/improvements
2019-12-23 13:39:43 +01:00
Vikram bir Singh
8d2e1ee546
Build and use gotestsum for running all tests
1. Dockerfile.Windows modified to build gotestsum.exe

2. Use gotestsum.exe in invoking the execution of:

   (a) Unit tests (run in containers),
   (b) Integration tests (run outside containers)
   (c) Integration-cli (run outside containers)

No changes made to other categories of tests (e.g.
LCOW).

3. Copy .xml files produced by gotestsum in
   appropriate paths where Jenkins can ingest them

4. Modify Jenkinsfile to mark results output as
   being jUnit "type" as well as to archive the
   .xml test result files as artifacts.

Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-12-19 20:31:22 +01:00
Sebastiaan van Stijn
d3c0261876
hack/validate/deprecate-integration-cli: add integration-cli/*_api_*.go
The intent of this check is to not add integration tests that use the
CLI to test API functionality, so check for changes in "API" tests as
well, now that the `integration/` suite has matured enough to develop
new tests, or migrate existing ones.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-12-19 18:07:12 +01:00
Sebastiaan van Stijn
1f0fdcc6f5
hack/validate: clean-up output, move golang-ci-lint to end
The golang-ci-lint takes longest to run, so do the other
checks first; this also makes the output a bit more readable.

While at it, fix some consistency issues in the output of the
other checks.

Before this change:

    TEST_FORCE_VALIDATE=1 hack/validate/default
    Congratulations!  All commits are properly signed with the DCO!
    Congratulations! Seccomp profile generation is done correctly.
    INFO [config_reader] Used config file hack/validate/golangci-lint.yml
    INFO [lintersdb] Active 15 linters: [deadcode gofmt goimports golint gosec gosimple govet ineffassign misspell staticcheck structcheck typecheck unconvert unused varcheck]
    ...
    INFO Execution took 2m12.4287302s
    Congratulations!  "./pkg/..." is safely isolated from internal code.
    The swagger spec at "api/swagger.yaml" is valid against swagger specification 2.0
    Congratulations! All api changes are done the right way.
    Congratulations!  All toml source files changed here have valid syntax.
    Congratulations!  Changelog CHANGELOG.md is well-formed.
    Congratulations!  Changelog CHANGELOG.md dates are in descending order.

After this change:

    TEST_FORCE_VALIDATE=1 hack/validate/default
    Congratulations!  All commits are properly signed with the DCO!
    Congratulations!  Seccomp profile generation is done correctly.
    Congratulations!  Packages in "./pkg/..." are safely isolated from internal code.
    Congratulations!  The swagger spec at "api/swagger.yaml" is valid against swagger specification 2.0
    Congratulations!  All API changes are done the right way.
    Congratulations!  All TOML source files changed here have valid syntax.
    Congratulations!  Changelog CHANGELOG.md is well-formed.
    Congratulations!  Changelog CHANGELOG.md dates are in descending order.
    Congratulations!  No new tests were added to integration-cli.

    INFO Start validation with golang-ci-lint
    INFO [config_reader] Used config file hack/validate/golangci-lint.yml
    INFO [lintersdb] Active 15 linters: [deadcode gofmt goimports golint gosec gosimple govet ineffassign misspell staticcheck structcheck typecheck unconvert unused varcheck]
    ...
    INFO Execution took 2m12.4287302s

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-12-19 18:07:07 +01:00
Sebastiaan van Stijn
545e81714b
hack/validate: allow forcing validation
This introduces a FORCE_VALIDATE environment variable, which allows
forcing some validation steps, even if no changes were detected.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-12-19 18:03:55 +01:00
Sebastiaan van Stijn
9dde8f2b46
hack/validate/deprecate-integration-cli: don't exit on success
The `hack/validate/deprecate-integration-cli` script exited on
success. As a result, validation steps to run afterwards would
not be executed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-12-19 17:59:23 +01:00
Sebastiaan van Stijn
e21019fa51
hack/validate/default: fix minor linting issue
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-12-19 17:58:03 +01:00
Sebastiaan van Stijn
c8ecedbde9
Revert "hack: temporarily disable deprecate-integration-cli"
This reverts commit 5f67568bfd.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-12-19 17:57:56 +01:00
Tianon Gravi
3f2d1eb95d
Merge pull request #40317 from AkihiroSuda/bump-up-rootlesskit
rootless: fix proxying UDP packets
2019-12-18 14:49:49 -08:00
Akihiro Suda
8b8c01dd87
Merge pull request #40074 from yedamao/fix-integration-cli
integration-cli: Fix `SA1019: httputil.ClientConn is deprecated`
2019-12-19 03:30:34 +09:00
Akihiro Suda
658723badd rootless: fix proxying UDP packets
UDP reply packets were not proxied: https://github.com/rootless-containers/rootlesskit/issues/86

The issue was fixed in RootlessKit v0.7.1: https://github.com/rootless-containers/rootlesskit/pull/87

Full changes since v0.7.0: https://github.com/rootless-containers/rootlesskit/compare/v0.7.0...v0.7.1

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-12-18 19:33:27 +09:00
Tõnis Tiigi
550d5f9d43
Merge pull request #40180 from cpuguy83/no_more_dev_tools
Build binaries with minimal deps and remove autogen code
2019-12-12 11:29:30 -08:00
Jintao Zhang
7f809e1080 Update containerd to v1.3.2
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
2019-12-09 01:23:40 +08:00
Brian Goff
79a52bb1ba Add support for outputing binaries to custom dir
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-12-02 10:21:46 -08:00
Brian Goff
675b414f56 Use -X ldflags to set dockerversion package vars
This eliminates the need to lay down an auto-generated file.
IIRC this was originally hadded for gccgo which we no longer support.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-12-02 10:21:41 -08:00
Brian Goff
3152f94362
Merge pull request #40048 from tao12345666333/update-to-runc-1.0.0-rc9
Update containerd to v1.3.1, runc v1.0.0-rc9
2019-11-27 14:20:17 -08:00
Jintao Zhang
517946eb47 Update containerd to v1.3.1
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
2019-11-20 23:23:07 +08:00
Sebastiaan van Stijn
9bcbc66032
Merge pull request #40169 from cpuguy83/windows_version_quad_err
Windows: Only set VERSION_QUAD if unset
2019-11-11 17:14:12 -08:00
Sebastiaan van Stijn
9cf349d0f8
bump libnetwork 90afbb01e1d8acacb505a092744ea42b9f167377
full diff: 0025177e3d...90afbb01e1

includes:

- docker/libnetwork#/2459 Fix Error Check in NewNetwork
- docker/libnetwork#/2466 Revert "Merge pull request #2339 from phyber/iptables-check"
    - reverts docker/libnetwork#/2339 controller: Check if IPTables is enabled for arrangeUserFilterRule
    - re-opens docker/libnetwork#2158 dockerd when run with --iptables=false modifies iptables by adding DOCKER-USER
    - re-opens moby/moby#35777 With iptables=false dockerd still creates DOCKER-USER chain and rules
    - re-opens docker/for-linux#136 dockerd --iptables=false adds DOCKER-USER chain and modify FORWARD chain anyway

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-11-07 13:43:42 -08:00
Sebastiaan van Stijn
e9bd017b68
Merge pull request #40148 from kolyshkin/vendor-diff
hack/validate/vendor: print diff for modified files
2019-11-07 12:42:03 -08:00
Kirill Kolyshkin
c36460c437
Merge pull request #40077 from thaJeztah/fix_autogen_detection
Update "auto-generate" comments to improve detection by linters
2019-11-05 13:04:14 -08:00
Brian Goff
47c5c67ed8
Merge pull request #40032 from jmartin84/fix-grpc-withdialer-deprecation-warning
Fix grpc withdialer deprecation warning
2019-11-05 12:20:33 -08:00
Sebastiaan van Stijn
6186e9fe87
hack/make/.go-autogen: fix "generated code" comment not in correct format
As described in https://golang.org/s/generatedcode, Go has
a formalized format that should be used to indicate that a
file is generated.

Matching that format helps linters to skip generated files;

From https://golang.org/s/generatedcode (https://github.com/golang/go/issues/13560#issuecomment-288457920);

> Generated files are marked by a line of text that matches the regular expression, in Go syntax:
>
> ^// Code generated .* DO NOT EDIT\.$
>
> The `.*` means the tool can put whatever folderol it wants in there, but the comment
> must be a single line and must start with `Code generated` and end with `DO NOT EDIT.`,
> with a period.
>
> The text may appear anywhere in the file.

This patch updates the autogenerated code to match that format.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-11-05 11:32:43 -08:00
Brian Goff
ce931f28ea Windows: Only set VERSION_QUAD if unset
When trying to build with some pretty typical version strings this was
causing failures trying to generate the windows resource file.

The resource file is already gated by an `ifdef` for this var, so
instead of blindly setting based on "VERSION", which can contain some
characters which are incompatible (e.g. 1.2.3.rc.0 will fail due to the
".rc").

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-11-04 17:16:06 -08:00
Kir Kolyshkin
5791019028 hack/validate/vendor: simplify looking for license
It was suggested that we use '.*\(COPYING\|LICENSE\|COPYRIGHT\).*'
as an argument to `find -iregex`, and this is how it all started.

Next thing, there is no COPYRIGHT in any of the vendored packages,
so it can be removed for good.

Next, we should not look too deep inside the package directory, as
the license should be in its root directory, so add `-maxdepth 1`
to `find`. This should also speed things up.

Finally, since we're not using the recursion feature of `find`,
it can be replaced with `echo | grep`.

While at it,
* avoid temporary $pkgs variable as it is only used once;
* replace `ls -d "vendor/$f"  > /dev/null 2>&1` with `test -d`.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2019-10-29 10:59:05 -07:00
Kir Kolyshkin
4be12ad3d0 hack/validate/vendor: shellcheck fixes
The export statement is definitely not needed. The rest is obvious.

> In hack/validate/vendor line 3:
> export SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
>        ^-- SC2155: Declare and assign separately to avoid masking return values.
>
>
> In hack/validate/vendor line 43:
>	if ls -d vendor/$f  > /dev/null 2>&1; then
>                       ^-- SC2086: Double quote to prevent globbing and word splitting.
>
>
> In hack/validate/vendor line 44:
> 		found=$(find vendor/$f -iregex '.*LICENSE.*' -or -iregex '.*COPYRIGHT.*' -or -iregex '.*COPYING.*' | wc -l)
>                                   ^-- SC2086: Double quote to prevent globbing and word splitting.
>
>
> In hack/validate/vendor line 45:
>		if [ $found -eq 0 ]; then
>                    ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2019-10-28 19:27:55 -07:00
Kir Kolyshkin
9d4e81e8bf hack/validate/vendor: print diff for modified files
In case some files were modified (rather than merely added
or removed), we're curious to see the diff for those.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2019-10-28 19:22:28 -07:00
Jintao Zhang
5ce9df2a40 Update to runc v1.0.0-rc9
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
2019-10-29 01:59:24 +08:00
Brian Goff
221012c92c
Merge pull request #40110 from thaJeztah/fix_test_filter
Fix TEST_FILTER to work for both "integration" and "integration-cli"
2019-10-25 09:00:13 -07:00
HuanHuan Ye
203ba72fc5 integration-cli: Fix SA1019: httputil.ClientConn is deprecated
Rewrite sockRequestHijack to requestHijack which use writable
Transport's Response.Body to replace deprecated hijacked httputil.ClientConn.
```
// As of Go 1.12, the Body will also implement io.Writer
// on a successful "101 Switching Protocols" response,
// as used by WebSockets and HTTP/2's "h2c" mode.
Body io.ReadCloser
```.

TestPostContainersAttach and TestExecResizeImmediatelyAfterExecStart
replace all sockRequestHijack to requestHijack.

Signed-off-by: HuanHuan Ye <logindaveye@gmail.com>
2019-10-22 10:55:29 +08:00
Sebastiaan van Stijn
e7805653b8
Fix TEST_FILTER to work for both "integration" and "integration-cli"
The TEST_FILTER variable allows running a single integration or integration-cli
test. However, it failed to work properly for integration-cli tests.

Before:
-----------

    # Filtering "integration" tests works:
    make TEST_FILTER=TestInspectCpusetInConfigPre120 test-integration
    ...
    DONE 1 tests in 18.331s

    # But running a single test in "integration-cli" did not:

    make TEST_FILTER=TestSwarmNetworkCreateIssue27866 test-integration
    ...
    DONE 0 tests in 17.314s

Trying to manually add the `/` prefix, didn't work either, because that made the
"grep" fail to find which test-suites to run/skip:

    make TEST_FILTER=/TestSwarmNetworkCreateIssue27866 test-integration
    ---> Making bundle: test-integration (in bundles/test-integration)
    make: *** [test-integration] Error 1

After:
-----------

    make TEST_FILTER=TestInspectCpusetInConfigPre120 test-integration
    ...
    DONE 1 tests in 18.331s

    make TEST_FILTER=TestSwarmNetworkCreateIssue27866 test-integration
    ...
    DONE 12 tests in 26.527s

Note that the `12` tests is still a bit misleading, because every _suite_ is
started (which is counted as a test), but no tests are run. This is still
something that could be improved on.

This patch also makes a small modification to the code that's setting
`integration_api_dirs`, and no longer runs `go list` if not needed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-21 14:03:58 +02:00
Sebastiaan van Stijn
9cb9c6eba4
bump golangci-lint v1.20.0
release notes:

- v1.20.0: https://github.com/golangci/golangci-lint/releases/tag/v1.20.0
    - Add WSL linter (#771)
    - Add gocognit linter (#756)
    - govet: add more analyzers
    - dramatically reduce memory usage
    - reduce 1.5x memory usage on large repos on repeated runs
    - Rename deadline option to timeout and mark deadline as deprecated
- v1.19.1: https://github.com/golangci/golangci-lint/releases/tag/v1.19.1
- v1.19.0: https://github.com/golangci/golangci-lint/releases/tag/v1.19.0
    - Add dogsled linter.
    - Add whitespace linter
    - Add Godox linter support
    - Various performance optimizations
- v1.18.0: https://github.com/golangci/golangci-lint/releases/tag/v1.18.0
    - Support go1.13
    - Update format of junit xml output to mark failures as such
    - Speed up linting: use deduplicated packages
    - Add user supplied error messages in depguard issues
    - Add funlen linter

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-18 00:46:15 +02:00
Sebastiaan van Stijn
a06d7f5a3e
golangci-lint: include all enabled linters in config
This makes it clearer from the configuration what linters
are enabled.

Also disables the `gofmt` linter, because it's superseded
by the `goimports` linter.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-18 00:46:13 +02:00
Sebastiaan van Stijn
086b4541cf
golangci-lint: disable default exclusion rules
The default exclusion rules were too permissive; disable
them, but copy the relevant ones to the configuation.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-18 00:46:11 +02:00
Sebastiaan van Stijn
fe3c14d5ba
golangci-lint: fix "golint" not ignoring generated files
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-18 00:46:08 +02:00
Sebastiaan van Stijn
47502344b9
golangci-lint: update exclusion rules for todo's
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-18 00:46:04 +02:00
Grant Millar
df7b8f458a daemon: Use short libnetwork ID in exec-root & update libnetwork
Signed-off-by: Grant Millar <rid@cylo.io>
2019-10-15 11:40:24 +01:00
Brian Goff
28b645755a
Merge pull request #40062 from thaJeztah/testutil_improvements
Various testutil improvements
2019-10-11 15:37:48 -07:00
Sebastiaan van Stijn
9407a57522
hack/make: don't attempt to unmount non-existing daemon root-dir
Before:

    DONE 2 tests in 12.272s
    ---> Making bundle: .integration-daemon-stop (in bundles/test-integration)
    umount: bundles/test-integration/root: mountpoint not found

After:

    DONE 2 tests in 14.650s
    ---> Making bundle: .integration-daemon-stop (in bundles/test-integration)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-11 00:34:53 +02:00
Justen Martin
3b49bd1d84
replaced call to deprecated grpc method WithDialer with WithContextDialer
Signed-off-by: Justen Martin <jmart@the-coder.com>
2019-10-10 15:34:42 -05:00
Justen Martin
095ca77f48
Use build args to override binary commits in dockerfile
Signed-off-by: Justen Martin <jmart@the-coder.com>
2019-10-10 14:52:57 -05:00
Tibor Vass
d31633be7d
Merge pull request #39678 from thaJeztah/carry_39595_tailor_arm_ci
[Carry 39595] Tailor CI for ARM, skip legacy integration test.
2019-10-10 11:22:02 -07:00
Michael Zhao
48b06a2561
Tailor CI for ARM, skip legacy integration test.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-08 18:37:13 +02:00
Derek McGowan
6c94a50f41
update containerd binary v1.3.0
full diff: https://github.com/containerd/containerd/compare/v1.2.8..v1.3.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-10-04 15:05:32 -07:00
Sebastiaan van Stijn
5a703ccb46
bump docker-py to 4.1.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-03 16:51:03 +02:00
Sebastiaan van Stijn
48353e16fe
Temporarily switch docker-py to "master"
The docker-py tests were broken, because the version of
py-test that was used, used a dependency that had a new
major release with a breaking change.

Unfortunately, it was not pinned to a specific version,
so when the dependency did the release, py-test broke;

```
22:16:47  Traceback (most recent call last):
22:16:47    File "/usr/local/bin/pytest", line 10, in <module>
22:16:47      sys.exit(main())
22:16:47    File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 61, in main
22:16:47      config = _prepareconfig(args, plugins)
22:16:47    File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 182, in _prepareconfig
22:16:47      config = get_config()
22:16:47    File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 156, in get_config
22:16:47      pluginmanager.import_plugin(spec)
22:16:47    File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
22:16:47      __import__(importspec)
22:16:47    File "/usr/local/lib/python3.6/site-packages/_pytest/tmpdir.py", line 25, in <module>
22:16:47      class TempPathFactory(object):
22:16:47    File "/usr/local/lib/python3.6/site-packages/_pytest/tmpdir.py", line 35, in TempPathFactory
22:16:47      lambda p: Path(os.path.abspath(six.text_type(p)))
22:16:47  TypeError: attrib() got an unexpected keyword argument 'convert'
```

docker-py master has a fix for this (bumping the version of
`py-test`), but it's not in a release yet, and the docker cli that's used
in our CI is pinned to 17.06, which doesn't support building from a remote
git repository from a specific git commit.

To fix the immediate situation, this patch switches the docker-py
tests to run from the master branch.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-02 17:54:48 +02:00