Otherwise the cache gets written to bundles instead of to the volume
that we setup in the Makefile as expected.
Found this because vscode did not like having these huge objects in
bundles/ (didn't harm anything, just that it noticed and said hey I don't
like that).
This enables cache-reuse when building the test binaries.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
For CI, a temporary `DOCKER_ALLOW_SCHEMA1_PUSH_DONOTUSE` environment
variable was added while we work out a solution for testing schema 1
pulls (which currently require pushing them to a local registry first
for testing).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Usage: DOCKER_BUILD_ARGS="--build-arg CONTAINERD_COMMIT=master --build-arg RUNC_COMMIT=master" DOCKER_EXPERIMENTAL=1 TEST_SKIP_INTEGRATION_CLI=1 make test-integration
Depends on containerd master (v1.4) and runc master (v1.0.0-rc91).
Currently `TEST_SKIP_INTEGRATION_CLI=1` must be specified.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Usage: `dockerd-rootless-setuptool.sh install --force` .
See `--help` for further information.
Co-authored-by: Tianon Gravi <admwiggin@gmail.com>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The containerd.installer was updated to also copy `containerd-shim-runc-v2`,
but `hack/make/binary-daemon` did not copy it to the bundles directory.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Close#40484
Note that the support for cgroup v2 isn't ready for production yet,
regardless to rootful or rootless.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This test was disabled in the past, but re-enabled when we upgraded
docker-py to 4.2.0.
The test looks to be still flaky though, so skipping it again:
```
[2020-02-10T23:40:44.429Z] =================================== FAILURES ===================================
[2020-02-10T23:40:44.429Z] __________________ AttachContainerTest.test_attach_no_stream ___________________
[2020-02-10T23:40:44.429Z] tests/integration/api_container_test.py:1250: in test_attach_no_stream
[2020-02-10T23:40:44.429Z] assert output == 'hello\n'.encode(encoding='ascii')
[2020-02-10T23:40:44.429Z] E AssertionError: assert b'' == b'hello\n'
[2020-02-10T23:40:44.429Z] E Right contains more items, first extra item: 104
[2020-02-10T23:40:44.429Z] E Use -v to get the full diff
[2020-02-10T23:40:44.429Z] ------- generated xml file: /src/bundles/test-docker-py/junit-report.xml -------
````
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Otherwise some tests are skipped with the default API version
used:
SKIPPED [1] tests/integration/api_service_test.py:882: API version is too low (< 1.38)
SKIPPED [1] tests/integration/api_swarm_test.py:59: API version is too low (< 1.39)
SKIPPED [1] tests/integration/api_swarm_test.py:38: API version is too low (< 1.39)
SKIPPED [1] tests/integration/api_swarm_test.py:45: API version is too low (< 1.39)
SKIPPED [1] tests/integration/api_swarm_test.py:52: API version is too low (< 1.39)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Windows still writes to the autogen directory, but the source code is
mounted in as read-only.
In order to do enable this without taking a massive hit in doing an rw
mount (for the source code) we mount a tmpfs into the build at the
autogen dir.
In order for this to work the directory must alreay exist, so we create
it before entering the build.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The makefile for this binary has version 6.3 hardcoded,
which causes compilation on 8.3 to fail:
```
Building: bundles/cross/windows/amd64/containerutility.exe
In file included from /usr/x86_64-w64-mingw32/include/minwindef.h:163,
from /usr/x86_64-w64-mingw32/include/windef.h:8,
from /usr/x86_64-w64-mingw32/include/windows.h:69,
from containerutility.h:3,
from argumentstream.cpp:1:
/usr/x86_64-w64-mingw32/include/winnt.h:1554:11: fatal error: x86intrin.h: No such file or directory
# include <x86intrin.h>
^~~~~~~~~~~~~
compilation terminated.
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
```
---> Making bundle: .integration-daemon-stop (in bundles/test-integration)
++++ cat bundles/test-integration/docker.pid
+++ kill 13137
+++ /etc/init.d/apparmor stop
Leaving: AppArmorNo profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.
To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run 'aa-teardown'."
script returned exit code 255
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The ImageCollectionTest.test_pull_multiple test performs a `docker pull` without
a `:tag` specified) to pull all tags of the given repository (image).
After pulling the image, the image(s) pulled are checked to verify if the list
of images contains the `:latest` tag.
However, the test assumes that all tags of the image are tags for the same
version of the image (same digest), and thus a *single* image is returned, which
is not always the case.
Currently, the `hello-world:latest` and `hello-world:linux` tags point to a
different digest, therefore the `client.images.pull()` returns multiple images:
one image for digest, making the test fail:
=================================== FAILURES ===================================
____________________ ImageCollectionTest.test_pull_multiple ____________________
tests/integration/models_images_test.py:90: in test_pull_multiple
assert len(images) == 1
E AssertionError: assert 2 == 1
E + where 2 = len([<Image: 'hello-world:linux'>, <Image: 'hello-world:latest'>])
This patch temporarily skips the broken test until it is fixed upstream.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This eliminates the need to lay down an auto-generated file.
IIRC this was originally hadded for gccgo which we no longer support.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
As described in https://golang.org/s/generatedcode, Go has
a formalized format that should be used to indicate that a
file is generated.
Matching that format helps linters to skip generated files;
From https://golang.org/s/generatedcode (https://github.com/golang/go/issues/13560#issuecomment-288457920);
> Generated files are marked by a line of text that matches the regular expression, in Go syntax:
>
> ^// Code generated .* DO NOT EDIT\.$
>
> The `.*` means the tool can put whatever folderol it wants in there, but the comment
> must be a single line and must start with `Code generated` and end with `DO NOT EDIT.`,
> with a period.
>
> The text may appear anywhere in the file.
This patch updates the autogenerated code to match that format.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When trying to build with some pretty typical version strings this was
causing failures trying to generate the windows resource file.
The resource file is already gated by an `ifdef` for this var, so
instead of blindly setting based on "VERSION", which can contain some
characters which are incompatible (e.g. 1.2.3.rc.0 will fail due to the
".rc").
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The TEST_FILTER variable allows running a single integration or integration-cli
test. However, it failed to work properly for integration-cli tests.
Before:
-----------
# Filtering "integration" tests works:
make TEST_FILTER=TestInspectCpusetInConfigPre120 test-integration
...
DONE 1 tests in 18.331s
# But running a single test in "integration-cli" did not:
make TEST_FILTER=TestSwarmNetworkCreateIssue27866 test-integration
...
DONE 0 tests in 17.314s
Trying to manually add the `/` prefix, didn't work either, because that made the
"grep" fail to find which test-suites to run/skip:
make TEST_FILTER=/TestSwarmNetworkCreateIssue27866 test-integration
---> Making bundle: test-integration (in bundles/test-integration)
make: *** [test-integration] Error 1
After:
-----------
make TEST_FILTER=TestInspectCpusetInConfigPre120 test-integration
...
DONE 1 tests in 18.331s
make TEST_FILTER=TestSwarmNetworkCreateIssue27866 test-integration
...
DONE 12 tests in 26.527s
Note that the `12` tests is still a bit misleading, because every _suite_ is
started (which is counted as a test), but no tests are run. This is still
something that could be improved on.
This patch also makes a small modification to the code that's setting
`integration_api_dirs`, and no longer runs `go list` if not needed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before:
DONE 2 tests in 12.272s
---> Making bundle: .integration-daemon-stop (in bundles/test-integration)
umount: bundles/test-integration/root: mountpoint not found
After:
DONE 2 tests in 14.650s
---> Making bundle: .integration-daemon-stop (in bundles/test-integration)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The docker-py tests were broken, because the version of
py-test that was used, used a dependency that had a new
major release with a breaking change.
Unfortunately, it was not pinned to a specific version,
so when the dependency did the release, py-test broke;
```
22:16:47 Traceback (most recent call last):
22:16:47 File "/usr/local/bin/pytest", line 10, in <module>
22:16:47 sys.exit(main())
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 61, in main
22:16:47 config = _prepareconfig(args, plugins)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 182, in _prepareconfig
22:16:47 config = get_config()
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 156, in get_config
22:16:47 pluginmanager.import_plugin(spec)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
22:16:47 __import__(importspec)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/tmpdir.py", line 25, in <module>
22:16:47 class TempPathFactory(object):
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/tmpdir.py", line 35, in TempPathFactory
22:16:47 lambda p: Path(os.path.abspath(six.text_type(p)))
22:16:47 TypeError: attrib() got an unexpected keyword argument 'convert'
```
docker-py master has a fix for this (bumping the version of
`py-test`), but it's not in a release yet, and the docker cli that's used
in our CI is pinned to 17.06, which doesn't support building from a remote
git repository from a specific git commit.
To fix the immediate situation, this patch switches the docker-py
tests to run from the master branch.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Generate more unique names, based on architecture and test-suite name.
Clean up the path to this integration test to create a useful package name.
"$dir" can be either absolute (/go/src/github.com/docker/docker/integration/foo)
or relative (./integration/foo). To account for both, first we strip the
absolute path, then any leading periods and slashes.
For the package-name, we use periods as separator instead of slashes, to be more
in-line with Java package names (which is what junit.xml was originally designed
for).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Without these options set, test2json does not include a `Time`
field in the generated JSON;
{"Action":"run","Test":"TestCgroupNamespacesBuild"}
{"Action":"output","Test":"TestCgroupNamespacesBuild","Output":"=== RUN TestCgroupNamespacesBuild\n"}
{"Action":"output","Test":"TestCgroupNamespacesBuild","Output":"--- PASS: TestCgroupNamespacesBuild (1.70s)\n"}
...
{"Action":"pass","Test":"TestCgroupNamespacesBuild"}
As a result, `gotestsum` was not able to calculate test-duration, and
reported `time="0.000000"` for all tests;
<testcase classname="amd64.integration.build" name="TestCgroupNamespacesBuild" time="0.000000"></testcase>
With this patch applied:
{"Time":"2019-08-23T22:42:41.644361357Z","Action":"run","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild"}
{"Time":"2019-08-23T22:42:41.644367647Z","Action":"output","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Output":"=== RUN TestCgroupNamespacesBuild\n"}
{"Time":"2019-08-23T22:42:44.926933252Z","Action":"output","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Output":"--- PASS: TestCgroupNamespacesBuild (3.28s)\n"}
...
{"Time":"2019-08-23T22:42:44.927003836Z","Action":"pass","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Elapsed":3.28}
Which now correctly reports the test's duration:
<testcase classname="amd64.integration.build" name="TestCgroupNamespacesBuild" time="3.280000"></testcase>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
'Namespace' parallel runs by bind-mounting a different directory
in the container, instead of making the tests running inside
the container aware of the namespaced location.
This makes it transparent to the tests, and slightly reduces
complexity.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
the files used by the docker cli were moved to the docker/cli repository,
so are no longer needed here.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- remove -check.* flags
- use (per-test) -timeout flag
- allow user to override TEST_SKIP_* regardless of TESTFLAGS
- remove test-imports validation
Signed-off-by: Tibor Vass <tibor@docker.com>
Seen failing a couple of times:
```
[2019-09-02T08:40:15.796Z] =================================== FAILURES ===================================
[2019-09-02T08:40:15.796Z] __________________ AttachContainerTest.test_attach_no_stream ___________________
[2019-09-02T08:40:15.796Z] tests/integration/api_container_test.py:1250: in test_attach_no_stream
[2019-09-02T08:40:15.796Z] assert output == 'hello\n'.encode(encoding='ascii')
[2019-09-02T08:40:15.796Z] E AssertionError: assert b'' == b'hello\n'
[2019-09-02T08:40:15.796Z] E Right contains more items, first extra item: 104
[2019-09-02T08:40:15.796Z] E Use -v to get the full diff
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Add quotes to prevent word splitting in `cp` statement (SC2046)
- Replace legacy back tics with `$()`
- Replace `which` with `command -v` (SC2230)
- Fix incorrect (`==`) comparison
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- SC2006: use $(...) notation instead of legacy backticked `...`
- SC2086: double quote to prevent globbing and word splitting
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
and remove `PullImageTest::test_build_invalid_platform` from the list,
which was a copy/paste error in f8cde0b32d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The "new test" detector in test-integration-flaky was a bit flaky since
it would detect function signatures that are not new tests.
In addition, the test calls `return` outside of a function which is not
allowed.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This is basically taking some stuff that make a custom shell function
for.
This takes a test filter, builds the appropriate TESTFLAGS, and sets the
integration API test dirs that match the given filter to avoid building
all test dirs.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
1. Use `go list` to get list of integration dirs to build. This means we
do not need to have a valid `.go` in every subdirectory and also
filters out other dirs like "bundles" which may have been created.
2. Add option to specify custom flags for integration and
integration-cli. This is needed so both suites can be run AND set
custom flags... since the cli suite does not support standard go
flags.
3. Add options to skip an entire integration suite.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Since commit applied in moby [61a3285 Support cross-compile for arm]
it hardcoded var-CC to support cross-compile for arm
Correct it with "${parameter:-word}" format, it is helpful for user
define toolchains
(Use Default Values. If parameter is unset or null, the expansion of
word is substituted. Otherwise, the value of parameter is substituted.)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
When building this image docker-in-docker, the DNS in the environment
may not be usable for the build-container, causing resolution to fail:
```
02:35:31 W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release.gpg Temporary failure resolving 'deb.debian.org'
```
This patch detects if we're building from within a container, and if
so, skips creating a networking namespace for the build by using
`--network=host`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This removes all the installation steps for docker-py from the
Dockerfile, and instead builds the upstream Dockerfile, and runs
docker-py tests in a container.
To test;
```
make test-docker-py
...
Removing bundles/
---> Making bundle: dynbinary (in bundles/dynbinary)
Building: bundles/dynbinary-daemon/dockerd-dev
Created binary: bundles/dynbinary-daemon/dockerd-dev
---> Making bundle: test-docker-py (in bundles/test-docker-py)
---> Making bundle: .integration-daemon-start (in bundles/test-docker-py)
Using test binary docker
Starting dockerd
INFO: Waiting for daemon to start...
.
INFO: Building docker-sdk-python3:3.7.0...
sha256:686428ae28479e9b5c8fdad1cadc9b7a39b462e66bd13a7e35bd79c6a152a402
INFO: Starting docker-py tests...
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-4.1.0, py-1.8.0, pluggy-0.9.0
rootdir: /src, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
collected 359 items
tests/integration/api_build_test.py .......s....
....
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
integration-on-swarm had unnecessary complexity and was too hard to
maintain. Also, it didn't support the new non-CLI integration test suite.
I'm now doing some experiments out of the repo using Kubernetes:
https://github.com/AkihiroSuda/kube-moby-integration
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Pretty much cross-compile doesn't work because of this:
> profiles/seccomp/seccomp.go:13:2: build constraints exclude all Go files in /go/src/github.com/docker/docker/vendor/github.com/seccomp/libseccomp-golang
This changes adds a new Dockerfile target for cross compilation with the
neccesary arch specific libseccomp packages and CC toolchains.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Now `docker run -p` ports can be exposed to the host namespace automatically when `dockerd-rootless.sh` is launched with
`--userland-proxy --userland-proxy-path $(which rootlesskit-docker-proxy)`.
This is akin to how Docker for Mac/Win works with `--userland-proxy-path=/path/to/vpnkit-expose-port`.
The port number on the host namespace needs to be set to >= 1024.
SCTP ports are currently unsupported.
RootlessKit changes: 7bbbc48a6f...ed26714429
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Here's what happens:
1. One runs `make binary` once
2. Days go by...
3. One makes changes to dockerd sources
4. One runs `make test-integration` to test the changes
5. One spends a long time figuring out why on Earth
those changes in step 3 are ignored by step 4.
6. One writes this patch
7. ...
8. PROFIT!!
OK, so `make test-integration` builds a dockerd binary
in bundles/dynbinary-daemon/, when starts a daemon instance
for testing. The problem is, the script that starts the
daemon sets PATH to try `bundles/binary-daemon/` first,
and `bundles/dynbinary-daemon/` second.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Please refer to `docs/rootless.md`.
TLDR:
* Make sure `/etc/subuid` and `/etc/subgid` contain the entry for you
* `dockerd-rootless.sh --experimental`
* `docker -H unix://$XDG_RUNTIME_DIR/docker.sock run ...`
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
comparing PR commit(s) to HEAD of moby/moby master branch and if founds
new (or renamed) integration tests will run stress tests for them.
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
These scripts explicitly use Bash, so we should be able to use
`[[` instead of `[` (which seems to be recommended).
Also added curly brackets to some bare variables, and quoted some paths.
This makes my IDE a bit more silent :-)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
With containerd reaching 1.0, the runtime now
has a stable API, so there's no need to do a check
if the installed version matches the expected version.
Current versions of Docker now also package containerd
and runc separately, and can be _updated_ separately.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* Expose license status in Info
This wires up a new field in the Info payload that exposes the license.
For moby this is hardcoded to always report a community edition.
Downstream enterprise dockerd will have additional licensing logic wired
into this function to report details about the current license status.
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Code review comments
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Add windows autogen support
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.
NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.
The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>
On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.
Signed-off-by: Salahuddin Khan <salah@docker.com>
Go 1.11 includes a fix to os/user to be working in a static binary
(fixing https://github.com/golang/go/issues/23265). The fix requires
`osusergo` build tag to be set for static binaries, which is what
this commit adds (also for containerd).
[v2: sort build tags alphabetically]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
