1. Add = between the option and the argument, otherwise the argument
appears to be passed on to the linters directly, as in:
> DEBUG: [golint.8]: executing /home/kir/go/bin/golint
> -min_confidence 0.800000 ./10m ./api ./api/errdefs <...>
2. Fix setting the default for GOMETALINTER_OPTS -- the default
was -deadline (rather than --deadline).
Fixes: b96093fa56 ("gometalinter: add per-platform configurable options")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The Golang built-in gzip library is serialized, and fairly slow
at decompressing. It also only decompresses on demand, versus
pipelining decompression.
This change switches to using the pigz external command
for gzip decompression, as opposed to using the built-in
golang one. This code is not vendored, but will be used
if it autodetected as part of the OS.
This also switches to using context, versus a manually
managed channel to manage cancellations, and synchronization.
There is a little bit of weirdness around manually having
to cancel in the error cases.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Upgrade the frozen images to the multi-arch ones.
Since issue #35963 is not fixed yet on linux/amd64, so we keep the busybox
image on amd64 untouched.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
go1.8.5 (released 2017/10/25) includes fixes to the compiler, linker, runtime,
documentation, go command, and the crypto/x509 and net/smtp packages. It
includes a fix to a bug introduced in Go 1.8.4 that broke go get of non-Git
repositories under certain conditions. See the Go 1.8.5 milestone on our issue
tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.8.5
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When running 'make all' on armhf, I got this:
> ---> Making bundle: .integration-daemon-start (in bundles/17.06.0-dev/test-docker-py)
> Using test binary docker
> INFO: Waiting for daemon to start...
> Starting dockerd
> .
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/_pytest/config.py", line
> 320, in _importconftest
> mod = conftestpath.pyimport()
> File "/usr/local/lib/python2.7/dist-packages/py/_path/local.py", line
> 662, in pyimport
> __import__(modname)
> File "/docker-py/tests/integration/conftest.py", line 6, in <module>
> import docker.errors
> File "/docker-py/docker/__init__.py", line 2, in <module>
> from .api import APIClient
> File "/docker-py/docker/api/__init__.py", line 2, in <module>
> from .client import APIClient
> File "/docker-py/docker/api/client.py", line 11, in <module>
> from .build import BuildApiMixin
> File "/docker-py/docker/api/build.py", line 6, in <module>
> from .. import auth
> File "/docker-py/docker/auth.py", line 6, in <module>
> import dockerpycreds
> ImportError: No module named dockerpycreds
> ERROR: could not load /docker-py/tests/integration/conftest.py
The fix for this was already provided by commit 0ec8f56a3 and
commit c7c923594, but for some reason it did not made its way
to Dockerfiles for all architectures.
While at it, remove excessive comments.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Presumably after switch to debian-stretch as a base, the following
errors happens in Jenkins:
10:48:03 ---> Making bundle: test-docker-py (in
bundles/17.06.0-dev/test-docker-py)
10:48:03 ---> Making bundle: .integration-daemon-start (in
bundles/17.06.0-dev/test-docker-py)
10:48:03 Using test binary docker
10:48:03 # DOCKER_EXPERIMENTAL is set: starting daemon with experimental
features enabled!
10:48:03 /etc/init.d/apparmor: 130: /etc/init.d/apparmor:
systemd-detect-virt: not found
10:48:03 Starting AppArmor profiles:Warning from stdin (line 1):
/sbin/apparmor_parser: cannot use or update cache, disable, or
force-complain via stdin
10:48:03 Warning failed to create cache: (null)
10:48:03 .
10:48:03 INFO: Waiting for daemon to start...
10:48:03 Starting dockerd
10:48:05 .
10:48:06 Traceback (most recent call last):
10:48:06 File
"/usr/local/lib/python2.7/dist-packages/_pytest/config.py", line 320, in
_importconftest
10:48:06 mod = conftestpath.pyimport()
10:48:06 File
"/usr/local/lib/python2.7/dist-packages/py/_path/local.py", line 662, in
pyimport
10:48:06 __import__(modname)
10:48:06 File "/docker-py/tests/integration/conftest.py", line 6, in
<module>
10:48:06 import docker.errors
10:48:06 File "/docker-py/docker/__init__.py", line 2, in <module>
10:48:06 from .api import APIClient
10:48:06 File "/docker-py/docker/api/__init__.py", line 2, in <module>
10:48:06 from .client import APIClient
10:48:06 File "/docker-py/docker/api/client.py", line 6, in <module>
10:48:06 import requests
10:48:06 ImportError: No module named requests
10:48:06 ERROR: could not load /docker-py/tests/integration/conftest.py
10:48:06
and
00:38:55 File "/docker-py/docker/transport/ssladapter.py", line 21, in
<module>
00:38:55 from backports.ssl_match_hostname import match_hostname
00:38:55 ImportError: No module named backports.ssl_match_hostname
00:38:55 ERROR: could not load /docker-py/tests/integration/conftest.py
To fix, install the missing python modules.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Static build with devmapper is impossible now since libudev is required
and no static version of libudev is available (as static libraries are
not supported by systemd which udev is part of).
This should not hurt anyone as "[t]he primary user of static builds
is the Editions, and docker in docker via the containers, and none
of those use device mapper".
Also, since the need for static libdevmapper is gone, there is no need
to self-compile libdevmapper -- let's use the one from Debian Stretch.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The main gain here is that they all use exactly the same distro; previously
arm64 was using Ubuntu Xenial because Debian jessie was too old.
Does not seem that we can change any of the downloaded dependencies still,
as eg libseccomp is still not the version we are using.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Version 2.02.173 has disappeared, let's revert back to latest stable
one.
https://github.com/moby/moby/issues/34843
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
I have run into two separate issues while doing 'make all' on armhf
(a Scaleway C1 machine, same as used in CI). This commit fixes both.
1. There were a lot of "not enough memory" errors, and after that
in a few runs gometalinter just stuck forever on FUTEX_WAIT with
no children left.
Looking into docs, I found the --enable-gc option which solved the issue.
[Update: this has already been added]
2. Timeout of 2 minutes is not enough for the abovementioned platform.
The longest running linter is goimports which takes almost 6 minutes to run.
Set the timeout to the observable run time roughly doubled.
In addition, ARM platforms does not have too much RAM (2GB), so
running too many processes in parallel might be problematic. Limit
it by using -j2
[v2: make the timeout arch-dependent, also tested on aarch64 (2m15s)]
[v3: moved timeout setting to Dockerfiles]
[v4: generalized to GOMETALINTER_OPTS, added -j2 for ARM platforms]
[v5: rebase to master]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Since commit d7e2c4ce7 ("Use gometalinter for linting") command
"make all" fails on all the non-default platforms (i.e. ARMs, PPC, and
s390) in this way:
# make all
...
Congratulations! All commits are properly signed with the DCO!
/go/src/github.com/docker/docker/hack/validate/gometalinter: line 6: gometalinter: command not found
Makefile:105: recipe for target 'all' failed
make: *** [all] Error 127
Make sure gometalinter is installed for those platforms
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Let's use latest lvm2 sources to compile the libdevmapper library.
Initial reason for compiling devmapper lib from sources was a need to
have the static version of the library at hand, in order to build
the static dockerd, but note that the same headers/solib are used
for dynamic build (dynbinary) as well.
The reason for this patch is to enable the deferral removal feature.
The supplied devmapper library (and headers) are too old, lacking the
needed functions, so the daemon is built with 'libdm_no_deferred_remove'
build tag (see the check in hack/make.sh). Because of this, even if the
kernel dm driver is perfectly able to support the feature, it can not
be used. For more details and background story, see [1].
Surely, one can't just change the version number. While at it:
- improve the comments;
- remove obsoleted URLs;
- remove s390 and ppc configure updates that are no longer needed;
- use pkg-config instead of hardcoding the flags (newer lib added
some more dependencies);
[1] https://github.com/moby/moby/issues/34298
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
With docker-17.06.0 some images pulled do not extract properly. Some files don't appear in correct directories. This may or may not cause the pull to fail. These images can't be pushed or saved. 17.06 is the first version of Docker built with go1.8.
Cause
There are multiple updates to the tar package in go1.8.
https://go-review.googlesource.com/c/32234/ disables using "prefix" field when new tar archives are being written. Prefix field was previously set when a record in the archive used a path longer than 100 bytes.
Another change https://go-review.googlesource.com/c/31444/ makes the reader ignore the "prefix" field value if the record is in GNU format. GNU format defines that same area should be used for access and modified times. If the "prefix" field is not read, a file will only be extracted by the basename.
The problem is that with a previous version of the golang archive package headers could be written, that use the prefix field while at the same time setting the header format to GNU. This happens when numeric fields are big enough that they can not be written as octal strings and need to be written in binary. Usually, this shouldn't happen: uid, gid, devmajor, devminor can use up to 7 bytes, size and timestamp can use 11. If one of the records does overflow it switches the whole writer to GNU mode and all next files will be saved in GNU format.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Starting with this commit, integration tests should no longer rely on
the docker cli, they should be API tests instead. For the existing tests
the scripts will use a frozen version of the docker cli with a
DOCKER_API_VERSION frozen to 1.30, which should ensure that the CI remains
green at all times.
To help contributors develop and test manually with a modified docker
cli, this commit also adds a DOCKER_CLI_PATH environment variable to the
Makefile. This allows to set the path of a custom cli that will be
available inside the development container and used to run the
integration tests.
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
This updates the versions of the frozen images used to their current
version. The original reason for updating these images was to make sure
they are not affected by [CVE-2016-1252 / DSA-3733-1](https://lwn.net/Articles/709119/),
which is fixed in apt 1.0.9.8.4 and up.
Note that `CVE-2016-1252` won't affect our test-suite, because no packages
are installed during out tests. It is just "good practice" to keep these
images up to date.
The `debian:jessie`, and `buildpack-deps:jessie` in `Dockerfile.s390x`,
and `Dockerfile.armhf` have not been updated in this patch, because
those images have not yet been updated to contain apt 1.0.9.8.4.
While working on this, the `busybox` and `hello-world` were also updated
to their latest version.
Also removes a reference to `hack/make/.ensure-frozen-images`, which
was removed in ff91276d1f.
The new busybox image has one layer less than the original,
so updated `TestBuildSquashParent` to take that into account.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This drops support for migrations from pre-1.10 Docker versions, which
should be done via an external tool or an intermediate upgrade.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
There is no reason to duplicate efforts and tini is well built and
better than grimes. It is a much stronger option for the default init
and @krallin has done a great job maintaining it and helping make
changes so that it will work with Docker.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
this switches all go download links to use a redirect,
https://golang.org/dl/ instead of the direct source.
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
This means we can vendor libnetwork without special casing, and
it is built the same way as the other external binaries.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
