Adds `DOCKER_BINDDIR_MOUNT_OPTS` to easily tweak the BINDDIR mount
options... primarily adding so I can control the caching mode for
osxfs because compiling takes > 1min for me with the default and < 30s
with both `cached` and `delegated`.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
integration-on-swarm had unnecessary complexity and was too hard to
maintain. Also, it didn't support the new non-CLI integration test suite.
I'm now doing some experiments out of the repo using Kubernetes:
https://github.com/AkihiroSuda/kube-moby-integration
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Not exactly sure why, but this line;
build: DOCKER_CROSS ?= false
Always overwrote `DOCKER_CROSS` when running `make cross`.
Perhaps because it is set in `cross: DOCKER_CROSS := true`,
and in a different scope? May also be dependent on the
version of `make` in use.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Pretty much cross-compile doesn't work because of this:
> profiles/seccomp/seccomp.go:13:2: build constraints exclude all Go files in /go/src/github.com/docker/docker/vendor/github.com/seccomp/libseccomp-golang
This changes adds a new Dockerfile target for cross compilation with the
neccesary arch specific libseccomp packages and CC toolchains.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of having to go through files or registry values as is currently the
case.
While adding GMSA support to Kubernetes (https://github.com/kubernetes/kubernetes/pull/73726)
I stumbled upon the fact that Docker currently only allows passing Windows
credential specs through files or registry values, forcing the Kubelet
to perform a rather awkward dance of writing-then-deleting to either the
disk or the registry to be able to create a Windows container with cred
specs.
This patch solves this problem by making it possible to directly pass
whole base64-encoded cred specs to the engine's API. I took the opportunity
to slightly refactor the method responsible for Windows cred spec as it
seemed hard to read to me.
Added some unit tests on Windows credential specs handling, as there were
previously none.
Added/amended the relevant integration tests.
I have also tested it manually: given a Windows container using a cred spec
that you would normally start with e.g.
```powershell
docker run --rm --security-opt "credentialspec=file://win.json" mcr.microsoft.com/windows/servercore:ltsc2019 nltest /parentdomain
# output:
# my.ad.domain.com. (1)
# The command completed successfully
```
can now equivalently be started with
```powershell
$rawCredSpec = & cat 'C:\ProgramData\docker\credentialspecs\win.json'
$escaped = $rawCredSpec.Replace('"', '\"')
docker run --rm --security-opt "credentialspec=raw://$escaped" mcr.microsoft.com/windows/servercore:ltsc2019 nltest /parentdomain
# same output!
```
I'll do another PR on Swarmkit after this is merged to allow services to use
the same option.
(It's worth noting that @dperny faced the same problem adding GMSA support
to Swarmkit, to which he came up with an interesting solution - see
https://github.com/moby/moby/pull/38632 - but alas these tricks are not
available to the Kubelet.)
Signed-off-by: Jean Rouge <rougej+github@gmail.com>
CIs are assumed to do a git fetch and git merge before running tests.
Therefore, no need for a git fetch inside our validate scripts in CI.
If VALIDATE_ORIGIN_BRANCH is set, then git fetch is skipped and
VALIDATE_ORIGIN_BRANCH is used in validate scripts.
Otherwise, behavior is unchanged.
Signed-off-by: Tibor Vass <tibor@docker.com>
When running CI in other repositories (e.g. Docker's downstream
docker/engine repository), or other branches, the validation
scripts were calculating the list of changes based on the wrong
information.
This lead to weird failures in CI in a branch where these values
were not updated ':-) (CI on a pull request failed because it detected
that new tests were added to the deprecated `integration-cli` test-suite,
but the pull request did not actually make changes in that area).
This patch allows overriding the target repository (and branch)
to compare to (without having to edit the scripts).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
comparing PR commit(s) to HEAD of moby/moby master branch and if founds
new (or renamed) integration tests will run stress tests for them.
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
This patch;
- adds support for multiple newlines
- removes the 1-space indentation of wrapped lines
- allows numerical characters in targets (0-9)
Given these targets:
```Makefile
.PHONY: foobar
foobar: ## runs the foobar lorum ipsum.\nand so pn\nand so on
echo foobar
.PHONY: e2e-tests
e2e-tests: ## runs the end-to-end tests
echo e2e-tests
```
Before this change, the output of `make help` was
```
foobar runs the foobar lorum ipsum.
and so pn\nand so on
```
After this change, the output is:
```
foobar runs the foobar lorum ipsum.
and so pn
and so on
e2e-tests runs the end-to-end tests
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
build the final stage of the Dockerfile (including COPY ...) if no BIND_DIR
is used.
if BIND_DIR is used, build the dev stage, thus skipping the COPY.
Original author: @thaJeztah
Signed-off-by: Mohammad Nasirifar <farnasirim@gmail.com>
Through the env variable of the same name.
The idea here is pretty simple: I/O perf on native mounted disks
on non-Linux (notably Mac OS) is just terrible, thus making it
a real pain to develop: one has to choose between re-building
the image after every single change (eg to run a test) or just
work directly inside the same container (eg with vim, but even then
one would have to re-configure their dev container every time
it gets destroyed - containers, after all, are not supposed to
be long-lived).
Allowing to override `DOCKER_MOUNT` makes it easy for everyone
to decide what their volume/syncing strategy is; for example
one can choose to use [docker-sync](https://github.com/EugenMayer/docker-sync)
This patch won't change anything for anyone who doesn't
set the `DOCKER_MOUNT` env variable in their environment.
Signed-off-by: Jean Rouge <jer329@cornell.edu>
Looks like -i (together with DOCKER_INCREMENTAL_BINARY etc)
were used to get faster incremental builds.
Nowdays (since Go 1.10) this is no longer the case, as
go build cache is used [1]. Here's a quote:
> You do not have to use "go test -i" or "go build -i" or
> "go install" just to get fast incremental builds. We will
> not have to teach new users those workarounds anymore.
> Everything will just be fast.
To enable go cache between builds, add a volume for /root/.cache.
[1] https://groups.google.com/forum/#!msg/golang-dev/qfa3mHN4ZPA/X2UzjNV1BAAJ
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
* Expose license status in Info
This wires up a new field in the Info payload that exposes the license.
For moby this is hardcoded to always report a community edition.
Downstream enterprise dockerd will have additional licensing logic wired
into this function to report details about the current license status.
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Code review comments
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Add windows autogen support
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
This introduces a PRODUCT environment variable that is used to set a constant
at dockerversion.ProductName.
That is then used to set BuildKit's ExportedProduct variable in order to show
useful error messages to users when a certain version of the product doesn't
support a BuildKit feature.
Signed-off-by: Tibor Vass <tibor@docker.com>
`make DOCKERD_ARGS=--init binary run` should start the daemon with
`--init` as flags (with any other "automagically" added ones).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This is left-over stuff from building Docker pacakges. These aren't
really maintained outside of bumping the golang version, and are never
tested.
These builders can be found at
https://github.com/docker/docker-ce-packaging where they are kept up to
date.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This change fixes bug when $CURDIR contains special character e.g. '&&'.
Change-Id: I658eb60d97c2229cf8b4e0a61dbbecf3edc9d573
Signed-off-by: Fangming.Fang <fangming.fang@arm.com>
This change adds a Platform struct with a Name field and a general
Components field to the Version API type. This will allow API
consumers to show version information for the whole platform and
it will allow API providers to set the versions for the various
components of the platform.
All changes here are backwards compatible.
Signed-off-by: Tibor Vass <tibor@docker.com>
Commit dbf580be57 removed
this helper script because it's no longer used in CI.
However, the "make test" target in the Makefile still
called this helper, resulting it to fail.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a way to specify a custom graphdriver priority list
during build. This can be done with something like
go build -ldflags "-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper"
As ldflags are already used by the engine build process, and it seems
that only one (last) `-ldflags` argument is taken into account by go,
an envoronment variable `DOCKER_LDFLAGS` is introduced in order to
be able to append some text to `-ldflags`. With this in place,
using the feature becomes
make DOCKER_LDFLAGS="-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper" dynbinary
The idea behind this is, the priority list might be different
for different distros, so vendors are now able to change it
without patching the source code.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Go has supported PIC builds for a while now, and given the security
benefits of using PIC binaries we should really enable them. There also
appears to be some indication that non-PIC builds have been interacting
oddly on ppc64le (the linker cannot load some shared libraries), and
using PIC builds appears to solve this problem.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
VERSION was hardcoded to be used as the `VERSION` file from the root
directory, this makes it so that you have the option to overwrite this.
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
Failures from the integration suite were not propagating to the outter shell
for some reason. Handle the failure with an if exit 1.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Commit 1fb615599a moved the unit tests out
of `hack/make.sh`, however the Makefile still used the old path, resulting
in a warning being printed when the unit tests were run:
---> Making bundle: test-unit (in bundles/17.06.0-dev/test-unit)
DEPRECATED: use hack/test/unit instead of hack/make.sh test-unit
This patch updates the Makefile to use the new command.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This adds a new package `integration` where `engine` integration tests
should live. Those integration tests should not depends on any `cli`
components (except from the `dockerd` daemon for now — to actually
start a daemon).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Recently, it's become necessary to build dynbinaries on certain
distributions, so make it easier for users to build dynbinaries. It
looks like this was just an oversight when we added dynbinary builds.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Adds an environment variable `DOCKER_BASH_COMPLETION_PATH` that allows to
include a bash completion script into the development container.
This is needid for development of bash completion.
Signed-off-by: Harald Albers <github@albersweb.de>
Starting with this commit, integration tests should no longer rely on
the docker cli, they should be API tests instead. For the existing tests
the scripts will use a frozen version of the docker cli with a
DOCKER_API_VERSION frozen to 1.30, which should ensure that the CI remains
green at all times.
To help contributors develop and test manually with a modified docker
cli, this commit also adds a DOCKER_CLI_PATH environment variable to the
Makefile. This allows to set the path of a custom cli that will be
available inside the development container and used to run the
integration tests.
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
This allows for the default to come from the image/`Dockerfile` rather
than being unconditionally overwritten by the environment.
Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
This commit adds contrib/integration-cli-on-swarm/integration-cli-on-swarm.sh,
which enables IT to be running in parallel, using Swarm-mode and Funker.
Please refer to contrib/integration-cli-on-swarm/README.md
The test takes almost 5 to 6 minutes, with 10 n1-standard-4 GCE instances.
$ ./contrib/integration-cli-on-swarm/integration-cli-on-swarm.sh --push-worker-image example.gcr.io/foo/bar --replicas 30 --shuffle
2016/12/29 08:32:15 Loaded 1618 tests (30 chunks)
2016/12/29 08:32:15 Executing 30 chunks in parallel, against "integration-cli-worker"
2016/12/29 08:32:15 Executing chunk 0 (contains 54 test filters)
..
2016/12/29 08:34:34 Finished chunk 28 [1/30] with 54 test filters in 2m19.098068003s, code=0.
2016/12/29 08:34:38 Finished chunk 12 [2/30] with 54 test filters in 2m23.088569511s, code=0.
2016/12/29 08:34:48 Finished chunk 10 [3/30] with 54 test filters in 2m33.880679079s, code=0.
2016/12/29 08:34:54 Finished chunk 20 [4/30] with 54 test filters in 2m39.973747028s, code=0.
2016/12/29 08:35:11 Finished chunk 18 [5/30] with 54 test filters in 2m56.28384361s, code=0.
2016/12/29 08:35:11 Finished chunk 29 [6/30] with 52 test filters in 2m56.54047088s, code=0.
2016/12/29 08:35:15 Finished chunk 1 [7/30] with 54 test filters in 3m0.285044426s, code=0.
2016/12/29 08:35:22 Finished chunk 6 [8/30] with 54 test filters in 3m7.211775338s, code=0.
2016/12/29 08:35:24 Finished chunk 25 [9/30] with 54 test filters in 3m9.938413009s, code=0.
2016/12/29 08:35:30 Finished chunk 27 [10/30] with 54 test filters in 3m15.219834368s, code=0.
2016/12/29 08:35:36 Finished chunk 9 [11/30] with 54 test filters in 3m21.615434162s, code=0.
2016/12/29 08:35:41 Finished chunk 13 [12/30] with 54 test filters in 3m26.576907401s, code=0.
2016/12/29 08:35:45 Finished chunk 17 [13/30] with 54 test filters in 3m30.290752537s, code=0.
2016/12/29 08:35:53 Finished chunk 2 [14/30] with 54 test filters in 3m38.148423321s, code=0.
2016/12/29 08:35:55 Finished chunk 24 [15/30] with 54 test filters in 3m40.09669137s, code=0.
2016/12/29 08:35:57 Finished chunk 8 [16/30] with 54 test filters in 3m42.299945108s, code=0.
2016/12/29 08:35:57 Finished chunk 22 [17/30] with 54 test filters in 3m42.946558809s, code=0.
2016/12/29 08:35:59 Finished chunk 23 [18/30] with 54 test filters in 3m44.232557165s, code=0.
2016/12/29 08:36:02 Finished chunk 3 [19/30] with 54 test filters in 3m47.112051358s, code=0.
2016/12/29 08:36:11 Finished chunk 15 [20/30] with 54 test filters in 3m56.340656645s, code=0.
2016/12/29 08:36:11 Finished chunk 11 [21/30] with 54 test filters in 3m56.882401231s, code=0.
2016/12/29 08:36:22 Finished chunk 19 [22/30] with 54 test filters in 4m7.551093516s, code=0.
2016/12/29 08:36:23 Finished chunk 21 [23/30] with 54 test filters in 4m8.221093446s, code=0.
2016/12/29 08:36:25 Finished chunk 16 [24/30] with 54 test filters in 4m10.450451705s, code=0.
2016/12/29 08:36:27 Finished chunk 5 [25/30] with 54 test filters in 4m12.162272692s, code=0.
2016/12/29 08:36:28 Finished chunk 14 [26/30] with 54 test filters in 4m13.977801031s, code=0.
2016/12/29 08:36:29 Finished chunk 0 [27/30] with 54 test filters in 4m14.34086812s, code=0.
2016/12/29 08:36:49 Finished chunk 26 [28/30] with 54 test filters in 4m34.437085539s, code=0.
2016/12/29 08:37:14 Finished chunk 7 [29/30] with 54 test filters in 4m59.22902721s, code=0.
2016/12/29 08:37:20 Finished chunk 4 [30/30] with 54 test filters in 5m5.103469214s, code=0.
2016/12/29 08:37:20 Executed 30 chunks in 5m5.104379119s. PASS: 30, FAIL: 0.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
- export DOCKER_GITCOMMIT in the Makefile
- prioritize DOCKER_GITCOMMIT against the `git` command in
`./hack/make.sh`
- Also add `integration-cli/bundles` to gitignore (it's generated when
using integration-cli shell)
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
By using `make CONTAINER_NAME=docker-dev shell` you're able to name
the docker-dev container to what you want. The idea behind it is by
knowing the name of your dev container you can automate easily some
thing using `docker exec` later.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Much easier than the previous method of copying over to the docs
repository and generating the docs.
And, as @cpuguy83 pointed out, that actually didn't work
because the PR that adds Swagger docs isn't merged yet. Oopsy.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
Changes the default when incremental build is enabled to use named
volumes instead of bindmounts.
The reason for this is, on Mac/Windows the bind mounts will go over the
shared fs, which is incredibly slow and itself uses lots of CPU.
Makes an incremental build on OSX go from ~40s to 10s.
To get the old behavior, can set `PKGCACHE_DIR=.go-pkg-cache`
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This make it more consistent with the other image builds and allow to
build manpages behind a proxy for example.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This pins the version of go-swagger used, because
the results generated by different versions
can differ quite a bit (tested between 0.7.2 - 0.7.4),
and can cause CI / validation to fail.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This speeds up docker build time drastically. It still possible to
disable this by setting `DOCKER_INCREMENTAL_BUILD` to `0` (and this is
what should be done on the CI).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Allow each script to run directly without the hack/make.sh wrapper. These
scripts do not produce artifacts and do not benefit from the "bundles"
framework.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
The bundles dir used to be automatically made a volume if BINDDIR was
not set.
The `PKGCACHE_*` stuff was added recently which gets added to
`DOCKER_MOUNT`. This breaks the case where someone wants to use the
package cache (DOCKER_INCREMENTAL_BINARY is enabled) but has not
specified a bind dir but needs the bundles dir made a volume.
Moves the package cache handling to after the bundles dir is made a
volume.
Before this patch, if you have `DOCKER_GRAPHDRIVER` set to anything
other than vfs, the tests in the daemon suite (and swarm suite, and
registry suite) will fail to run since they cannot startup a daemon
(except for speicifc cases where <driver> over <driver> is possible,
like devmapper).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Passses down BUILD_APT_MIRROR to the docker env.
Ensures BUILD_APT_MIRROR is used when building debs, but only when the
consuming `Dockerfile` actually uses it, otherwise it will cause the
build to fail (e.g. on Ubuntu builds we aren't using APT_MIRROR).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Adds a new bundle `verify-integration-tests` which pre-compiles a test
binary for the integration tests.
This makes sure that the integration tests will actually compile before
doing other tasks which take much longer, such as building dockerd and
loading test fixtures.
When it comes time to actually run the tests, the pre-compiled binary
will be used so it doesn't have to compile the tests a 2nd time.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
DOCKER_BUILD_APT_MIRROR
This allows passthrough of other options like HTTP_PROXY from
DOCKER_BUILD_ARGS to work on all builds (for example,
ensure-syscall-test) when APT_MIRROR is set.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
When running a `make manpages` the image `docker-manpage-dev` will be build and started to create
the man pages. But the container will not be deleted afterwards. So I propose to start it with
`docker run --rm ...`, otherwise we'll collect some trash.
```
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e3d9f721d862 docker-manpage-dev "man/generate.sh" 19 minutes ago Exited (0) 16 minutes ago berserk_jang
5738a23fb682 docker-manpage-dev "man/generate.sh" 9 hours ago Exited (0) 9 hours ago boring_bartik
15490b5e63f7 docker-manpage-dev "man/generate.sh" 10 hours ago Exited (0) 10 hours ago lonely_joliot
```
Signed-off-by: Dieter Reuter <dieter.reuter@me.com>
`make run` allows you to fire up a daemon (in a container) just using
the existing built binaries. This allows for more rapid iteration
instead of dealing with firing up a shell just to start the daemon.
By default the daemon will listen on port 2375 on the default network
interface.
If a port forward is required to access the daemon, the user can set
`make DOCKER_PORT=2375 run` to get a port forward on a random port with
the daemon listening on port 2375, or `make DOCKER_PORT=2375:2375 run`
to get a daemon running with port 2375 forwarded to the daemon running
on 2375.
Note that the daemon is automatically configured to listen on whatever
port is set for the container side of the `DOCKER_PORT` port spec.
When running on docker4mac, the user must do the following:
```
$ make BINDDIR=. DOCKER_PORT=2375 run
```
This makes sure the binaries are loaded in the container and a port is
forwarded, since it is currently impossible to route traffic from the
mac directly to a container IP.
To get a fresh binary:
```
$ make BINDDIR=. DOCKER_PORT=2375 binary run
```
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This installs docker and dockerd to `$DOCKER_MAKE_INSTALL_PREFIX/bin`, which
defaults to `/usr/local/bin`
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a proxy to support 'docker daemon'
Fix configFile option, and remove a test that is no longer relevant.
Remove daemon build tag.
Remove DOCKER_CLIENTONLY from build scripts.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Change docker-daemon to dockerd.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Instead of being split between three files, let's let `hack/make/.detect-daemon-osarch` be our single source of truth for multiarch detection/vars. Not only does it make it slightly easier to make sure we change everything properly when these bits have to change, but it also makes it so that all bits of `hack/make.sh` (especially `hack/make/.ensure-frozen-images`) work properly outside the context of the `Makefile` on all platforms.
Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
This allows the test suite to be able to run without worrying about
the underlying fs used by the container running the daemon (e.g.
aufs-on-aufs), so long as the host running the container is running a
supported fs.
The volume will be cleaned up when the container is removed due to
`--rm`.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
