Windows still writes to the autogen directory, but the source code is
mounted in as read-only.
In order to do enable this without taking a massive hit in doing an rw
mount (for the source code) we mount a tmpfs into the build at the
autogen dir.
In order for this to work the directory must alreay exist, so we create
it before entering the build.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This introduces a FORCE_VALIDATE environment variable, which allows
forcing some validation steps, even if no changes were detected.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The binary targets now use buildkit to build/output binaries instead of
doing it in a DOCKER_RUN_DOCKER container. With that change caused
issues when trying to call multiple make targets such as `make binary
cross` since those targets are updating the variables (with conflicting
data) used by the shared `build` prerequisite.
This change has those binary output targets call `docker build` (or
`buildx build`) directly since that is the action they are preforming
and no longer have any pre-reqs.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Un-indent the comment, so that it doesn't get printed by
the shell script (moved it above the target, as it looked
slightly less cluttered)
Also fixed the "help" comment, so that it shows up in
`make help`, and removed the un-needed dummy `buildx:` target.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This simplifies the makefile a bit, while preserving
the functionality. Using a non-existing Dockerfile
to demonstrate:
make buildx
Successfully tagged moby-buildx:latest
92059305df7371f8b5b3638d4d405d49ff909031a7bc6d2f515cb0a0df03c2f4
github.com/docker/buildx v0.3.0 c967f1d
make BUILDX_DOCKERFILE=foo buildx
BUILDX_DOCKERFILE=foo buildx
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /Users/sebastiaan/go/src/github.com/docker/docker/foo: no such file or directory
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This patch removes the `BUILDX_COMMIT` make variable. With the
make variable removed, it no longer "masks" environment variables,
and there is no longer a need to export the variable.
A side effect of this change, is that (by default), the buildx
image is tagged as `moby-buildx:latest`. This likely isn't a
problem, because the build-cache would still be preserved in
intermediate images. Having the image tagged as `:latest` also
makes cleaning up easier (without having to remove the image
for each version tagged.
Otherwise, the behavior remains the same as before:
# default
rm -f bundles/buildx && make buildx
# => => naming to docker.io/library/moby-buildx:latest
github.com/docker/buildx v0.3.0 c967f1d
# using a make variable:
rm -f bundles/buildx && make BUILDX_COMMIT=v0.2.1 buildx
# => => naming to docker.io/library/moby-buildx:v0.2.1
github.com/docker/buildx v0.2.1 0eb2df5
# using an environment variable:
rm -f bundles/buildx && BUILDX_COMMIT=v0.2.2 make buildx
# => => naming to docker.io/library/moby-buildx:v0.2.2
github.com/docker/buildx v0.2.2 ab5fe3d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `BUILDX_COMMIT` variable was set as a Make variable,
which isn't exported, and thus not available in scripts,
unless referenced through `$(VAR)` (non-curly-brackets).
As a result `--build-arg BUILDX_COMMIT` did not set the
`BUILDX_COMMIT` build-arg, and the default from the Dockerfile
(`master`) was used instead.
This patch exports the default version that's set in the
Makefile, so that it can be used as a regular environment
variable. The script was also slighly modified to no longer
use the `Make` variable.
In addition, the `buildx` target now calls `buildx version`,
which is useful to confirm if the binary was successfully built
(and with the correct version).
Before:
rm -f bundles/buildx && make buildx && ./bundles/buildx version
# => => naming to docker.io/library/moby-buildx:v0.3.0
github.com/docker/buildx v0.3.1 6db68d0
# using a make variable:
rm -f bundles/buildx && make BUILDX_COMMIT=v0.2.1 buildx && ./bundles/buildx version
# => => naming to docker.io/library/moby-buildx:v0.2.1
github.com/docker/buildx v0.3.1 6db68d0
# using an environment variable:
rm -f bundles/buildx && BUILDX_COMMIT=v0.2.2 make buildx && ./bundles/buildx version
# => => naming to docker.io/library/moby-buildx:v0.2.2
github.com/docker/buildx v0.3.1 6db68d0
After:
# default
rm -f bundles/buildx && make buildx
# => => naming to docker.io/library/moby-buildx:v0.3.0
github.com/docker/buildx v0.3.0 c967f1d
# using a make variable:
rm -f bundles/buildx && make BUILDX_COMMIT=v0.2.1 buildx
# => => naming to docker.io/library/moby-buildx:v0.2.1
github.com/docker/buildx v0.2.1 0eb2df5
# using an environment variable:
rm -f bundles/buildx && BUILDX_COMMIT=v0.2.2 make buildx
# => => naming to docker.io/library/moby-buildx:v0.2.2
github.com/docker/buildx v0.2.2 ab5fe3d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was inadvertently removed when updating the Dockerfile for buildkit
specific features.
Trick selects a different build target depending on if the source is
going to be bind-mounted in anyway, which prevents the need to copy the
whole source tree to the builder.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Since the dockerfile now requires buildkit, let's just use buildx can
bootstrap itself into even an old version of Docker which does not
support buildkit.
This also decouples the Dockerfile/build from the version of Docker
which is installed.
One major downside:
If buildx needs to setup a container driver (ie, docker does not support
buildkit), the `make shell` target (and others which call
`DOCKER_RUN_DOCKER`) must export the image from buildkit and into
docker. This added an extra 70s to a full build for me (agan only for targets
which call `DOCKER_RUN_DOCKER`) and 40s on a rebuild (with no changes).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
'Namespace' parallel runs by bind-mounting a different directory
in the container, instead of making the tests running inside
the container aware of the namespaced location.
This makes it transparent to the tests, and slightly reduces
complexity.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is basically taking some stuff that make a custom shell function
for.
This takes a test filter, builds the appropriate TESTFLAGS, and sets the
integration API test dirs that match the given filter to avoid building
all test dirs.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
1. Use `go list` to get list of integration dirs to build. This means we
do not need to have a valid `.go` in every subdirectory and also
filters out other dirs like "bundles" which may have been created.
2. Add option to specify custom flags for integration and
integration-cli. This is needed so both suites can be run AND set
custom flags... since the cli suite does not support standard go
flags.
3. Add options to skip an entire integration suite.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This allows overriding the version of Go without making modifications in the
source code, which can be useful to test against multiple versions.
For example:
make GO_VERSION=1.13beta1 shell
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Adds `DOCKER_BINDDIR_MOUNT_OPTS` to easily tweak the BINDDIR mount
options... primarily adding so I can control the caching mode for
osxfs because compiling takes > 1min for me with the default and < 30s
with both `cached` and `delegated`.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
integration-on-swarm had unnecessary complexity and was too hard to
maintain. Also, it didn't support the new non-CLI integration test suite.
I'm now doing some experiments out of the repo using Kubernetes:
https://github.com/AkihiroSuda/kube-moby-integration
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Not exactly sure why, but this line;
build: DOCKER_CROSS ?= false
Always overwrote `DOCKER_CROSS` when running `make cross`.
Perhaps because it is set in `cross: DOCKER_CROSS := true`,
and in a different scope? May also be dependent on the
version of `make` in use.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Pretty much cross-compile doesn't work because of this:
> profiles/seccomp/seccomp.go:13:2: build constraints exclude all Go files in /go/src/github.com/docker/docker/vendor/github.com/seccomp/libseccomp-golang
This changes adds a new Dockerfile target for cross compilation with the
neccesary arch specific libseccomp packages and CC toolchains.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of having to go through files or registry values as is currently the
case.
While adding GMSA support to Kubernetes (https://github.com/kubernetes/kubernetes/pull/73726)
I stumbled upon the fact that Docker currently only allows passing Windows
credential specs through files or registry values, forcing the Kubelet
to perform a rather awkward dance of writing-then-deleting to either the
disk or the registry to be able to create a Windows container with cred
specs.
This patch solves this problem by making it possible to directly pass
whole base64-encoded cred specs to the engine's API. I took the opportunity
to slightly refactor the method responsible for Windows cred spec as it
seemed hard to read to me.
Added some unit tests on Windows credential specs handling, as there were
previously none.
Added/amended the relevant integration tests.
I have also tested it manually: given a Windows container using a cred spec
that you would normally start with e.g.
```powershell
docker run --rm --security-opt "credentialspec=file://win.json" mcr.microsoft.com/windows/servercore:ltsc2019 nltest /parentdomain
# output:
# my.ad.domain.com. (1)
# The command completed successfully
```
can now equivalently be started with
```powershell
$rawCredSpec = & cat 'C:\ProgramData\docker\credentialspecs\win.json'
$escaped = $rawCredSpec.Replace('"', '\"')
docker run --rm --security-opt "credentialspec=raw://$escaped" mcr.microsoft.com/windows/servercore:ltsc2019 nltest /parentdomain
# same output!
```
I'll do another PR on Swarmkit after this is merged to allow services to use
the same option.
(It's worth noting that @dperny faced the same problem adding GMSA support
to Swarmkit, to which he came up with an interesting solution - see
https://github.com/moby/moby/pull/38632 - but alas these tricks are not
available to the Kubelet.)
Signed-off-by: Jean Rouge <rougej+github@gmail.com>
CIs are assumed to do a git fetch and git merge before running tests.
Therefore, no need for a git fetch inside our validate scripts in CI.
If VALIDATE_ORIGIN_BRANCH is set, then git fetch is skipped and
VALIDATE_ORIGIN_BRANCH is used in validate scripts.
Otherwise, behavior is unchanged.
Signed-off-by: Tibor Vass <tibor@docker.com>
When running CI in other repositories (e.g. Docker's downstream
docker/engine repository), or other branches, the validation
scripts were calculating the list of changes based on the wrong
information.
This lead to weird failures in CI in a branch where these values
were not updated ':-) (CI on a pull request failed because it detected
that new tests were added to the deprecated `integration-cli` test-suite,
but the pull request did not actually make changes in that area).
This patch allows overriding the target repository (and branch)
to compare to (without having to edit the scripts).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
comparing PR commit(s) to HEAD of moby/moby master branch and if founds
new (or renamed) integration tests will run stress tests for them.
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
This patch;
- adds support for multiple newlines
- removes the 1-space indentation of wrapped lines
- allows numerical characters in targets (0-9)
Given these targets:
```Makefile
.PHONY: foobar
foobar: ## runs the foobar lorum ipsum.\nand so pn\nand so on
echo foobar
.PHONY: e2e-tests
e2e-tests: ## runs the end-to-end tests
echo e2e-tests
```
Before this change, the output of `make help` was
```
foobar runs the foobar lorum ipsum.
and so pn\nand so on
```
After this change, the output is:
```
foobar runs the foobar lorum ipsum.
and so pn
and so on
e2e-tests runs the end-to-end tests
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
build the final stage of the Dockerfile (including COPY ...) if no BIND_DIR
is used.
if BIND_DIR is used, build the dev stage, thus skipping the COPY.
Original author: @thaJeztah
Signed-off-by: Mohammad Nasirifar <farnasirim@gmail.com>
Through the env variable of the same name.
The idea here is pretty simple: I/O perf on native mounted disks
on non-Linux (notably Mac OS) is just terrible, thus making it
a real pain to develop: one has to choose between re-building
the image after every single change (eg to run a test) or just
work directly inside the same container (eg with vim, but even then
one would have to re-configure their dev container every time
it gets destroyed - containers, after all, are not supposed to
be long-lived).
Allowing to override `DOCKER_MOUNT` makes it easy for everyone
to decide what their volume/syncing strategy is; for example
one can choose to use [docker-sync](https://github.com/EugenMayer/docker-sync)
This patch won't change anything for anyone who doesn't
set the `DOCKER_MOUNT` env variable in their environment.
Signed-off-by: Jean Rouge <jer329@cornell.edu>
Looks like -i (together with DOCKER_INCREMENTAL_BINARY etc)
were used to get faster incremental builds.
Nowdays (since Go 1.10) this is no longer the case, as
go build cache is used [1]. Here's a quote:
> You do not have to use "go test -i" or "go build -i" or
> "go install" just to get fast incremental builds. We will
> not have to teach new users those workarounds anymore.
> Everything will just be fast.
To enable go cache between builds, add a volume for /root/.cache.
[1] https://groups.google.com/forum/#!msg/golang-dev/qfa3mHN4ZPA/X2UzjNV1BAAJ
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
* Expose license status in Info
This wires up a new field in the Info payload that exposes the license.
For moby this is hardcoded to always report a community edition.
Downstream enterprise dockerd will have additional licensing logic wired
into this function to report details about the current license status.
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Code review comments
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Add windows autogen support
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
This introduces a PRODUCT environment variable that is used to set a constant
at dockerversion.ProductName.
That is then used to set BuildKit's ExportedProduct variable in order to show
useful error messages to users when a certain version of the product doesn't
support a BuildKit feature.
Signed-off-by: Tibor Vass <tibor@docker.com>
`make DOCKERD_ARGS=--init binary run` should start the daemon with
`--init` as flags (with any other "automagically" added ones).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This is left-over stuff from building Docker pacakges. These aren't
really maintained outside of bumping the golang version, and are never
tested.
These builders can be found at
https://github.com/docker/docker-ce-packaging where they are kept up to
date.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This change fixes bug when $CURDIR contains special character e.g. '&&'.
Change-Id: I658eb60d97c2229cf8b4e0a61dbbecf3edc9d573
Signed-off-by: Fangming.Fang <fangming.fang@arm.com>
This change adds a Platform struct with a Name field and a general
Components field to the Version API type. This will allow API
consumers to show version information for the whole platform and
it will allow API providers to set the versions for the various
components of the platform.
All changes here are backwards compatible.
Signed-off-by: Tibor Vass <tibor@docker.com>
Commit dbf580be57 removed
this helper script because it's no longer used in CI.
However, the "make test" target in the Makefile still
called this helper, resulting it to fail.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a way to specify a custom graphdriver priority list
during build. This can be done with something like
go build -ldflags "-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper"
As ldflags are already used by the engine build process, and it seems
that only one (last) `-ldflags` argument is taken into account by go,
an envoronment variable `DOCKER_LDFLAGS` is introduced in order to
be able to append some text to `-ldflags`. With this in place,
using the feature becomes
make DOCKER_LDFLAGS="-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper" dynbinary
The idea behind this is, the priority list might be different
for different distros, so vendors are now able to change it
without patching the source code.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Go has supported PIC builds for a while now, and given the security
benefits of using PIC binaries we should really enable them. There also
appears to be some indication that non-PIC builds have been interacting
oddly on ppc64le (the linker cannot load some shared libraries), and
using PIC builds appears to solve this problem.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
VERSION was hardcoded to be used as the `VERSION` file from the root
directory, this makes it so that you have the option to overwrite this.
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
Failures from the integration suite were not propagating to the outter shell
for some reason. Handle the failure with an if exit 1.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
