These scripts explicitly use Bash, so we should be able to use
`[[` instead of `[` (which seems to be recommended).
Also added curly brackets to some bare variables, and quoted some paths.
This makes my IDE a bit more silent :-)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I noticed this error in CI:
```
20:37:25 INFO: Non-base image count on control daemon to delete is 9
20:37:25 "docker rmi" requires at least 1 argument.
20:37:25 See 'docker rmi --help'.
20:37:25
20:37:25 Usage: docker rmi [OPTIONS] IMAGE [IMAGE...]
20:37:25
20:37:25 Remove one or more images
```
Which indicated that the PowerShell script managed to find images to delete, but
not actually passing the images to `docker rmi`.
The reason for this failing was that the script attempted to convert the
collection/array to a string, which produces;
```powershell
Write-Output $(docker images --format "{{.Repository}}:{{.ID}}" | `
>> select-string -NotMatch "windowsservercore" | `
>> select-string -NotMatch "nanoserver" | `
>> select-string -NotMatch "docker" `
>> ).ToString()
System.Object[]
```
Which, when trying to split by the chosen separator (`:`), will return the same;
```powershell
Write-Output "System.Object[]".Split(":")[0]
```
This patch:
- Adds an intermediate variable (`$allImages`) to make the code better readable
- Switches the separator to `#`, to prevent breaking on images pulled from a
repository with a port in its name (`myregistry:5000/my/image`)
- Switches to use a comma-separated list for `-NotMatch` (for readability)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: John Howard <jhoward@microsoft.com>
This is a follow-on from https://github.com/moby/moby/pull/38277
but had to be done in a couple of stages to ensure that CI didn't
break. v1.1 of the busybox image is now based on a CMD of "sh"
rather than using an entrypoint. And it also uses the bin directory
rather than `c:\busybox`. This makes it look a lot closer to the
Linux busybox image, and means that a couple of Windows-isms in
CI tests can be reverted back to be identical to their Linux
equivalents.
The previous update used a commit from master. Now that
all the fixes are backported to the containerd 1.2 release
branch, we can switch back to that branch.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
release notes: https://github.com/containerd/containerd/releases/tag/v1.2.0
- New V2 Runtime with a stable gRPC interface for managing containers through
external shims.
- Updated CRI Plugin, validated against Kubernetes v1.11 and v1.12, but it is
also compatible with Kubernetes v1.10.
- Support for Kubernetes Runtime Class, introduced in Kubernetes 1.12
- A new proxy plugin configuration has been added to allow external
snapshotters be connected to containerd using gRPC.-
- A new Install method on the containerd client allows users to publish host
level binaries using standard container build tooling and container
distribution tooling to download containerd related binaries on their systems.
- Add support for cleaning up leases and content ingests to garbage collections.
- Improved multi-arch image support using more precise matching and ranking
- Added a runtime `options` field for shim v2 runtime. Use the `options` field to
config runtime specific options, e.g. `NoPivotRoot` and `SystemdCgroup` for
runtime type `io.containerd.runc.v1`.
- Some Minor API additions
- Add `ListStream` method to containers API. This allows listing a larger
number of containers without hitting message size limts.
- Add `Sync` flag to `Delete` in leases API. Setting this option will ensure
a garbage collection completes before the removal call is returned. This can
be used to guarantee unreferenced objects are removed from disk after a lease.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Containerd should be "leading" when specifying which version of runc to use.
From the RUNC.MD document in the containerd repository
(https://github.com/containerd/containerd/blob/b1e202c32724e82779544365528a1a082
b335553/RUNC.md);
> We depend on a specific runc version when dealing with advanced features. You
> should have a specific runc build for development. The current supported runc
> commit is described in vendor.conf. Please refer to the line that starts with
> github.com/opencontainers/runc.
This patch adds a note to vendor.conf and runc.installer to describe the order
in which runc should be updated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
No code changes, but this aligns it to a tagged version,
and updates some nested license files to MIT.
vndr doesn't vendor those nested files, so no code changes in
the vendor directory.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case we're running on RHEL7 kernel, which has non-working
and broken kernel memory controller, add 'nokmem' build tag
so that runc never enables kmem accounting.
For more info, see the following runc commit:
https://github.com/opencontainers/runc/commit/6a2c1559684
This behavior can be overriden by having `RUNC_NOKMEM` environment
variable set (e.g. to empty value to disable setting nokmem).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
- DSR support for linux
- increase max DNS pending query from 100->1024
- DNs to handle NXDOMAIN, REFUSED
diff:
20461b8539...d7b61745d1
Signed-off-by: Flavio Crisciani <flavio.crisciani@docker.com>
With containerd reaching 1.0, the runtime now
has a stable API, so there's no need to do a check
if the installed version matches the expected version.
Current versions of Docker now also package containerd
and runc separately, and can be _updated_ separately.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Add windows CI entrypoint script.
Signed-off-by: John Howard <jhoward@microsoft.com>
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Looks like -i (together with DOCKER_INCREMENTAL_BINARY etc)
were used to get faster incremental builds.
Nowdays (since Go 1.10) this is no longer the case, as
go build cache is used [1]. Here's a quote:
> You do not have to use "go test -i" or "go build -i" or
> "go install" just to get fast incremental builds. We will
> not have to teach new users those workarounds anymore.
> Everything will just be fast.
To enable go cache between builds, add a volume for /root/.cache.
[1] https://groups.google.com/forum/#!msg/golang-dev/qfa3mHN4ZPA/X2UzjNV1BAAJ
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
We can do that now as we're no longer carrying archive/tar.
Note that latest vndr removes vendor/ subdir so we don't have to,
thus the change in hack/validate/vendor.
While at it, re-run a new vndr version to make sure everything
that should be there is.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It's that time of year again! Go 1.11 is released, time to use it.
This commit also
* removes our archive/tar fork, since upstream archive/tar
is fixed for static builds, and osusergo build tag is set.
* removes ENV GO_VERSION from Dockerfile as it's not needed
anymore since PR #37592 is merged.
[v2: switch to beta2]
[v3: switch to beta3]
[v4: rc1]
[v5: remove ENV GO_VERSION as PR #37592 is now merged]
[v6: rc2]
[v7: final!]
[v8: use 1.11.0]
[v9: back to 1.11]
[v8: use 1.11.0]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
We would like to use a version with .0 suffix (like 1.11.0) in
Dockerfile, so that once a .1 version is out (like 1.11.1) we
won't accidentally switch to it.
Unfortunately it's not possible to use .0 suffix currently
as it breaks the check in make.ps1. This patch fixes that.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case of an exception, it makes great sense to print out some
information telling where exactly it happened.
_.InvocationInfo.PositionMessage gives script name, line number,
character position and (depending on the PS version) highlights
the part where error has happened.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
* Expose license status in Info
This wires up a new field in the Info payload that exposes the license.
For moby this is hardcoded to always report a community edition.
Downstream enterprise dockerd will have additional licensing logic wired
into this function to report details about the current license status.
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Code review comments
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Add windows autogen support
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
Modify hack/make.ps1 to use the version value used in
"FROM golang" statement.
While at it:
1. Make search expression a bit more strict (use ^ to match at BOL only).
2. Simplify by removing Get-Contents as Select-String can read files.
After this, ENV GO_VERSION can be removed from Dockerfile.
Unfortunately it can't be done in one commit as Windows CI
fails (presumably because Dockerfile is being modified in
place).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.
NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.
The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>
On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.
Signed-off-by: Salahuddin Khan <salah@docker.com>
Go 1.11 includes a fix to os/user to be working in a static binary
(fixing https://github.com/golang/go/issues/23265). The fix requires
`osusergo` build tag to be set for static binaries, which is what
this commit adds (also for containerd).
[v2: sort build tags alphabetically]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This introduces a PRODUCT environment variable that is used to set a constant
at dockerversion.ProductName.
That is then used to set BuildKit's ExportedProduct variable in order to show
useful error messages to users when a certain version of the product doesn't
support a BuildKit feature.
Signed-off-by: Tibor Vass <tibor@docker.com>
Bump libnetwork to 3ac297bc7fd0afec9051bbb47024c9bc1d75bf5b in order to
get fix 0c3d9f00 which addresses a flaw that the scalable load balancing
code revealed. Attempting to print sandbox IDs where the sandbox name
was too short results in a goroutine panic. This can occur with
sandboxes with names of 1 or 2 characters in the previous code. But due
to naming updates in the scalable load balancing code, it could now
occur for networks whose name was 3 characters and at least one of the
integration tests employed such networks (named 'foo', 'bar' and 'baz').
This update also brings in several changes as well:
* 6c7c6017 - Fix error handling about bridgeSetup
* 5ed38221 - Optimize networkDB queue
* cfa9afdb - ndots: produce error on negative numbers
* 5586e226 - improve error message for invalid ndots number
* 449672e5 - Allows to set generic knobs on the Sandbox
* 6b4c4af7 - do not ignore user-provided "ndots:0" option
* 843a0e42 - Adjust corner case for reconnect logic
Signed-off-by: Chris Telfer <ctelfer@docker.com>
Bump libnetwork to b0186632522c68f4e1222c4f6d7dbe518882024f. This
includes the following changes:
* Dockerize protocol buffer generation and update (78d9390a..e12dd44c)
* Use new plugin interfaces provided by plugin pkg (be94e134)
* Improve linux load-balancing scalability (5111c24e..366b9110)
Signed-off-by: Chris Telfer <ctelfer@docker.com>
Bump libnetwork to 430c00a6a6b3dfdd774f21e1abd4ad6b0216c629. This
includes the following moby-affecting changes:
* Update vendoring for go-sockaddr (8df9f31a)
* Fix inconsistent subnet allocation by preventing allocation of
overlapping subnets (8579c5d2)
* Handle IPv6 literals correctly in port bindings (474fcaf4)
* Update vendoring for miekg/dns (8f307ac8)
* Avoid subnet reallocation until required (9756ff7ed)
* Bump libnetwork build to use go version 1.10.2 (603d2c1a)
* Unwrap error type returned by PluginGetter (aacec8e1)
* Update vendored components to match moby (d768021dd)
* Add retry field to cluster-peers probe (dbbd06a7)
* Fix net driver response loss on createEndpoint (1ab6e506)
(fixes https://github.com/docker/for-linux/issues/348)
Signed-off-by: Chris Telfer <ctelfer@docker.com>
`make DOCKERD_ARGS=--init binary run` should start the daemon with
`--init` as flags (with any other "automagically" added ones).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
`arm64` needs get more time duration for the test to finish.
`pty.Start()` opens a file, so the caller should close it explicitly,
else the file I/O can result in unexpected data synchronization issue.
All those changes will not affect the test itself.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
In Go 1.10.1, the Deps for pkg\tarsum\tarsum_test.go are empty ([]) and
the PowerShell script ends up setting its import list to a string value
of False instead of an empty array. This can be remedied by forcing the
result to be an array, by concatenating to an empty array (@() + ...)
Signed-off-by: John Stephens <johnstep@docker.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ec3f9230d70506c536a24e844da0f0b3af9b43f6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* libnetwork#2121: Retry other external DNS servers on ServFail
* libnetwork#2125: Fix README flag and expose orphan network peers
* libnetwork#2126: Adding goreport card
* libnetwork#2130: Modify awk to use cut in check_ip_overlap
* libnetwork#2117: [Carry 1534] Improve scalabiltiy of bridge network isolation rules
Full changes: 2bf63300c5...5c1218c956
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Update libnetwork to 1b91bc94094ecfdae41daa465cc0c8df37dfb3dd to bring in a fix
for stale HNS endpoints on Windows:
When Windows Server 2016 is restarted with the Docker service running, it is
possible for endpoints to be deleted from the libnetwork store without being
deleted from HNS. This does not occur if the Docker service is stopped cleanly
first, or forcibly terminated (since the endpoints still exist in both). This
change works around the issue by removing any stale HNS endpoints for a network
when creating it.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This updates libnetwork to 8892d7537c67232591f1f3af60587e3e77e61d41 to bring in
IPAM fixes for duplicate IP addresses.
- IPAM tests (libnetwork PR 2104) (no changes in vendored files)
- Fix for Duplicate IP issues (libnetwork PR 2105)
Also bump golang/x/sync to match libnetwork (no code-changes, other
than the README being updated)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Since now we have only one Dockerfile, so the arch-specific suffix
of the Dockerfile is not needed anymore.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Release notes: https://github.com/opencontainers/runc/releases/tag/v1.0.0-rc5
Possibly relevant changes included:
- chroot when no mount namespaces is provided
- fix systemd slice expansion so that it could be consumed by cAdvisor
- libcontainer/capabilities_linux: Drop os.Getpid() call
- Update console dependency to fix runc exec on BE (causing: `container_linux.go:265: starting container process caused "open /dev/pts/4294967296: no such file or directory"`)
- libcontainer: setupUserNamespace is always called (fixes: Devices are mounted with wrong uid/gid)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Originally I worked on this for the multi-stage build Dockerfile
changes. Decided to split this out as we are still waiting for
multi-stage to be available on CI and rebasing these is pretty annoying.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The BurntSushi/toml code is now re-licensed as MIT. While
the vendored package was already updated, the tomlv binary
used was still using the old license type.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
dm_task_deferred_remove is not supported by all distributions, due to
out-dated versions of devicemapper. However, in the case where the
devicemapper library was updated without rebuilding Docker (which can
happen in some distributions) then we should attempt to dynamically load
the relevant object rather than try to link to it.
This can only be done if Docker was built dynamically, for obvious
reasons.
In order to avoid having issues arise when dlsym(3) was unnecessary,
gate the whole dlsym(3) logic behind a buildflag that we disable by
default (libdm_dlsym_deferred_remove).
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This fix is related to 36219
This fix updates runc to:
```
-RUNC_COMMIT=9f9c96235cc97674e935002fc3d78361b696a69e
+RUNC_COMMIT=6c55f98695e902427906eed2c799e566e3d3dfb5
-github.com/opencontainers/runc 9f9c96235cc97674e935002fc3d78361b696a69e
+github.com/opencontainers/runc 6c55f98695e902427906eed2c799e566e3d3dfb5
```
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Both names have no real sense, but one allows to make sure these packages
aren't used outside of `integration`.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Previously adding files to vendor/ without adding to vendor.conf would not fail the
validation.
Also be consistent with indentation and use tabs.
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
1. Add = between the option and the argument, otherwise the argument
appears to be passed on to the linters directly, as in:
> DEBUG: [golint.8]: executing /home/kir/go/bin/golint
> -min_confidence 0.800000 ./10m ./api ./api/errdefs <...>
2. Fix setting the default for GOMETALINTER_OPTS -- the default
was -deadline (rather than --deadline).
Fixes: b96093fa56 ("gometalinter: add per-platform configurable options")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Because we merge master into the branch before running tests, so the
actual git sha does not exist on any git remote.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
We don't need the test image namespace anymore since we've already
upgrade those images to the latest multi-arch ones.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
When compiling containerd binaries statically, linker rightfully
complains:
+ make BUILDTAGS=static_build 'EXTRA_FLAGS=-buildmode pie' 'EXTRA_LDFLAGS=-extldflags "-fno-PIC -static"'
🇩 bin/ctr
# github.com/containerd/containerd/cmd/ctr
/tmp/go-link-343047789/000000.o: In function `_cgo_b0c710f30cfd_C2func_getaddrinfo':
/tmp/go-build/net/_obj/cgo-gcc-prolog:46: warning: Using 'getaddrinfo'
in statically linked applications requires at runtime the shared
libraries from the glibc version used for linking
The same error appears for ctr, containerd, and containerd-stress
binaries.
The fix is to use Go's own DNS resolver functions, rather than
glibc's getaddrinfo() -- this option is turned on by `netgo` build tag.
See https://golang.org/pkg/net/ (look for "Name Resolution") for more
details.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Currently we only support 'application/vnd.docker.distribution.manifest.v2+json'
manifest images download, with more multi-arch images used, we need to support
download images with 'application/vnd.docker.distribution.manifest.list.v2+json'
format(aka "fat manifest"), else we will fail to download those multi-arch ones.
This PR adds 'application/vnd.docker.distribution.manifest.list.v2+json' manifest
support, thus we can download both multi-arch and legacy images.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
The `docker info` command compares the installed version
of containerd using a Git-sha. We currently use a tag for
this, but that tag is not returned by the version-API of
containerd, resulting in the `docker info` output to show:
containerd version: 89623f28b87a6004d4b785663257362d1658a729 (expected: v1.0.0)
This patch changes the `v1.0.0` tag to the commit that
corresponds with the tag, so that the `docker info` output
does not show the `expected:` string.
This should be considered a temporary workaround; the check
for the exact version of containerd that's installed was needed
when we still used the 0.2.x branch, because it did not have
stable releases yet.
With containerd reaching 1.0, and using SemVer, we can likely
do a comparison for "Major" version, or make this a "packaging"
issue, and remove the check entirely (we can still _print_ the
version that's installed if we think it's usefule).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change adds a Platform struct with a Name field and a general
Components field to the Version API type. This will allow API
consumers to show version information for the whole platform and
it will allow API providers to set the versions for the various
components of the platform.
All changes here are backwards compatible.
Signed-off-by: Tibor Vass <tibor@docker.com>
Add a way to specify a custom graphdriver priority list
during build. This can be done with something like
go build -ldflags "-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper"
As ldflags are already used by the engine build process, and it seems
that only one (last) `-ldflags` argument is taken into account by go,
an envoronment variable `DOCKER_LDFLAGS` is introduced in order to
be able to append some text to `-ldflags`. With this in place,
using the feature becomes
make DOCKER_LDFLAGS="-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper" dynbinary
The idea behind this is, the priority list might be different
for different distros, so vendors are now able to change it
without patching the source code.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Go has supported PIC builds for a while now, and given the security
benefits of using PIC binaries we should really enable them. There also
appears to be some indication that non-PIC builds have been interacting
oddly on ppc64le (the linker cannot load some shared libraries), and
using PIC builds appears to solve this problem.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Remove Dockerfile.solaris reference in `hack/make/.detect-daemon-osarch`
as `Dockerfile.solaris` has been removed.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
The `.integration-test-helpers` script was sourced by `/etc/bash/bash.rc`.
However, the `$SCRIPTDIR` environment variable is set through `hack/make.sh`,
so will not be set when calling the `.integration-test-helpers` script directly.
Before this patch;
make BIND_DIR=. shell
...
bash: /make/.go-autogen: No such file or directory
After this patch, the warning is no longer printed
Also removed sourcing `.go-autogen` from test-integration and build-integration-test-binary,
as they already sourced `.integration-test-helpers` (which sources
`.go-autogen`).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These scripts have not been used for a while now, and should not be used again
because they are for releasing docker, not moby
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This fix updates runc to 0351df1c5a66838d0c392b4ac4cf9450de844e2d
With this fix the warnings generated by netgo and dlopen by go 1.9
are addressed.
See
- opencontainers/runc#1577
- opencontainers/runc#1579
This fix is part of the efforts for go 1.9 (#33892)
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
I have run into two separate issues while doing 'make all' on armhf
(a Scaleway C1 machine, same as used in CI). This commit fixes both.
1. There were a lot of "not enough memory" errors, and after that
in a few runs gometalinter just stuck forever on FUTEX_WAIT with
no children left.
Looking into docs, I found the --enable-gc option which solved the issue.
[Update: this has already been added]
2. Timeout of 2 minutes is not enough for the abovementioned platform.
The longest running linter is goimports which takes almost 6 minutes to run.
Set the timeout to the observable run time roughly doubled.
In addition, ARM platforms does not have too much RAM (2GB), so
running too many processes in parallel might be problematic. Limit
it by using -j2
[v2: make the timeout arch-dependent, also tested on aarch64 (2m15s)]
[v3: moved timeout setting to Dockerfiles]
[v4: generalized to GOMETALINTER_OPTS, added -j2 for ARM platforms]
[v5: rebase to master]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
VERSION was hardcoded to be used as the `VERSION` file from the root
directory, this makes it so that you have the option to overwrite this.
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
Failures from the integration suite were not propagating to the outter shell
for some reason. Handle the failure with an if exit 1.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This picks up 66eb2a3e8fc930e1bb6703561152edf5ab550bff, which fixes
readonly containers in user namespaces.
Signed-off-by: Tycho Andersen <tycho@docker.com>
