builder/remotecontext/detect_test.go:64:66: empty-lines: extra empty line at the end of a block (revive)
builder/remotecontext/detect_test.go:78:46: empty-lines: extra empty line at the end of a block (revive)
builder/remotecontext/detect_test.go:91:51: empty-lines: extra empty line at the end of a block (revive)
builder/dockerfile/internals_test.go:95:38: empty-lines: extra empty line at the end of a block (revive)
builder/dockerfile/copy.go:86:112: empty-lines: extra empty line at the end of a block (revive)
builder/dockerfile/dispatchers_test.go:286:39: empty-lines: extra empty line at the start of a block (revive)
builder/dockerfile/builder.go:280:38: empty-lines: extra empty line at the end of a block (revive)
builder/dockerfile/dispatchers.go:66:85: empty-lines: extra empty line at the start of a block (revive)
builder/dockerfile/dispatchers.go:559:85: empty-lines: extra empty line at the start of a block (revive)
builder/builder-next/adapters/localinlinecache/inlinecache.go:26:183: empty-lines: extra empty line at the start of a block (revive)
builder/builder-next/adapters/containerimage/pull.go:441:9: empty-lines: extra empty line at the start of a block (revive)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
It was unclear what the distinction was between these configuration
structs, so merging them to simplify.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Driver abstraction was needed for Linux Containers on Windows,
support for which has since been removed.
There is no direct equivalent to Lchmod() in the standard library so
continue to use the containerd/continuity version.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The OCI image spec is considering to change the Image struct and embedding the
Platform type (see opencontainers/image-spec#959) in the go implementation.
Moby currently uses some struct-literals to propagate the platform fields,
which will break once those changes in the OCI spec are merged.
Ideally (once that change arrives) we would update the code to set the Platform
information as a whole, instead of assigning related fields individually, but
in some cases in the code, image platform information is only partially set
(for example, OSVersion and OSFeatures are not preserved in all cases). This
may be on purpose, so needs to be reviewed.
This patch keeps the current behavior (assigning only specific fields), but
removes the use of struct-literals to make the code compatible with the
upcoming changes in the image-spec module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The fillGo18FileTypeBits func was added in 1a451d9a7b
to keep the tar headers consistent with headers created with go1.8 and older.
go1.8 and older incorrectly preserved all file-mode bits, including file-type,
instead of stripping those bits and only preserving the _permission_ bits, as
defined in;
- the GNU tar spec: https://www.gnu.org/software/tar/manual/html_node/Standard.html
- and POSIX: http://pubs.opengroup.org/onlinepubs/009695399/basedefs/tar.h.html
We decided at the time to copy the "wrong" behavior to prevent a cache-bust and
to keep the archives identical, however:
- It's not matching the standards, which causes differences between our tar
implementation and the standard tar implementations, as well as implementations
in other languages, such as Python (see docker/compose#883).
- BuildKit does not implement this hack.
- We don't _need_ this extra information (as it's already preserved in the
type header; https://pkg.go.dev/archive/tar#pkg-constants
In short; let's remove this hack.
This reverts commit 1a451d9a7b.
This reverts commit 41eb61d5c2.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
rename some variables that collided with imports or (upcoming)
changes, e.g. `ctx`, which is commonly used for `context.Context`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Older versions of Go don't format comments, so committing this as
a separate commit, so that we can already make these changes before
we upgrade to Go 1.19.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This allows the postContainersKill() handler to pass values as-is. As part of
the rewrite, I also moved the daemon.GetContainer(name) call later in the
function, so that we can fail early if an invalid signal is passed, before
doing the (heavier) fetching of the container.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Simplify some of the logic, and add documentation about the package,
as well as warnings that this package should not be used as a general-
purpose utility.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
pkg/urlutil (despite its poorly chosen name) is not really intended as a generic
utility to handle URLs, and should only be used by the builder to handle (remote)
build contexts.
- IsURL() only does a very rudimentary check for http(s):// prefixes, without any
other validation, but due to its name may give incorrect expectations.
- IsGitURL() is written specifically with docker build remote git contexts in
mind, and has handling for backward-compatibility, where strings that are
not URLs, but start with "github.com/" are accepted.
Because of the above, this patch:
- moves the package inside builder/remotecontext, close to where it's intended
to be used (ideally this would be part of build/remotecontext itself, but this
package imports many other dependencies, which would introduce those as extra
dependencies in the CLI).
- deprecates pkg/urlutil, but adds aliases as there are some external consumers.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Finish the refactor which was partially completed with commit
34536c498d, passing around IdentityMapping structs instead of pairs of
[]IDMap slices.
Existing code which uses []IDMap relies on zero-valued fields to be
valid, empty mappings. So in order to successfully finish the
refactoring without introducing bugs, their replacement therefore also
needs to have a useful zero value which represents an empty mapping.
Change IdentityMapping to be a pass-by-value type so that there are no
nil pointers to worry about.
The functionality provided by the deprecated NewIDMappingsFromMaps
function is required by unit tests to to construct arbitrary
IdentityMapping values. And the daemon will always need to access the
mappings to pass them to the Linux kernel. Accommodate these use cases
by exporting the struct fields instead. BuildKit currently depends on
the UIDs and GIDs methods so we cannot get rid of them yet.
Signed-off-by: Cory Snider <csnider@mirantis.com>
This package was deprecated in de56a90929, which
was part of the 20.10 release, so consumers of this package should've been
able to migrate to the new location.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I think this was there for historic reasons (may have been goimports expected
this, and we used to have a linter that wanted it), but it's not needed, so
let's remove it (to make my IDE less complaining about unneeded aliases).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This removes some of the checks that were added in 0cba7740d4,
but should no longer be needed.
- `dockerfile.BuildFromConfig()` is used for `docker (container) commmit` and
`docker (image) import`. For `docker import`, we're failing early already.
For `commit`, it won't be possible to have a container that doesn't have the
right operating-system, so there's no need to validate.
- `dispatchRequest.getImageOrStage()`: simplify the check; all checks resulted
in an error on Windows, so it came down to "Windows does not support FROM scratch".
- `dispatchState.beginStage()`: `image.OperatingSystem()` already defaults to the
`runtime.GOOS` if unset, so remove the local default fallback.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
None of the implementations used return an error, so removing the error
return can simplify using these.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This argument was added for LCOW support, but it was only used to verify if
the passed platform (OS) matched the host. Given that all uses of this function
(except for one) passed runtime.GOOS, we may as well move the check to that
location.
We should do more cleaning up after this, and perform such validations early,
instead of passing platform around in too many places where it's only used for
similar validations. This is a first step in that direction.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This interface only had a single implementation (xfer.LayerDownloadManager),
and all places where it was used already imported the xfer package.
Removing the interface, also makes it a closer match to the "upload" part,
as `xfer.LayerUploadManager()` did not use an interface.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
These consts were used in combination with idtools utilities, which
makes it a more logical location for these consts to live.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The LCOW implementation in dockerd has been deprecated in favor of re-implementation
in containerd (in progress). Microsoft started removing the LCOW V1 code from the
build dependencies we use in Microsoft/opengcs (soon to be part of Microsoft/hcshhim),
which means that we need to start removing this code.
This first step removes the lcow graphdriver, the LCOW initialization code, and
some LCOW-related utilities.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
After moving libnetwork to this repo, we need to update all the import
paths for libnetwork to point to docker/docker/libnetwork instead of
docker/libnetwork.
This change implements that.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Schema1 images can not have a config based cache key
before the layers are pulled. Avoid validation and reuse
manifest digest as a second key.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Relates to a82fff6377/docs/packages.md (proxies)
> (..) the first four of these are the standard built-in build-arg options
> available for `docker build`
> (..) The last, `all_proxy`, is a standard var used for socks proxying. Since
> it is not built into `docker build`, if you want to use it, you will need to
> add the following line to the dockerfile:
>
> ARG all_proxy
Given the we support all other commonly known proxy env-vars by default, it makes
sense to add this one as well.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
With the promotion of the experimental Dockerfile syntax to "stable", the Dockerfile
syntax now includes some options that are supported by BuildKit, but not (yet)
supported in the classic builder.
As a result, parsing a Dockerfile may succeed, but any flag that's known to BuildKit,
but not supported by the classic builder is silently ignored;
$ mkdir buildkit_flags && cd buildkit_flags
$ touch foo.txt
For example, `RUN --mount`:
DOCKER_BUILDKIT=0 docker build --no-cache -f- . <<EOF
FROM busybox
RUN --mount=type=cache,target=/foo echo hello
EOF
Sending build context to Docker daemon 2.095kB
Step 1/2 : FROM busybox
---> 219ee5171f80
Step 2/2 : RUN --mount=type=cache,target=/foo echo hello
---> Running in 022fdb856bc8
hello
Removing intermediate container 022fdb856bc8
---> e9f0988844d1
Successfully built e9f0988844d1
Or `COPY --chmod` (same for `ADD --chmod`):
DOCKER_BUILDKIT=0 docker build --no-cache -f- . <<EOF
FROM busybox
COPY --chmod=0777 /foo.txt /foo.txt
EOF
Sending build context to Docker daemon 2.095kB
Step 1/2 : FROM busybox
---> 219ee5171f80
Step 2/2 : COPY --chmod=0777 /foo.txt /foo.txt
---> 8b7117932a2a
Successfully built 8b7117932a2a
Note that unknown flags still produce and error, for example, the below fails because `--hello` is an unknown flag;
DOCKER_BUILDKIT=0 docker build -<<EOF
FROM busybox
RUN --hello echo hello
EOF
Sending build context to Docker daemon 2.048kB
Error response from daemon: dockerfile parse error line 2: Unknown flag: hello
With this patch applied
----------------------------
With this patch applied, flags that are known in the Dockerfile spec, but are not
supported by the classic builder, produce an error, which includes a link to the
documentation how to enable BuildKit:
DOCKER_BUILDKIT=0 docker build --no-cache -f- . <<EOF
FROM busybox
RUN --mount=type=cache,target=/foo echo hello
EOF
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM busybox
---> b97242f89c8a
Step 2/2 : RUN --mount=type=cache,target=/foo echo hello
the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled
DOCKER_BUILDKIT=0 docker build --no-cache -f- . <<EOF
FROM busybox
COPY --chmod=0777 /foo.txt /foo.txt
EOF
Sending build context to Docker daemon 2.095kB
Step 1/2 : FROM busybox
---> b97242f89c8a
Step 2/2 : COPY --chmod=0777 /foo.txt /foo.txt
the --chmod option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Config resolution was synchronized based on a wrong key as ref
variable is initialized only after in the same function. Using
the right key isn't fully correct either as the synchronized method
changes properties of the puller instance and can't be just skipped.
Added better error handling for the same case as well.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Otherwise a malformed or empty digest may cause a panic.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit a7d4af84bd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 6861f17f15...v0.8.0-rc2
- dockerfile: rename experimental channel to labs
- dockerfile build: fix not exit when meet error in load config metadata
- copy containerd.UnknownExitStatus to local const to reduce dependency graph in client
- executor: switch to docker seccomp profile
- add retry handlers to push/pull
- SSH-based auth for llb.Git operations
- Allow gateway exec-ing into a failed solve with an exec op
- Fix parsing ssh-based git sources
- Fix sshkeyscan to work with ipv6
- fix assumption that ssh port must be 2 digits
- vendor: github.com/Microsoft/go-winio v0.4.15
- vendor: github.com/tonistiigi/fsutil v0.0.0-20201103201449-0834f99b7b85
- vendor: containerd v1.4.1-0.20201117152358-0edc412565dc
- vendor: golang.org/x/crypto v0.0.0-20201117144127-c1f2f97bffc9
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This applies the 127GB default WCOW Sandbox size to not just `RUN` under
`docker build` (as was previously the case) but to `COPY` and `ADD`
under `docker build` and also to `docker run`.
It also removes an inconsistency that the 127GB size was not applied
when `--platform windows` was not passed to `docker build`, but WCOW was
still used as a platform default, e.g. Docker Desktop for Windows in
Windows Containers mode.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Commit 6ca3ec88ae deprecated the experimental
"--stream" option for the legacy builder, adding an error message is a client
attempted to use this feature.
However, the detection used the session-ID (`session=xxx` query parameter),
which happens to be set automatically by the CLI if it detects that the daemon
has session support. Because of this, builds fail when trying to perform them
on a daemon with the `--experimental` flag set.
This patch changes the detection to look for the `remote` query parameter, which
is set to "client-session" when using the `--stream` option with the classic
(non-Buildkit) builder.
Before this change, running `docker build` with an older (19.03 or older) cli
against a daemon with `--experimental` enabled caused an error:
$ dockerd --experimental &
$ docker pull docker:18.09
$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker:18.09 sh -c 'echo "FROM scratch" | docker build -'
Sending build context to Docker daemon 2.048kB
Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead
docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -w /foo docker:18.09 sh -c 'echo "FROM scratch" > Dockerfile && docker build --stream .'
Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead
With this patch, the error only occurs when trying to use the experimental
`--stream` option:
$ dockerd --experimental &
$ docker pull docker:18.09
$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker:18.09 sh -c 'echo "FROM scratch" | docker build -'
Step 1/1 : FROM scratch
--->
No image was generated. Is your Dockerfile empty?
$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -w /foo docker:18.09 sh -c 'echo "FROM scratch" > Dockerfile && docker build --stream .'
Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before this change, the error returned to the user would include the physical
path inside the tmp dir on the daemon host. These paths should be considered
an implementation detail, and provide no value to the user. Printing the tmp
path can confuse users, and will be even more confusing if the daemon is running
remotely (or in a VM, such as on Docker Desktop), in which case the path in the
error message does not exist on the local machine;
echo -e "FROM busybox\nCOPY /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
Sending build context to Docker daemon 1.57kB
Step 1/2 : FROM busybox
---> 1c35c4412082
Step 2/2 : COPY /some/non-existing/file.txt .
COPY failed: stat /var/lib/docker/tmp/docker-builder405687992/some/non-existing/file.txt: no such file or directory
When copying files from an image or a build stage, using `--from`, the error
is similarly confusing:
echo -e "FROM busybox\nCOPY --from=busybox /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
Sending build context to Docker daemon 4.671kB
Step 1/2 : FROM busybox
---> 018c9d7b792b
Step 2/2 : COPY --from=busybox /some/non-existing/file.txt .
COPY failed: stat /var/lib/docker/overlay2/ef34239c80526c779b7afaeaedbf11c1b201d7f7681d45613102c4541da0e156/merged/some/non-existing/file.txt: no such file or directory
This patch updates the error messages to be more user-friendly. Changes are slightly
different, depending on if the source was a local path, or an image (or build-stage),
using `--from`.
If `--from` is used, only the path is updated, and we print the relative path
instead of the full path;
echo -e "FROM busybox\nCOPY --from=busybox /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
Sending build context to Docker daemon 1.583kB
Step 1/2 : FROM busybox
---> 018c9d7b792b
Step 2/2 : COPY --from=busybox /some/non-existing/file.txt .
COPY failed: stat some/non-existing/file.txt: file does not exist
In other cases, additional information is added to mention "build context" and
".dockerignore", which could provide the user some hints to find the problem:
echo -e "FROM busybox\nCOPY /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
Sending build context to Docker daemon 1.583kB
Step 1/2 : FROM busybox
---> 018c9d7b792b
Step 2/2 : COPY /some/non-existing/file.txt .
COPY failed: file not found in build context or excluded by .dockerignore: stat some/non-existing/file.txt: file does not exist
echo -e "FROM busybox\nADD /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
Sending build context to Docker daemon 1.583kB
Step 1/2 : FROM busybox
---> 018c9d7b792b
Step 2/2 : ADD /some/non-existing/file.txt .
ADD failed: file not found in build context or excluded by .dockerignore: stat some/non-existing/file.txt: file does not exist
This patch only improves the error for the classic builder. Similar changes could
be made for BuildKit, which produces equally, or even more confusing errors;
echo -e "FROM busybox\nCOPY /some/non-existing/file.txt ." | DOCKER_BUILDKIT=1 docker build -f- .
[+] Building 1.2s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 85B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/busybox:latest 1.2s
=> [internal] load build context 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [1/2] FROM docker.io/library/busybox@sha256:4f47c01... 0.0s
=> ERROR [2/2] COPY /some/non-existing/file.txt . 0.0s
------
> [2/2] COPY /some/non-existing/file.txt .:
------
failed to compute cache key: failed to walk /var/lib/docker/tmp/buildkit-mount181923793/some/non-existing:
lstat /var/lib/docker/tmp/buildkit-mount181923793/some/non-existing: no such file or directory
echo -e "FROM busybox\nCOPY --from=busybox /some/non-existing/file.txt ." | DOCKER_BUILDKIT=1 docker build -f- .
[+] Building 2.5s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 100B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/busybox:latest 1.2s
=> FROM docker.io/library/busybox:latest 1.2s
=> => resolve docker.io/library/busybox:latest 1.2s
=> CACHED [stage-0 1/2] FROM docker.io/library/busybox@sha256:4f47c01... 0.0s
=> ERROR [stage-0 2/2] COPY --from=busybox /some/non-existing/file.txt . 0.0s
------
> [stage-0 2/2] COPY --from=busybox /some/non-existing/file.txt .:
------
failed to compute cache key: failed to walk /var/lib/docker/overlay2/2a796d91e46fc038648c6010f062bdfd612ee62b0e8fe77bc632688e3fba32d9/merged/some/non-existing:
lstat /var/lib/docker/overlay2/2a796d91e46fc038648c6010f062bdfd612ee62b0e8fe77bc632688e3fba32d9/merged/some/non-existing: no such file or directory
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Errors should not be capitalized. This error was marked as
"being compatible" with the old error, However, looking at
the original error that was in place before d1faf3df27,
I noticed that the error was lowercase before, so it should
be ok to change it back to be lowercase.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
After switch to leases all ref IDs use generated IDs meaning that
EnsureLayer hack needs to save the chainID for pulled layers.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
