Commit graph

524 commits

Author SHA1 Message Date
Cory Snider
098a44c07f Finish refactor of UID/GID usage to a new struct
Finish the refactor which was partially completed with commit
34536c498d, passing around IdentityMapping structs instead of pairs of
[]IDMap slices.

Existing code which uses []IDMap relies on zero-valued fields to be
valid, empty mappings. So in order to successfully finish the
refactoring without introducing bugs, their replacement therefore also
needs to have a useful zero value which represents an empty mapping.
Change IdentityMapping to be a pass-by-value type so that there are no
nil pointers to worry about.

The functionality provided by the deprecated NewIDMappingsFromMaps
function is required by unit tests to to construct arbitrary
IdentityMapping values. And the daemon will always need to access the
mappings to pass them to the Linux kernel. Accommodate these use cases
by exporting the struct fields instead. BuildKit currently depends on
the UIDs and GIDs methods so we cannot get rid of them yet.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2022-03-14 16:28:57 -04:00
Stéphane Este-Gracias
a768652fd6
Support STOPSIGNAL instruction in commit change
Signed-off-by: Stéphane Este-Gracias <sestegra@gmail.com>
2022-03-12 17:10:35 +01:00
Sebastiaan van Stijn
cfddecc3d2
builder/dockerfile: remove leftover LCOW platform checks
This removes some of the checks that were added in 0cba7740d4,
but should no longer be needed.

- `dockerfile.BuildFromConfig()` is used for `docker (container) commmit` and
  `docker (image) import`. For `docker import`, we're failing early already.
  For `commit`, it won't be possible to have a container that doesn't have the
  right operating-system, so there's no need to validate.
- `dispatchRequest.getImageOrStage()`: simplify the check; all checks resulted
  in an error on Windows, so it came down to "Windows does not support FROM scratch".
- `dispatchState.beginStage()`: `image.OperatingSystem()` already defaults to the
  `runtime.GOOS` if unset, so remove the local default fallback.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-25 15:23:11 +01:00
Eng Zer Jun
c55a4ac779
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-27 14:56:57 +08:00
Sebastiaan van Stijn
686be57d0a
Update to Go 1.17.0, and gofmt with Go 1.17
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-24 23:33:27 +02:00
Brian Goff
6a60efc39b
Merge pull request #42684 from thaJeztah/remove_lcow_step7
Remove LCOW (step 7): remove LCOW bits from builder/dockerfile (copy)
2021-08-05 15:16:24 -07:00
Sebastiaan van Stijn
e05fc8ca35
builder/dockerfile: make normalizeDest() platform-specific again
Removing various bits that were added in 7a7357dae1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 23:14:40 +02:00
Sebastiaan van Stijn
337500f374
builder/dockerfile: make containsWildcards() platform-specific again
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 23:14:38 +02:00
Sebastiaan van Stijn
131fac6eb4
builder/dockerfile: remove fromSlash() and separator() utils
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 23:14:36 +02:00
Sebastiaan van Stijn
190b6f64e3
builder/dockerfile: copy: remove LCOW code
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 23:14:28 +02:00
Sebastiaan van Stijn
0998c7e363
system: remove LCOWSupported() utility
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 13:36:19 +02:00
Sebastiaan van Stijn
28409ca6c7
replace pkg/signal with moby/sys/signal v0.5.0
This code was moved to the moby/sys repository

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-23 09:32:54 +02:00
Sebastiaan van Stijn
46c591b045
pkg/system: deprecate some consts and move them to pkg/idtools
These consts were used in combination with idtools utilities, which
makes it a more logical location for these consts to live.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-18 23:16:15 +02:00
Akihiro Suda
9e8cf1016e
Merge pull request #42473 from thaJeztah/unfork_buildkit
revendor BuildKit (master branch)
2021-06-17 10:56:25 +09:00
Sebastiaan van Stijn
3eb1257698
revendor BuildKit (master branch)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-16 01:17:48 +02:00
Sebastiaan van Stijn
d13997b4ba
gosec: G601: Implicit memory aliasing in for loop
plugin/v2/plugin.go:141:50: G601: Implicit memory aliasing in for loop. (gosec)
                    updateSettingsEnv(&p.PluginObj.Settings.Env, &s)
                                                                 ^
    libcontainerd/remote/client.go:572:13: G601: Implicit memory aliasing in for loop. (gosec)
                cpDesc = &m
                         ^
    distribution/push_v2.go:400:34: G601: Implicit memory aliasing in for loop. (gosec)
                (metadata.CheckV2MetadataHMAC(&mountCandidate, pd.hmacKey) ||
                                              ^
    builder/dockerfile/builder.go:261:84: G601: Implicit memory aliasing in for loop. (gosec)
            currentCommandIndex = printCommand(b.Stdout, currentCommandIndex, totalCommands, &meta)
                                                                                             ^
    builder/dockerfile/builder.go:278:46: G601: Implicit memory aliasing in for loop. (gosec)
            if err := initializeStage(dispatchRequest, &stage); err != nil {
                                                       ^
    daemon/container.go:283:40: G601: Implicit memory aliasing in for loop. (gosec)
            if err := parser.ValidateMountConfig(&cfg); err != nil {
                                                 ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:29 +02:00
Sebastiaan van Stijn
e047d984dc
Remove LCOW code (step 1)
The LCOW implementation in dockerd has been deprecated in favor of re-implementation
in containerd (in progress). Microsoft started removing the LCOW V1 code from the
build dependencies we use in Microsoft/opengcs (soon to be part of Microsoft/hcshhim),
which means that we need to start removing this code.

This first step removes the lcow graphdriver, the LCOW initialization code, and
some LCOW-related utilities.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-03 21:16:21 +02:00
Sebastiaan van Stijn
5a5f8564ba
builder/dockerfile: add "ALL_PROXY" to list of default build args
Relates to a82fff6377/docs/packages.md (proxies)

> (..) the first four of these are the standard built-in build-arg options
> available for `docker build`
> (..) The last, `all_proxy`, is a standard var used for socks proxying. Since
> it is not built into `docker build`, if you want to use it, you will need to
> add the following line to the dockerfile:
>
> ARG all_proxy

Given the we support all other commonly known proxy env-vars by default, it makes
sense to add this one as well.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-24 18:25:16 +02:00
Sebastiaan van Stijn
a09c0276a2
builder: produce error when using unsupported Dockerfile option
With the promotion of the experimental Dockerfile syntax to "stable", the Dockerfile
syntax now includes some options that are supported by BuildKit, but not (yet)
supported in the classic builder.

As a result, parsing a Dockerfile may succeed, but any flag that's known to BuildKit,
but not supported by the classic builder is silently ignored;

    $ mkdir buildkit_flags && cd buildkit_flags
    $ touch foo.txt

For example, `RUN --mount`:

    DOCKER_BUILDKIT=0 docker build --no-cache -f- . <<EOF
    FROM busybox
    RUN --mount=type=cache,target=/foo echo hello
    EOF

    Sending build context to Docker daemon  2.095kB
    Step 1/2 : FROM busybox
     ---> 219ee5171f80
    Step 2/2 : RUN --mount=type=cache,target=/foo echo hello
     ---> Running in 022fdb856bc8
    hello
    Removing intermediate container 022fdb856bc8
     ---> e9f0988844d1
    Successfully built e9f0988844d1

Or `COPY --chmod` (same for `ADD --chmod`):

    DOCKER_BUILDKIT=0 docker build --no-cache -f- . <<EOF
    FROM busybox
    COPY --chmod=0777 /foo.txt /foo.txt
    EOF

    Sending build context to Docker daemon  2.095kB
    Step 1/2 : FROM busybox
     ---> 219ee5171f80
    Step 2/2 : COPY --chmod=0777 /foo.txt /foo.txt
     ---> 8b7117932a2a
    Successfully built 8b7117932a2a

Note that unknown flags still produce and error, for example, the below fails because `--hello` is an unknown flag;

    DOCKER_BUILDKIT=0 docker build -<<EOF
    FROM busybox
    RUN --hello echo hello
    EOF

    Sending build context to Docker daemon  2.048kB
    Error response from daemon: dockerfile parse error line 2: Unknown flag: hello

With this patch applied
----------------------------

With this patch applied, flags that are known in the Dockerfile spec, but are not
supported by the classic builder, produce an error, which includes a link to the
documentation how to enable BuildKit:

    DOCKER_BUILDKIT=0 docker build --no-cache -f- . <<EOF
    FROM busybox
    RUN --mount=type=cache,target=/foo echo hello
    EOF

    Sending build context to Docker daemon  2.048kB
    Step 1/2 : FROM busybox
     ---> b97242f89c8a
    Step 2/2 : RUN --mount=type=cache,target=/foo echo hello
    the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled

    DOCKER_BUILDKIT=0 docker build --no-cache -f- . <<EOF
    FROM busybox
    COPY --chmod=0777 /foo.txt /foo.txt
    EOF

    Sending build context to Docker daemon  2.095kB
    Step 1/2 : FROM busybox
     ---> b97242f89c8a
    Step 2/2 : COPY --chmod=0777 /foo.txt /foo.txt
    the --chmod option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-03-14 14:11:08 +01:00
Sebastiaan van Stijn
f266f13965
Merge pull request #41636 from TBBle/37352-test-and-fix
Set 127GB default sandbox size for WCOW, and ensure storage-opts is honoured on all paths under WCOW and LCOW
2021-01-25 14:34:34 +01:00
Tibor Vass
de56a90929 vendor buildkit 6861f17f15364de0fe1fd1e6e8da07598a485123
Signed-off-by: Tibor Vass <tibor@docker.com>
2020-11-14 03:57:36 +00:00
Paul "TBBle" Hampson
56d378a88f Apply a 127GB default WCOW Sandbox size globally
This applies the 127GB default WCOW Sandbox size to not just `RUN` under
`docker build` (as was previously the case) but to `COPY` and `ADD`
under `docker build` and also to `docker run`.

It also removes an inconsistency that the 127GB size was not applied
when `--platform windows` was not passed to `docker build`, but WCOW was
still used as a platform default, e.g. Docker Desktop for Windows in
Windows Containers mode.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-10 19:51:46 +11:00
Sebastiaan van Stijn
1bb3491601
builder: fix detection of experimental --stream option (deprecated)
Commit 6ca3ec88ae deprecated the experimental
"--stream" option for the legacy builder, adding an error message is a client
attempted to use this feature.

However, the detection used the session-ID (`session=xxx` query parameter),
which happens to be set automatically by the CLI if it detects that the daemon
has session support. Because of this, builds fail when trying to perform them
on a daemon with the `--experimental` flag set.

This patch changes the detection to look for the `remote` query parameter, which
is set to "client-session" when using the `--stream` option with the classic
(non-Buildkit) builder.

Before this change, running `docker build` with an older (19.03 or older) cli
against a daemon with `--experimental` enabled caused an error:

    $ dockerd --experimental &
    $ docker pull docker:18.09
    $ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker:18.09 sh -c 'echo "FROM scratch" | docker build -'

    Sending build context to Docker daemon  2.048kB
    Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead

    docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -w /foo docker:18.09 sh -c 'echo "FROM scratch" > Dockerfile && docker build --stream .'
    Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead

With this patch, the error only occurs when trying to use the experimental
`--stream` option:

    $ dockerd --experimental &
    $ docker pull docker:18.09
    $ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker:18.09 sh -c 'echo "FROM scratch" | docker build -'

    Step 1/1 : FROM scratch
     --->
    No image was generated. Is your Dockerfile empty?

    $ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -w /foo docker:18.09 sh -c 'echo "FROM scratch" > Dockerfile && docker build --stream .'

    Error response from daemon: experimental session with v1 builder is no longer supported, use builder version v2 (BuildKit) instead

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-06 21:13:28 +01:00
Sebastiaan van Stijn
dc3c382b34
replace pkg/symlink with github.com/moby/sys/symlink
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-03 11:17:12 +01:00
Brian Goff
bf6f0d82bc
Merge pull request #41360 from lzhfromustc/GL_5
Builder: Add 1 buffer to two channels to avoid blocking goroutine
2020-09-12 21:43:05 -07:00
lzhfromustc
d1c2f953c5 Test: Add buffer to two channels to avoid blocking goroutine
Signed-off-by: Ziheng Liu <lzhfromustc@gmail.com>
2020-08-27 18:53:43 -04:00
Sebastiaan van Stijn
010adeec55
Builder: print relative path if COPY/ADD source path was not found
Before this change, the error returned to the user would include the physical
path inside the tmp dir on the daemon host. These paths should be considered
an implementation detail, and provide no value to the user. Printing the tmp
path can confuse users, and will be even more confusing if the daemon is running
remotely (or in a VM, such as on Docker Desktop), in which case the path in the
error message does not exist on the local machine;

    echo -e "FROM busybox\nCOPY /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .

    Sending build context to Docker daemon   1.57kB
    Step 1/2 : FROM busybox
     ---> 1c35c4412082
    Step 2/2 : COPY /some/non-existing/file.txt .
    COPY failed: stat /var/lib/docker/tmp/docker-builder405687992/some/non-existing/file.txt: no such file or directory

When copying files from an image or a build stage, using `--from`, the error
is similarly confusing:

    echo -e "FROM busybox\nCOPY --from=busybox /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
    Sending build context to Docker daemon  4.671kB
    Step 1/2 : FROM busybox
     ---> 018c9d7b792b
    Step 2/2 : COPY --from=busybox /some/non-existing/file.txt .
    COPY failed: stat /var/lib/docker/overlay2/ef34239c80526c779b7afaeaedbf11c1b201d7f7681d45613102c4541da0e156/merged/some/non-existing/file.txt: no such file or directory

This patch updates the error messages to be more user-friendly. Changes are slightly
different, depending on if the source was a local path, or an image (or build-stage),
using `--from`.

If `--from` is used, only the path is updated, and we print the relative path
instead of the full path;

    echo -e "FROM busybox\nCOPY --from=busybox /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
    Sending build context to Docker daemon  1.583kB
    Step 1/2 : FROM busybox
     ---> 018c9d7b792b
    Step 2/2 : COPY --from=busybox /some/non-existing/file.txt .
    COPY failed: stat some/non-existing/file.txt: file does not exist

In other cases, additional information is added to mention "build context" and
".dockerignore", which could provide the user some hints to find the problem:

    echo -e "FROM busybox\nCOPY /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
    Sending build context to Docker daemon  1.583kB
    Step 1/2 : FROM busybox
     ---> 018c9d7b792b
    Step 2/2 : COPY /some/non-existing/file.txt .
    COPY failed: file not found in build context or excluded by .dockerignore: stat some/non-existing/file.txt: file does not exist

    echo -e "FROM busybox\nADD /some/non-existing/file.txt ." | DOCKER_BUILDKIT=0 docker build -f- .
    Sending build context to Docker daemon  1.583kB
    Step 1/2 : FROM busybox
     ---> 018c9d7b792b
    Step 2/2 : ADD /some/non-existing/file.txt .
    ADD failed: file not found in build context or excluded by .dockerignore: stat some/non-existing/file.txt: file does not exist

This patch only improves the error for the classic builder. Similar changes could
be made for BuildKit, which produces equally, or even more confusing errors;

    echo -e "FROM busybox\nCOPY /some/non-existing/file.txt ." | DOCKER_BUILDKIT=1 docker build -f- .
    [+] Building 1.2s (6/6) FINISHED
     => [internal] load build definition from Dockerfile                 0.0s
     => => transferring dockerfile: 85B                                  0.0s
     => [internal] load .dockerignore                                    0.0s
     => => transferring context: 2B                                      0.0s
     => [internal] load metadata for docker.io/library/busybox:latest    1.2s
     => [internal] load build context                                    0.0s
     => => transferring context: 2B                                      0.0s
     => CACHED [1/2] FROM docker.io/library/busybox@sha256:4f47c01...    0.0s
     => ERROR [2/2] COPY /some/non-existing/file.txt .                   0.0s
    ------
     > [2/2] COPY /some/non-existing/file.txt .:
    ------
    failed to compute cache key: failed to walk /var/lib/docker/tmp/buildkit-mount181923793/some/non-existing:
    lstat /var/lib/docker/tmp/buildkit-mount181923793/some/non-existing: no such file or directory

    echo -e "FROM busybox\nCOPY --from=busybox /some/non-existing/file.txt ." | DOCKER_BUILDKIT=1 docker build -f- .
    [+] Building 2.5s (6/6) FINISHED
     => [internal] load build definition from Dockerfile                        0.0s
     => => transferring dockerfile: 100B                                        0.0s
     => [internal] load .dockerignore                                           0.0s
     => => transferring context: 2B                                             0.0s
     => [internal] load metadata for docker.io/library/busybox:latest           1.2s
     => FROM docker.io/library/busybox:latest                                   1.2s
     => => resolve docker.io/library/busybox:latest                             1.2s
     => CACHED [stage-0 1/2] FROM docker.io/library/busybox@sha256:4f47c01...   0.0s
     => ERROR [stage-0 2/2] COPY --from=busybox /some/non-existing/file.txt .   0.0s
    ------
     > [stage-0 2/2] COPY --from=busybox /some/non-existing/file.txt .:
    ------
    failed to compute cache key: failed to walk /var/lib/docker/overlay2/2a796d91e46fc038648c6010f062bdfd612ee62b0e8fe77bc632688e3fba32d9/merged/some/non-existing:
    lstat /var/lib/docker/overlay2/2a796d91e46fc038648c6010f062bdfd612ee62b0e8fe77bc632688e3fba32d9/merged/some/non-existing: no such file or directory

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-08-17 12:03:18 +02:00
Sebastiaan van Stijn
afd9a6c2b2
builder/remotecontext: use lowercase for error
Errors should not be capitalized. This error was marked as
"being compatible" with the old error, However, looking at
the original error that was in place before d1faf3df27,
I noticed that the error was lowercase before, so it should
be ok to change it back to be lowercase.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-08-11 10:46:18 +02:00
Sebastiaan van Stijn
3895dd585f
Replace uses of blacklist/whitelist
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-07-14 10:41:34 +02:00
Tonis Tiigi
9b28939345 vendor: update buildkit to df35e9818
Update to new buildkit and fix upgrade bugs

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-06-15 09:44:41 -07:00
Tibor Vass
751d5f879a
Merge pull request #40646 from thaJeztah/builder_use_pkg_sys_windows
pkg/system: deprecate SetNamedSecurityInfo, GetSecurityDescriptorDacl
2020-05-08 08:21:17 -07:00
Sebastiaan van Stijn
07d60bc257
Replace errors.Cause() with errors.Is() / errors.As()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-04-29 00:28:41 +02:00
Sebastiaan van Stijn
70a4d886cd
builder: fixPermissionsWindows(): use golang.org/x/sys/windows
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-03-09 12:43:02 +01:00
Sebastiaan van Stijn
9f0b3f5609
bump gotest.tools v3.0.1 for compatibility with Go 1.14
full diff: https://github.com/gotestyourself/gotest.tools/compare/v2.3.0...v3.0.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-11 00:06:42 +01:00
Sebastiaan van Stijn
f22ff19668
builder: remove use of deprecated pkg/system constants
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-11-25 15:40:17 +01:00
Tõnis Tiigi
d1d5f64766
Merge pull request #40021 from thaJeztah/carry_40017
Use newer x/sys/windows SecurityAttributes struct (carry 40017)
2019-11-21 08:57:22 -08:00
Sebastiaan van Stijn
301a2fbeca
builder/dockerfile/mockbackend_test.go: suppress SA9005 (staticcheck)
```
builder/dockerfile/mockbackend_test.go:107:21: SA9005: struct doesn't have any exported fields, nor custom marshaling (staticcheck)
	return json.Marshal(rawImage(*i))
	                   ^
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-18 00:45:27 +02:00
Tibor Vass
6ca3ec88ae builder: remove legacy build's session handling
This feature was used by docker build --stream and it was kept experimental.

Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-10-02 20:29:15 +00:00
Jason A. Donenfeld
c3a0a37446
Use newer x/sys/windows SecurityAttributes struct
This struct now has a properly typed member, so use the properly typed
functions with it.

Also update the vendor directory and hope nothing explodes.

Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-02 21:12:23 +02:00
Chris Price
c21a3cf432 Add variant to image.Image and legacy builder
This commit adds the image variant to the image.(Image) type and
updates related functionality. Images built from another will
inherit the OS, architecture and variant.

Note that if a base image does not specify an architecture, the
local machine's architecture is used for inherited images. On the
other hand, the variant is set equal to the parent image's variant,
even when the parent image's variant is unset.

The legacy builder is also updated to allow the user to specify
a '--platform' argument on the command line when creating an image
FROM scratch. A complete platform specification, including variant,
is supported. The built image will include the variant, as will any
derived images.

Signed-off-by: Chris Price <chris.price@docker.com>
2019-09-24 22:18:16 +00:00
Sebastiaan van Stijn
a46ad5e704
Builder: remove unused field
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:57:32 +02:00
Kir Kolyshkin
d8ec1ee57d
builder/copy-unix: fix filepath.Walk args
Mark unused WalkFunc arguments as such, to get rid of the
following linterwarning:

> builder/dockerfile/copy_unix.go:28:71: SA4009: argument err is overwritten before first use (staticcheck)
> 	return filepath.Walk(source, func(fullpath string, info os.FileInfo, err error) error {

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2019-09-18 12:57:18 +02:00
Sebastiaan van Stijn
07ff4f1de8
goimports: fix imports
Format the source according to latest goimports.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:56:54 +02:00
Sebastiaan van Stijn
a3f9cb5b63
TestDispatch: refactor to use subtests again, and fix linting (structcheck)
Instead of using a `initDispatchTestCases()` function, declare the test-table
inside `TestDispatch` itself, and run the tests as subtests.

```
[2019-08-27T15:14:51.072Z] builder/dockerfile/evaluator_test.go:18:2: `name` is unused (structcheck)
[2019-08-27T15:14:51.072Z] 	name, expectedError string
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-11 11:13:03 +02:00
Sebastiaan van Stijn
7e8f7efdd1
fix build after merge of outdated PR
this broke after e554ab5589,
was merged but the PR was outdated.

```
[2019-08-28T00:27:12.031Z] builder/dockerfile/copy.go:566:29: too many arguments in call to system.MkdirAll
[2019-08-28T00:27:12.031Z] 	have (string, number, string)
[2019-08-28T00:27:12.031Z] 	want (string, os.FileMode)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-28 03:14:28 +02:00
Vikram bir Singh
b146b2382c Fixups for 39695 - edit comments, redundant lines
1. Modify comments added in 5858a99267
Windows Volume GUID path format is: \\?\Volume{<GUID Value>}\<path>
Rewrote the example given in comments to conform to the format..

2. Remove two redundant asserts[assert.NilError]. They are redundant
because the last statement will not change the value of err.

Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>
2019-08-15 21:13:46 +00:00
Sebastiaan van Stijn
5858a99267
Builder: fix "COPY --from" to non-existing directory on Windows
This fixes a regression introduced in 6d87f19142,
causing `COPY --from` to fail if the target directory does not exist:

```
FROM mcr.microsoft.com/windows/servercore:ltsc2019 as s1
RUN echo "Hello World" > /hello

FROM mcr.microsoft.com/windows/servercore:ltsc2019
COPY --from=s1 /hello /hello/another/world
```

Would produce an error:

```
Step 4/4 : COPY --from=s1 /hello /hello/another/world
failed to copy files: mkdir \\?: The filename, directory name, or volume label syntax is incorrect.
```

The cause for this was that Go's `os.MkdirAll()` does not support/detect volume GUID paths
(`\\?\Volume{dae8d3ac-b9a1-11e9-88eb-e8554b2ba1db}\hello\another}`), and as a result
attempted to create the volume as a directory (`\\?`), causing it to fail.

This patch replaces `os.MkdirAll()` with our own `system.MkdirAll()` function, which
is capable of detecting GUID volumes.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-08 16:45:10 +02:00
Tibor Vass
29de017df7
Merge pull request #38599 from thaJeztah/builder_fix_copy_permissions
builder: fix `COPY --from` should preserve ownership
2019-03-22 09:38:13 -07:00
John Howard
20833b06a0 Windows: (WCOW) Generate OCI spec that remote runtime can escape
Signed-off-by: John Howard <jhoward@microsoft.com>

Also fixes https://github.com/moby/moby/issues/22874

This commit is a pre-requisite to moving moby/moby on Windows to using
Containerd for its runtime.

The reason for this is that the interface between moby and containerd
for the runtime is an OCI spec which must be unambigious.

It is the responsibility of the runtime (runhcs in the case of
containerd on Windows) to ensure that arguments are escaped prior
to calling into HCS and onwards to the Win32 CreateProcess call.

Previously, the builder was always escaping arguments which has
led to several bugs in moby. Because the local runtime in
libcontainerd had context of whether or not arguments were escaped,
it was possible to hack around in daemon/oci_windows.go with
knowledge of the context of the call (from builder or not).

With a remote runtime, this is not possible as there's rightly
no context of the caller passed across in the OCI spec. Put another
way, as I put above, the OCI spec must be unambigious.

The other previous limitation (which leads to various subtle bugs)
is that moby is coded entirely from a Linux-centric point of view.

Unfortunately, Windows != Linux. Windows CreateProcess uses a
command line, not an array of arguments. And it has very specific
rules about how to escape a command line. Some interesting reading
links about this are:

https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/
https://stackoverflow.com/questions/31838469/how-do-i-convert-argv-to-lpcommandline-parameter-of-createprocess
https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments?view=vs-2017

For this reason, the OCI spec has recently been updated to cater
for more natural syntax by including a CommandLine option in
Process.

What does this commit do?

Primary objective is to ensure that the built OCI spec is unambigious.

It changes the builder so that `ArgsEscaped` as commited in a
layer is only controlled by the use of CMD or ENTRYPOINT.

Subsequently, when calling in to create a container from the builder,
if follows a different path to both `docker run` and `docker create`
using the added `ContainerCreateIgnoreImagesArgsEscaped`. This allows
a RUN from the builder to control how to escape in the OCI spec.

It changes the builder so that when shell form is used for RUN,
CMD or ENTRYPOINT, it builds (for WCOW) a more natural command line
using the original as put by the user in the dockerfile, not
the parsed version as a set of args which loses fidelity.
This command line is put into args[0] and `ArgsEscaped` is set
to true for CMD or ENTRYPOINT. A RUN statement does not commit
`ArgsEscaped` to the commited layer regardless or whether shell
or exec form were used.
2019-03-12 18:41:55 -07:00
Sebastiaan van Stijn
6d87f19142
builder: fix COPY --from should preserve ownership
When copying between stages, or copying from an image,
ownership of the copied files should not be changed, unless
the `--chown` option is set (in which case ownership of copied
files should be updated to the specified user/group).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-03-13 00:55:04 +01:00
John Howard
63f9c7784b LCOW: Log stderr on failures
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-26 13:23:04 -07:00
mooncake
5160003b4d fix typos
Signed-off-by: mooncake <xcoder@tenxcloud.com>
Signed-off-by: Bily Zhang <xcoder@tenxcloud.com>
2018-09-01 21:26:38 +08:00
Yong Tang
14d5569f19
Merge pull request #37563 from Microsoft/jjh/fix-vso17531561v2
LCOW: Ensure platform is populated on COPY/ADD
2018-08-17 12:51:17 -07:00
Salahuddin Khan
763d839261 Add ADD/COPY --chown flag support to Windows
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.

NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.

The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>

On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.

Signed-off-by: Salahuddin Khan <salah@docker.com>
2018-08-13 21:59:11 -07:00
John Howard
ae6b5828ce LCOW: Ensure platform is populated on COPY/ADD
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-07-30 12:49:33 -07:00
Tibor Vass
9ebed53c54
Merge pull request #37439 from tiborvass/vendor-buildkit
Set BuildKit's ExportedProduct variable to show useful errors in the future
2018-07-17 12:10:23 -07:00
Tibor Vass
81599222fc validate: please vet
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-07-16 21:41:54 +00:00
Yong Tang
492545e139
Merge pull request #37396 from yui-knk/error_when_base_name_resolved_to_blank
Return error if basename is expanded to blank
2018-07-15 15:42:57 -07:00
Yuichiro Kaneko
c9542d313e Return error if basename is expanded to blank
Fix: https://github.com/moby/moby/issues/37325

Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
2018-07-15 22:29:03 +09:00
Yuichiro Kaneko
44e08d8a7d Ensure RUN instruction to run without Healthcheck
Before this commit Healthcheck run if HEALTHCHECK
instruction appears before RUN instruction.
By passing `withoutHealthcheck` to `copyRunConfig`,
always RUN instruction run without Healthcheck.

Fix: https://github.com/moby/moby/issues/37362

Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
2018-07-08 16:49:04 +09:00
Tibor Vass
facad55744 api: Change Platform field back to string (temporary workaround)
This partially reverts https://github.com/moby/moby/pull/37350

Although specs.Platform is desirable in the API, there is more work
to be done on helper functions, namely containerd's platforms.Parse
that assumes the default platform of the Go runtime.

That prevents a client to use the recommended Parse function to
retrieve a specs.Platform object.

With this change, no parsing is expected from the client.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-07-03 22:33:42 +00:00
Sebastiaan van Stijn
1da7d2eebf
Merge pull request #37368 from tiborvass/buildkit-iidfile
builder: return image ID in API when using buildkit
2018-07-04 00:20:25 +02:00
Tibor Vass
ca8022ec63 builder: return image ID in API when using buildkit
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-07-03 19:11:02 +00:00
Yuichiro Kaneko
9b374801ac Update documents of dispatchAdd
`ADD` does not support git.

Ref: https://github.com/moby/moby/issues/14704#issuecomment-298134333

Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
2018-07-03 08:51:07 +09:00
Tonis Tiigi
f099771665 system: add back lcow validation function
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-27 15:24:26 -07:00
Tonis Tiigi
337ba71fc1 distribution: fix passing platform struct to puller
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-27 14:59:31 -07:00
Tonis Tiigi
81f862a1fe api: fix platform type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-26 22:59:17 -07:00
Tonis Tiigi
48b93419dc builder: buildkit rebase update
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-26 22:59:17 -07:00
Dennis Chen
7f334d3acf Initial support for OCI multi-platform image
Add the OCI spec compatible image support in client side.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-06-26 11:39:29 -07:00
John Howard
35193c0e7d LCOW: Auto-select OS
Signed-off-by: John Howard <jhoward@microsoft.com>

Addresses https://github.com/moby/moby/pull/35089#issuecomment-367802698.
This change enables the daemon to automatically select an image under LCOW
that can be used if the API doesn't specify an explicit platform.

For example:

FROM supertest2014/nyan
ADD Dockerfile /

And docker build . will download the linux image (not a multi-manifest image)

And similarly docker pull ubuntu will match linux/amd64
2018-06-26 11:37:43 -07:00
Vincent Demeester
3845728524
Update tests to use gotest.tools 👼
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-06-13 09:04:30 +02:00
Sebastiaan van Stijn
52ea99eda2
Merge pull request #37189 from arm64b/create-intermediate-for-COPY-ADD
Refactor and cleanup the intermediate container creation
2018-06-07 21:20:17 -07:00
Dennis Chen
7f280f6f65 Refactor and cleanup the intermediate container creation
This PR is trying to refactor the `probeAndCreate` and cleanup
related codes based on the refactoring.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-06-04 12:55:49 +08:00
Tonis Tiigi
c9ebd2f13b vendor: use dockerfile parser from buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-02 11:10:34 -07:00
Vincent Demeester
3e494c4b93
Merge pull request #37178 from tonistiigi/layer-leak
builder: fix layer leak on multi-stage wildcard copy
2018-05-31 09:39:28 +02:00
Tonis Tiigi
4a18c11bdc builder: fix layer leak on multi-stage wildcard copy
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-30 15:18:40 -07:00
Sebastiaan van Stijn
2628896b5e
Handle multi-digit positional parameters
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:51:11 +02:00
Sebastiaan van Stijn
9654e9b6f8
Add detection of "special parameters" for substitution
Detect Special parameters as defined in
http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_05_02

Treat these as parameters that are not set, instead of
producing an error that a modifier is missing.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:20:22 +02:00
Sebastiaan van Stijn
8687a3f4b8
Add more test-cases for positional parameters
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:20:10 +02:00
Sebastiaan van Stijn
b80e0309d2
Add line-numbers to asserts
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:20:01 +02:00
Sebastiaan van Stijn
334bf3ea76
Fix detection for missing parameter in substitution
`${}`, `${:}` and so on are invalid because there's
no parameter within the brackets; fix detection for
this situation and add/update tests.

There were some existing test-cases that were testing
for the wrong behavior, which are now updated.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:19:32 +02:00
Sebastiaan van Stijn
955a6ad95f
builder: fix processing of invalid substitusion syntax
The builder did not detect syntax errors in substitusions in the
Dockerfile, causing those values to be processed incorrectly instead of
producing an error.

Example 1: missing `}`

    docker build --no-cache -<<'EOF'
    FROM busybox
    ARG var=${aaa:-bbb
    RUN echo $var
    EOF

Before:

    Step 3/3 : RUN echo $var
     ---> Running in f06571e77146
    bbb

After:

    Step 2/3 : ARG var=${aaa:-bbb
    failed to process "${aaa:-bbb": syntax error: missing '}'

Example 2: missing closing `}`, no default value

    docker build --no-cache -<<'EOF'
    FROM busybox
    ARG var=${aaa
    RUN echo $var
    EOF

Before:

    Step 2/3 : ARG var=${aaa
    failed to process "${aaa": missing ':' in substitution

After:

    Step 2/3 : ARG var=${aaa
    failed to process "${aaa": syntax error: missing '}'

Example 3: double opening bracket (`{`)

    docker build --no-cache -<<'EOF'
    FROM busybox
    ARG var=${{aaa:-bbb}
    RUN echo $var
    EOF

Before:

    Step 2/3 : ARG var=${{aaa:-bbb}
    failed to process "${{aaa:-bbb}": missing ':' in substitution

After:

    Step 2/3 : ARG var=${{aaa:-bbb}
    failed to process "${{aaa:-bbb}": syntax error: bad substitution

Example 4: double opening bracket (`{`), no default value

    docker build --no-cache -<<'EOF'
    FROM busybox
    ARG var=${{aaa}
    RUN echo $var
    EOF

Before:

    Step 2/3 : ARG var=${{aaa}
    failed to process "${{aaa}": missing ':' in substitution

After:

    Step 2/3 : ARG var=${{aaa}
    failed to process "${{aaa}": syntax error: bad substitution

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:19:17 +02:00
Sebastiaan van Stijn
f23c00d870
Various code-cleanup
remove unnescessary import aliases, brackets, and so on.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-23 17:50:54 +02:00
Yong Tang
9bd5d9912f
Merge pull request #37110 from thaJeztah/remove-testutil-errorcontains
Replace deprecated testutil.ErrorContains()
2018-05-21 11:04:45 -07:00
Sebastiaan van Stijn
8974fd47c7
Merge pull request #37011 from arm64b/ReAdd-LABEL-command-4-target-option
Construct and add 'LABEL' command from 'label' option to last stage
2018-05-21 00:15:02 +02:00
Sebastiaan van Stijn
55bebbaecf
Replace deprecated testutil.ErrorContains()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-21 00:13:04 +02:00
Eli Uriegas
80bc8b4766 Remove references to old release process
This hasn't been the way to release Docker for the past year so let's
just remove them altogether

Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
2018-05-18 18:28:43 +00:00
Dennis Chen
c7b543164d Remove unused 'label' related functions
Since we use `NewLabelCommand()` instead of `addNodesForLabelOption()`
to create the 'LABEL' commands from '--label' options, so all the related
functions should be removed.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-05-17 09:41:06 +08:00
Dennis Chen
9c238ebd55 Add 'LABEL' command from '--label' to the last stage
This PR is tring to fix issue #36996.

Currently for multi-stage build, if `--target` specified, the `--label` option
will be ignored. The root cause is the last stage build will remove the `LABEL`
command(s) node created from the `--label` option. In order to address this issue,
we can create `LABEL` command(s) and add it/tem to the last stage.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-05-17 09:40:19 +08:00
Priya Wadhwa
593255ffb0
Export BuildArgs
Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
2018-05-08 14:57:26 -07:00
Dennis Chen
7c0570473c Fix the target name issue for multi-stage build
This PR is trying to fix issue #36956.

The stage name is case-insensitive by design, so we should use
`strings.EqualFold()` as the comparison method to eliminate the
case sensitive noise.

Also we need to return a pre-defined error code order to avoid below
message like:
"FIXME: Got an API for which error does not match any expected type!!!:
failed to reach build target dev in Dockerfile"

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-05-01 10:28:40 +08:00
Kir Kolyshkin
7d62e40f7e Switch from x/net/context -> context
Since Go 1.7, context is a standard package. Since Go 1.9, everything
that is provided by "x/net/context" is a couple of type aliases to
types in "context".

Many vendored packages still use x/net/context, so vendor entry remains
for now.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-23 13:52:44 -07:00
Vincent Demeester
a7999aaa53
Skip some tests requires root uid when run as user
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-23 10:14:39 +02:00
John Stephens
29fc64b590
Merge pull request #35089 from Microsoft/jjh/fromplatformbuilder
LCOW - Change platform parser directive to FROM statement flag
2018-03-26 14:17:49 -07:00
John Howard
14429056d3 Builder: Review feedback
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:36 -07:00
John Howard
317513d698 Builder: Fix CI issues
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:36 -07:00
John Howard
69fa84bc3d Builder: Plumbing through platform in FROM statement
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:36 -07:00
John Howard
7f0c2d23e1 Builder - add --platform to FROM statement
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:36 -07:00
John Howard
9cae03900f Builder - Parser. Remove platform parser directive
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:35 -07:00
John Howard
735e5d22b7 Builder - dockerfile - just use API for now, and unit test fix
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:35 -07:00