Commit graph

1220 commits

Author SHA1 Message Date
Tonis Tiigi
0289aa436c builder-next: support runtime platform detection
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 00:33:47 -07:00
Tonis Tiigi
b8385c98e8 builder-next: support for inline cache from local images
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 00:33:47 -07:00
Tonis Tiigi
ddaba80467 builder-next: fixes for inline cache support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 00:33:46 -07:00
Tonis Tiigi
7ac4e33e63 builder-next: fixes for updated buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 00:33:46 -07:00
Tonis Tiigi
5166013119 vendor: update buildkit and fsutil
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 00:04:15 -07:00
John Howard
20833b06a0 Windows: (WCOW) Generate OCI spec that remote runtime can escape
Signed-off-by: John Howard <jhoward@microsoft.com>

Also fixes https://github.com/moby/moby/issues/22874

This commit is a pre-requisite to moving moby/moby on Windows to using
Containerd for its runtime.

The reason for this is that the interface between moby and containerd
for the runtime is an OCI spec which must be unambigious.

It is the responsibility of the runtime (runhcs in the case of
containerd on Windows) to ensure that arguments are escaped prior
to calling into HCS and onwards to the Win32 CreateProcess call.

Previously, the builder was always escaping arguments which has
led to several bugs in moby. Because the local runtime in
libcontainerd had context of whether or not arguments were escaped,
it was possible to hack around in daemon/oci_windows.go with
knowledge of the context of the call (from builder or not).

With a remote runtime, this is not possible as there's rightly
no context of the caller passed across in the OCI spec. Put another
way, as I put above, the OCI spec must be unambigious.

The other previous limitation (which leads to various subtle bugs)
is that moby is coded entirely from a Linux-centric point of view.

Unfortunately, Windows != Linux. Windows CreateProcess uses a
command line, not an array of arguments. And it has very specific
rules about how to escape a command line. Some interesting reading
links about this are:

https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/
https://stackoverflow.com/questions/31838469/how-do-i-convert-argv-to-lpcommandline-parameter-of-createprocess
https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments?view=vs-2017

For this reason, the OCI spec has recently been updated to cater
for more natural syntax by including a CommandLine option in
Process.

What does this commit do?

Primary objective is to ensure that the built OCI spec is unambigious.

It changes the builder so that `ArgsEscaped` as commited in a
layer is only controlled by the use of CMD or ENTRYPOINT.

Subsequently, when calling in to create a container from the builder,
if follows a different path to both `docker run` and `docker create`
using the added `ContainerCreateIgnoreImagesArgsEscaped`. This allows
a RUN from the builder to control how to escape in the OCI spec.

It changes the builder so that when shell form is used for RUN,
CMD or ENTRYPOINT, it builds (for WCOW) a more natural command line
using the original as put by the user in the dockerfile, not
the parsed version as a set of args which loses fidelity.
This command line is put into args[0] and `ArgsEscaped` is set
to true for CMD or ENTRYPOINT. A RUN statement does not commit
`ArgsEscaped` to the commited layer regardless or whether shell
or exec form were used.
2019-03-12 18:41:55 -07:00
Sebastiaan van Stijn
6d87f19142
builder: fix COPY --from should preserve ownership
When copying between stages, or copying from an image,
ownership of the copied files should not be changed, unless
the `--chown` option is set (in which case ownership of copied
files should be updated to the specified user/group).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-03-13 00:55:04 +01:00
Tonis Tiigi
f9b9d5f584 builder-next: fixes for rootless mode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-28 10:44:21 -08:00
linuxmercedes
64466b0cd9 Convert parse errors to more informative format
- Wrap parse errors in errdefs.InvalidParameters
- Include dockerfile in error names

Signed-off-by: Natasha Jarus <linuxmercedes@gmail.com>
2019-01-14 20:01:00 -06:00
Tonis Tiigi
bcf1967dd0 builder: delete sandbox in a goroutine for performance
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-11-28 22:55:39 -08:00
Tonis Tiigi
1ad272c7e4 builder: set exernalkey option for faster hook processing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-11-28 22:28:57 -08:00
Sebastiaan van Stijn
2483e03531
Merge pull request #38238 from tiborvass/fix-38208
builder: deprecate prune filter `unused-for` in favor of `until`
2018-11-21 23:36:14 +01:00
Tibor Vass
62923f29f5 builder: ignore label and label! prune filters
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-11-21 01:39:53 +00:00
Tibor Vass
369da264ba builder: deprecate prune filter unused-for in favor of until
This is to keep the UX consistent. `unused-for` is still accepted and a synonym.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-11-21 01:39:53 +00:00
Tonis Tiigi
effb2bd9d2 builder: avoid unset credentials in containerd
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-11-19 17:40:08 -08:00
Tibor Vass
48620057be builder: fix bugs when pruning buildkit cache with filters
Only the filters the user specified should be added as cache filters to buildkit.
Make an AND operation of the provided filters.
ID filter now does prefix-matching.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-11-05 23:15:35 +00:00
Tonis Tiigi
2732fe527f builder: fix duplicate mount release
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-22 17:24:20 -07:00
Tonis Tiigi
c693d45acf builder: fix private pulls on buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-17 17:45:44 -07:00
Akihiro Suda
b5ed4ebe06
Merge pull request #36537 from Microsoft/jjh/lcow-log-stderr
LCOW: Log stderr on failures to ease diagnosis
2018-10-06 11:05:55 +09:00
Akihiro Suda
837b9c6214 bump up buildkit
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-10-05 18:33:03 +09:00
John Howard
63f9c7784b LCOW: Log stderr on failures
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-26 13:23:04 -07:00
Tibor Vass
34eede0296 Remove 'docker-' prefix for containerd and runc binaries
This allows to run the daemon in environments that have upstream containerd installed.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-24 21:49:03 +00:00
Yong Tang
fb7883e980
Merge pull request #37846 from tiborvass/buildkit-gc
builder: use buildkit's GC for build cache
2018-09-22 10:00:06 -07:00
Tibor Vass
4a776d0ca7 builder: use buildkit's GC for build cache
This allows users to configure the buildkit GC.

The following enables the default GC:
```
{
  "builder": {
    "gc": {
      "enabled": true
    }
  }
}
```

The default GC policy has a simple config:
```
{
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "30GB"
    }
  }
}
```

A custom GC policy can be used instead by specifying a list of cache prune rules:
```
{
  "builder": {
    "gc": {
      "enabled": true,
      "policy": [
        {"keepStorage": "512MB", "filter": ["unused-for=1400h"]]},
        {"keepStorage": "30GB", "all": true}
      ]
    }
  }
}
```

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-21 22:06:00 +00:00
Tibor Vass
d0f00bc1fb builder: vendor buildkit to 39404586a50d1b9d0fb1c578cf0f4de7bdb7afe5
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-21 20:43:26 +00:00
Anda Xu
171d51c861 add support of registry-mirrors and insecure-registries to buildkit
Signed-off-by: Anda Xu <anda.xu@docker.com>
2018-09-20 11:53:02 -07:00
Akihiro Suda
5adee401d1
Merge pull request #37843 from Microsoft/jjh/boltdb
Revendor libnetwork @ 36d3bed0e9
2018-09-14 23:22:36 +09:00
Sebastiaan van Stijn
3735ea7b76
Merge pull request #37741 from mooncak/fix_typos
fix typos in experimental.go and buildargs.go
2018-09-14 11:53:54 +02:00
John Howard
5c20890b9b boltdb/bolt --> go.etcd.io/bbolt
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-13 12:38:39 -07:00
Anda Xu
54b3af4c7d update vendor
Signed-off-by: Anda Xu <anda.xu@docker.com>
2018-09-07 17:48:41 -07:00
Anda Xu
d52485c2f9 propagate the dockerd cgroup-parent config to buildkitd
Signed-off-by: Anda Xu <anda.xu@docker.com>
2018-09-07 17:48:41 -07:00
Tõnis Tiigi
4842f7a867
Merge pull request #37738 from tiborvass/remove-unused-field-in-builder
builder: remove unused netnsRoot field in builder-next
2018-09-06 13:33:35 -07:00
Tonis Tiigi
354c241041 builder: implement ref checker
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-01 23:40:06 +00:00
Tibor Vass
d47435a004 builder: fix pruning all cache
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-01 22:01:17 +00:00
Tibor Vass
8ff7847d1c builder: add prune options to the API
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-01 22:01:17 +00:00
mooncake
5160003b4d fix typos
Signed-off-by: mooncake <xcoder@tenxcloud.com>
Signed-off-by: Bily Zhang <xcoder@tenxcloud.com>
2018-09-01 21:26:38 +08:00
Tibor Vass
8ab9e78ee4 builder: remove unused netnsRoot field in builder-next
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-31 19:09:52 +00:00
Tibor Vass
dc7e472db9 builder: fix bridge networking when using buildkit
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-23 05:20:01 +00:00
Tibor Vass
16084ea8c8 builder: temporarily disable bridge networking when using buildkit
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-22 03:29:17 +00:00
Tibor Vass
4a40f921d3 Disable buildkit's subreaper until the issue is understood
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-20 18:55:01 +00:00
Tonis Tiigi
d46fa93cb6 builder: enable add-host for buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-20 18:55:01 +00:00
Tonis Tiigi
c6c680ad5b builder: allow setting host/none network mode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-20 18:55:01 +00:00
Tonis Tiigi
d6424a088d builder: setup code for a bridge networking
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-20 18:55:01 +00:00
John Stephens
8613b34a7e
Merge pull request #37659 from Microsoft/jjh/37356
LCOW: Mount to short container paths to avoid command-line length limit
2018-08-17 15:48:35 -07:00
Yong Tang
14d5569f19
Merge pull request #37563 from Microsoft/jjh/fix-vso17531561v2
LCOW: Ensure platform is populated on COPY/ADD
2018-08-17 12:51:17 -07:00
John Stephens
b3e9f7b13b
Merge pull request #35521 from salah-khan/35507
Add --chown flag support for ADD/COPY commands for Windows
2018-08-17 11:31:16 -07:00
Tibor Vass
7c1c8f1fe2 builder: implement PullParent option with buildkit
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-17 08:40:58 +00:00
John Howard
dffc966165 LCOW: Capture stderr on external process. Log actual error rather than throwaway
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-08-16 19:20:14 -07:00
Akihiro Suda
080c27dd13
Merge pull request #37597 from tiborvass/fix-cancel-bug
builder: do not cancel buildkit status request
2018-08-16 12:42:35 +09:00
Salahuddin Khan
763d839261 Add ADD/COPY --chown flag support to Windows
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.

NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.

The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>

On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.

Signed-off-by: Salahuddin Khan <salah@docker.com>
2018-08-13 21:59:11 -07:00