Commit graph

1211 commits

Author SHA1 Message Date
Tonis Tiigi
bcf1967dd0 builder: delete sandbox in a goroutine for performance
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-11-28 22:55:39 -08:00
Tonis Tiigi
1ad272c7e4 builder: set exernalkey option for faster hook processing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-11-28 22:28:57 -08:00
Sebastiaan van Stijn
2483e03531
Merge pull request #38238 from tiborvass/fix-38208
builder: deprecate prune filter `unused-for` in favor of `until`
2018-11-21 23:36:14 +01:00
Tibor Vass
62923f29f5 builder: ignore label and label! prune filters
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-11-21 01:39:53 +00:00
Tibor Vass
369da264ba builder: deprecate prune filter unused-for in favor of until
This is to keep the UX consistent. `unused-for` is still accepted and a synonym.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-11-21 01:39:53 +00:00
Tonis Tiigi
effb2bd9d2 builder: avoid unset credentials in containerd
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-11-19 17:40:08 -08:00
Tibor Vass
48620057be builder: fix bugs when pruning buildkit cache with filters
Only the filters the user specified should be added as cache filters to buildkit.
Make an AND operation of the provided filters.
ID filter now does prefix-matching.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-11-05 23:15:35 +00:00
Tonis Tiigi
2732fe527f builder: fix duplicate mount release
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-22 17:24:20 -07:00
Tonis Tiigi
c693d45acf builder: fix private pulls on buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-17 17:45:44 -07:00
Akihiro Suda
b5ed4ebe06
Merge pull request #36537 from Microsoft/jjh/lcow-log-stderr
LCOW: Log stderr on failures to ease diagnosis
2018-10-06 11:05:55 +09:00
Akihiro Suda
837b9c6214 bump up buildkit
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-10-05 18:33:03 +09:00
John Howard
63f9c7784b LCOW: Log stderr on failures
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-26 13:23:04 -07:00
Tibor Vass
34eede0296 Remove 'docker-' prefix for containerd and runc binaries
This allows to run the daemon in environments that have upstream containerd installed.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-24 21:49:03 +00:00
Yong Tang
fb7883e980
Merge pull request #37846 from tiborvass/buildkit-gc
builder: use buildkit's GC for build cache
2018-09-22 10:00:06 -07:00
Tibor Vass
4a776d0ca7 builder: use buildkit's GC for build cache
This allows users to configure the buildkit GC.

The following enables the default GC:
```
{
  "builder": {
    "gc": {
      "enabled": true
    }
  }
}
```

The default GC policy has a simple config:
```
{
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "30GB"
    }
  }
}
```

A custom GC policy can be used instead by specifying a list of cache prune rules:
```
{
  "builder": {
    "gc": {
      "enabled": true,
      "policy": [
        {"keepStorage": "512MB", "filter": ["unused-for=1400h"]]},
        {"keepStorage": "30GB", "all": true}
      ]
    }
  }
}
```

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-21 22:06:00 +00:00
Tibor Vass
d0f00bc1fb builder: vendor buildkit to 39404586a50d1b9d0fb1c578cf0f4de7bdb7afe5
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-21 20:43:26 +00:00
Anda Xu
171d51c861 add support of registry-mirrors and insecure-registries to buildkit
Signed-off-by: Anda Xu <anda.xu@docker.com>
2018-09-20 11:53:02 -07:00
Akihiro Suda
5adee401d1
Merge pull request #37843 from Microsoft/jjh/boltdb
Revendor libnetwork @ 36d3bed0e9
2018-09-14 23:22:36 +09:00
Sebastiaan van Stijn
3735ea7b76
Merge pull request #37741 from mooncak/fix_typos
fix typos in experimental.go and buildargs.go
2018-09-14 11:53:54 +02:00
John Howard
5c20890b9b boltdb/bolt --> go.etcd.io/bbolt
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-13 12:38:39 -07:00
Anda Xu
54b3af4c7d update vendor
Signed-off-by: Anda Xu <anda.xu@docker.com>
2018-09-07 17:48:41 -07:00
Anda Xu
d52485c2f9 propagate the dockerd cgroup-parent config to buildkitd
Signed-off-by: Anda Xu <anda.xu@docker.com>
2018-09-07 17:48:41 -07:00
Tõnis Tiigi
4842f7a867
Merge pull request #37738 from tiborvass/remove-unused-field-in-builder
builder: remove unused netnsRoot field in builder-next
2018-09-06 13:33:35 -07:00
Tonis Tiigi
354c241041 builder: implement ref checker
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-01 23:40:06 +00:00
Tibor Vass
d47435a004 builder: fix pruning all cache
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-01 22:01:17 +00:00
Tibor Vass
8ff7847d1c builder: add prune options to the API
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-09-01 22:01:17 +00:00
mooncake
5160003b4d fix typos
Signed-off-by: mooncake <xcoder@tenxcloud.com>
Signed-off-by: Bily Zhang <xcoder@tenxcloud.com>
2018-09-01 21:26:38 +08:00
Tibor Vass
8ab9e78ee4 builder: remove unused netnsRoot field in builder-next
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-31 19:09:52 +00:00
Tibor Vass
dc7e472db9 builder: fix bridge networking when using buildkit
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-23 05:20:01 +00:00
Tibor Vass
16084ea8c8 builder: temporarily disable bridge networking when using buildkit
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-22 03:29:17 +00:00
Tibor Vass
4a40f921d3 Disable buildkit's subreaper until the issue is understood
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-20 18:55:01 +00:00
Tonis Tiigi
d46fa93cb6 builder: enable add-host for buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-20 18:55:01 +00:00
Tonis Tiigi
c6c680ad5b builder: allow setting host/none network mode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-20 18:55:01 +00:00
Tonis Tiigi
d6424a088d builder: setup code for a bridge networking
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-20 18:55:01 +00:00
John Stephens
8613b34a7e
Merge pull request #37659 from Microsoft/jjh/37356
LCOW: Mount to short container paths to avoid command-line length limit
2018-08-17 15:48:35 -07:00
Yong Tang
14d5569f19
Merge pull request #37563 from Microsoft/jjh/fix-vso17531561v2
LCOW: Ensure platform is populated on COPY/ADD
2018-08-17 12:51:17 -07:00
John Stephens
b3e9f7b13b
Merge pull request #35521 from salah-khan/35507
Add --chown flag support for ADD/COPY commands for Windows
2018-08-17 11:31:16 -07:00
Tibor Vass
7c1c8f1fe2 builder: implement PullParent option with buildkit
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-17 08:40:58 +00:00
John Howard
dffc966165 LCOW: Capture stderr on external process. Log actual error rather than throwaway
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-08-16 19:20:14 -07:00
Akihiro Suda
080c27dd13
Merge pull request #37597 from tiborvass/fix-cancel-bug
builder: do not cancel buildkit status request
2018-08-16 12:42:35 +09:00
Salahuddin Khan
763d839261 Add ADD/COPY --chown flag support to Windows
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.

NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.

The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>

On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.

Signed-off-by: Salahuddin Khan <salah@docker.com>
2018-08-13 21:59:11 -07:00
Tibor Vass
f1ef16c736 builder: do not cancel buildkit status request
This fixes a bug when the build results in an error and instead of sending
the logs of the container to the client, it signals a cancelation.

The context should not be wired to the status request, as only the
build requests need to be canceled.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-10 17:29:32 +00:00
Akihiro Suda
4122eb1a43
Merge pull request #37609 from seratch/fix-typos
Fix the several typos detected by github.com/client9/misspell
2018-08-09 08:48:05 +09:00
Kazuhiro Sera
1e49fdcafc Fix the several typos detected by github.com/client9/misspell
Signed-off-by: Kazuhiro Sera <seratch@gmail.com>
2018-08-09 00:45:00 +09:00
Tibor Vass
c53d7e7104 builder: fix compilation errors after vendoring
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-08-07 06:03:10 +00:00
John Howard
ae6b5828ce LCOW: Ensure platform is populated on COPY/ADD
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-07-30 12:49:33 -07:00
Tibor Vass
9ebed53c54
Merge pull request #37439 from tiborvass/vendor-buildkit
Set BuildKit's ExportedProduct variable to show useful errors in the future
2018-07-17 12:10:23 -07:00
Tibor Vass
81599222fc validate: please vet
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-07-16 21:41:54 +00:00
Tibor Vass
0ab7c1c5ba vendor: buildkit to 98f1604134f945d48538ffca0e18662337b4a850
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-07-16 21:41:53 +00:00
Yong Tang
492545e139
Merge pull request #37396 from yui-knk/error_when_base_name_resolved_to_blank
Return error if basename is expanded to blank
2018-07-15 15:42:57 -07:00
Yuichiro Kaneko
c9542d313e Return error if basename is expanded to blank
Fix: https://github.com/moby/moby/issues/37325

Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
2018-07-15 22:29:03 +09:00
Sebastiaan van Stijn
35f7e7c50a
Fix golint issues
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-07-11 22:19:03 +02:00
Sebastiaan van Stijn
f0585d04d0
Merge pull request #37413 from yui-knk/do_not_Healthcheck_RUN_command
Ensure RUN instruction to run without Healthcheck
2018-07-11 20:38:03 +02:00
Tonis Tiigi
ffa7233d15 builder: fix duplicate calls to mountable
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-10 15:21:29 -07:00
Yuichiro Kaneko
44e08d8a7d Ensure RUN instruction to run without Healthcheck
Before this commit Healthcheck run if HEALTHCHECK
instruction appears before RUN instruction.
By passing `withoutHealthcheck` to `copyRunConfig`,
always RUN instruction run without Healthcheck.

Fix: https://github.com/moby/moby/issues/37362

Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
2018-07-08 16:49:04 +09:00
Sebastiaan van Stijn
8d56fa0bb7
Merge pull request #37380 from yui-knk/update_doc_Detect
Update documents of `Detect`
2018-07-05 14:49:18 +02:00
Tibor Vass
b711437bbd
Merge pull request #37381 from tiborvass/api-platforms-as-string-for-temp
api: Change Platform field back to string (temporary workaround)
2018-07-03 17:54:42 -07:00
Tibor Vass
4b3df087a2
Merge pull request #37385 from tonistiigi/update-buildkit
vendor: update buildkit to 9acf51e491
2018-07-03 17:33:54 -07:00
Tibor Vass
facad55744 api: Change Platform field back to string (temporary workaround)
This partially reverts https://github.com/moby/moby/pull/37350

Although specs.Platform is desirable in the API, there is more work
to be done on helper functions, namely containerd's platforms.Parse
that assumes the default platform of the Go runtime.

That prevents a client to use the recommended Parse function to
retrieve a specs.Platform object.

With this change, no parsing is expected from the client.

Signed-off-by: Tibor Vass <tibor@docker.com>
2018-07-03 22:33:42 +00:00
Sebastiaan van Stijn
1da7d2eebf
Merge pull request #37368 from tiborvass/buildkit-iidfile
builder: return image ID in API when using buildkit
2018-07-04 00:20:25 +02:00
Tonis Tiigi
6f7dd9428e builder: do not send duplicate status for completed jobs
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-03 14:47:38 -07:00
Tibor Vass
ca8022ec63 builder: return image ID in API when using buildkit
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-07-03 19:11:02 +00:00
Yuichiro Kaneko
0bbd476ceb Update documents of Detect
By 0296797f0f, `progressReader`
and `remoteURL` were removed from arguments. So developers who
use `Detect` not need to care about when `ProgressReaderFunc`
is used.

Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
2018-07-03 09:13:25 +09:00
Yuichiro Kaneko
9b374801ac Update documents of dispatchAdd
`ADD` does not support git.

Ref: https://github.com/moby/moby/issues/14704#issuecomment-298134333

Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
2018-07-03 08:51:07 +09:00
Sebastiaan van Stijn
1d7132e3d2
Merge pull request #37356 from Microsoft/jjh/new37316
LCOW: lazycontext: Use correct lstat, fix archive check
2018-06-28 12:32:31 +02:00
Tonis Tiigi
f099771665 system: add back lcow validation function
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-27 15:24:26 -07:00
Tonis Tiigi
0b50d6c315 builder: update platform support to puller
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-27 14:59:36 -07:00
Tonis Tiigi
337ba71fc1 distribution: fix passing platform struct to puller
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-27 14:59:31 -07:00
John Howard
a5aed699cf LCOW: lazycontext: Use correct lstat, fix archive check
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-06-27 13:29:21 -07:00
Tonis Tiigi
81f862a1fe api: fix platform type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-26 22:59:17 -07:00
Tonis Tiigi
48b93419dc builder: buildkit rebase update
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-26 22:59:17 -07:00
Dennis Chen
7f334d3acf Initial support for OCI multi-platform image
Add the OCI spec compatible image support in client side.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-06-26 11:39:29 -07:00
John Howard
35193c0e7d LCOW: Auto-select OS
Signed-off-by: John Howard <jhoward@microsoft.com>

Addresses https://github.com/moby/moby/pull/35089#issuecomment-367802698.
This change enables the daemon to automatically select an image under LCOW
that can be used if the API doesn't specify an explicit platform.

For example:

FROM supertest2014/nyan
ADD Dockerfile /

And docker build . will download the linux image (not a multi-manifest image)

And similarly docker pull ubuntu will match linux/amd64
2018-06-26 11:37:43 -07:00
John Howard
ca3e3fcf7e Don't fail to start daemon if builder source is not available
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-06-13 09:33:19 -07:00
Vincent Demeester
3845728524
Update tests to use gotest.tools 👼
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-06-13 09:04:30 +02:00
Tonis Tiigi
8ccbc2c40a builder: update ID of trace messages
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:32 -07:00
Tonis Tiigi
f41af1efd0 builder: updates for newer containerd
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:31 -07:00
Tonis Tiigi
0728fb2b63 builder: override history dates from ref metadata
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:31 -07:00
Derek McGowan
96c65a3adf Add support for schema 1 pull
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-10 10:05:31 -07:00
Tonis Tiigi
577732f655 builder: more experimental/windows validation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:30 -07:00
Tonis Tiigi
2567dd9afc builder: fix cancellation context issue
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:30 -07:00
Tonis Tiigi
4cdb6853dc builder: move tagging to exporter
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:30 -07:00
Tonis Tiigi
157b0b30db builder: lint fixes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:29 -07:00
Tonis Tiigi
9328f0479b builder: support for images without layers
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:29 -07:00
Tonis Tiigi
72d10ced7d builder: enable gateway through syntax directive
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:29 -07:00
Tonis Tiigi
a25846a675 builder: add support for separate upload-request
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:29 -07:00
Tonis Tiigi
f3ef8c93d6 builder: protect early progress writes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:29 -07:00
Tibor Vass
3a1da5c534 builder: fix compiling with buildkit on windows and integration tests
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-06-10 10:05:28 -07:00
Tonis Tiigi
92395261b0 builder: add support for building from tarball
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:28 -07:00
Tonis Tiigi
8900e3c758 builder: patch incomplete download handling
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:28 -07:00
Tonis Tiigi
ed651e796c builder: fixes after rebase
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:28 -07:00
Tonis Tiigi
760ecf958b builder: expand prune to buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:27 -07:00
Tonis Tiigi
f0a9e54d20 builder: add usage to snapshotter adapter
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:27 -07:00
Tonis Tiigi
b225258496 builder: export build cache records
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:27 -07:00
Tonis Tiigi
bb68c8132b builder: produce duplicate cache keys on pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:27 -07:00
Tonis Tiigi
b7424599f6 builder: adapter update after vendor update
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:27 -07:00
Tonis Tiigi
46bd229b51 builder: add cache-from support to buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:27 -07:00
Tonis Tiigi
0bddd4ccfe builder: add graceful cancellation endpoint
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:26 -07:00
Tonis Tiigi
22f7caee03 builder: experimental buildkit base
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:26 -07:00
Tonis Tiigi
27fa0e8a7b builder: snapshotter and exporter
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-10 10:05:26 -07:00
Sebastiaan van Stijn
52ea99eda2
Merge pull request #37189 from arm64b/create-intermediate-for-COPY-ADD
Refactor and cleanup the intermediate container creation
2018-06-07 21:20:17 -07:00
Dennis Chen
7f280f6f65 Refactor and cleanup the intermediate container creation
This PR is trying to refactor the `probeAndCreate` and cleanup
related codes based on the refactoring.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-06-04 12:55:49 +08:00
Tonis Tiigi
c9ebd2f13b vendor: use dockerfile parser from buildkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-02 11:10:34 -07:00
Vincent Demeester
3e494c4b93
Merge pull request #37178 from tonistiigi/layer-leak
builder: fix layer leak on multi-stage wildcard copy
2018-05-31 09:39:28 +02:00
Tonis Tiigi
4a18c11bdc builder: fix layer leak on multi-stage wildcard copy
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-30 15:18:40 -07:00
Sebastiaan van Stijn
2628896b5e
Handle multi-digit positional parameters
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:51:11 +02:00
Sebastiaan van Stijn
9654e9b6f8
Add detection of "special parameters" for substitution
Detect Special parameters as defined in
http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_05_02

Treat these as parameters that are not set, instead of
producing an error that a modifier is missing.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:20:22 +02:00
Sebastiaan van Stijn
8687a3f4b8
Add more test-cases for positional parameters
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:20:10 +02:00
Sebastiaan van Stijn
b80e0309d2
Add line-numbers to asserts
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:20:01 +02:00
Sebastiaan van Stijn
334bf3ea76
Fix detection for missing parameter in substitution
`${}`, `${:}` and so on are invalid because there's
no parameter within the brackets; fix detection for
this situation and add/update tests.

There were some existing test-cases that were testing
for the wrong behavior, which are now updated.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:19:32 +02:00
Sebastiaan van Stijn
955a6ad95f
builder: fix processing of invalid substitusion syntax
The builder did not detect syntax errors in substitusions in the
Dockerfile, causing those values to be processed incorrectly instead of
producing an error.

Example 1: missing `}`

    docker build --no-cache -<<'EOF'
    FROM busybox
    ARG var=${aaa:-bbb
    RUN echo $var
    EOF

Before:

    Step 3/3 : RUN echo $var
     ---> Running in f06571e77146
    bbb

After:

    Step 2/3 : ARG var=${aaa:-bbb
    failed to process "${aaa:-bbb": syntax error: missing '}'

Example 2: missing closing `}`, no default value

    docker build --no-cache -<<'EOF'
    FROM busybox
    ARG var=${aaa
    RUN echo $var
    EOF

Before:

    Step 2/3 : ARG var=${aaa
    failed to process "${aaa": missing ':' in substitution

After:

    Step 2/3 : ARG var=${aaa
    failed to process "${aaa": syntax error: missing '}'

Example 3: double opening bracket (`{`)

    docker build --no-cache -<<'EOF'
    FROM busybox
    ARG var=${{aaa:-bbb}
    RUN echo $var
    EOF

Before:

    Step 2/3 : ARG var=${{aaa:-bbb}
    failed to process "${{aaa:-bbb}": missing ':' in substitution

After:

    Step 2/3 : ARG var=${{aaa:-bbb}
    failed to process "${{aaa:-bbb}": syntax error: bad substitution

Example 4: double opening bracket (`{`), no default value

    docker build --no-cache -<<'EOF'
    FROM busybox
    ARG var=${{aaa}
    RUN echo $var
    EOF

Before:

    Step 2/3 : ARG var=${{aaa}
    failed to process "${{aaa}": missing ':' in substitution

After:

    Step 2/3 : ARG var=${{aaa}
    failed to process "${{aaa}": syntax error: bad substitution

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-25 18:19:17 +02:00
Sebastiaan van Stijn
f23c00d870
Various code-cleanup
remove unnescessary import aliases, brackets, and so on.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-23 17:50:54 +02:00
Yong Tang
9bd5d9912f
Merge pull request #37110 from thaJeztah/remove-testutil-errorcontains
Replace deprecated testutil.ErrorContains()
2018-05-21 11:04:45 -07:00
Sebastiaan van Stijn
8974fd47c7
Merge pull request #37011 from arm64b/ReAdd-LABEL-command-4-target-option
Construct and add 'LABEL' command from 'label' option to last stage
2018-05-21 00:15:02 +02:00
Sebastiaan van Stijn
55bebbaecf
Replace deprecated testutil.ErrorContains()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-21 00:13:04 +02:00
Tibor Vass
1b75e17584
Merge pull request #37091 from thaJeztah/fix-linting-errors
Fix some linting issues
2018-05-19 08:39:39 -10:00
Eli Uriegas
80bc8b4766 Remove references to old release process
This hasn't been the way to release Docker for the past year so let's
just remove them altogether

Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
2018-05-18 18:28:43 +00:00
Sebastiaan van Stijn
4f8c870d62
Fix some linting issues
These showed locally when running `make validate`. CI doesn't seem to have the
same (possibly it's disabled in the configuration)

    builder/fscache/fscache.go:618::error: github.com/docker/docker/vendor/github.com/tonistiigi/fsutil.StatInfo composite literal uses unkeyed fields (vet)
    client/swarm_unlock_test.go:44::error: github.com/docker/docker/api/types/swarm.UnlockRequest composite literal uses unkeyed fields (vet)
    client/swarm_unlock_test.go:20::error: github.com/docker/docker/api/types/swarm.UnlockRequest composite literal uses unkeyed fields (vet)
    cmd/dockerd/daemon_unix.go:113::error: github.com/docker/docker/cmd/dockerd/hack.MalformedHostHeaderOverride composite literal uses unkeyed fields (vet)
    cmd/dockerd/daemon_unix.go:110::error: github.com/docker/docker/cmd/dockerd/hack.MalformedHostHeaderOverride composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay/overlay.go:171::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay/overlay.go:413::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay2/overlay.go:203::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay2/overlay.go:584::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/zfs/zfs.go:109::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/zfs/zfs.go:388::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/volumes_windows.go:27::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    integration/service/network_test.go:31::error: github.com/docker/docker/api/types/network.NetworkingConfig composite literal uses unkeyed fields (vet)
    api/server/server.go:129:10⚠️ should not use basic type string as key in context.WithValue (golint)
    integration/service/network_test.go:54::error: github.com/docker/docker/api/types/network.NetworkingConfig composite literal uses unkeyed fields (vet)
    libcontainerd/client_daemon_linux.go:61::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    libcontainerd/client_daemon_linux.go:74::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    pkg/archive/archive_windows.go:76::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    plugin/manager_linux.go:56::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-17 19:28:27 +02:00
Dennis Chen
c7b543164d Remove unused 'label' related functions
Since we use `NewLabelCommand()` instead of `addNodesForLabelOption()`
to create the 'LABEL' commands from '--label' options, so all the related
functions should be removed.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-05-17 09:41:06 +08:00
Dennis Chen
9c238ebd55 Add 'LABEL' command from '--label' to the last stage
This PR is tring to fix issue #36996.

Currently for multi-stage build, if `--target` specified, the `--label` option
will be ignored. The root cause is the last stage build will remove the `LABEL`
command(s) node created from the `--label` option. In order to address this issue,
we can create `LABEL` command(s) and add it/tem to the last stage.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-05-17 09:40:19 +08:00
Priya Wadhwa
593255ffb0
Export BuildArgs
Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
2018-05-08 14:57:26 -07:00
Dennis Chen
7c0570473c Fix the target name issue for multi-stage build
This PR is trying to fix issue #36956.

The stage name is case-insensitive by design, so we should use
`strings.EqualFold()` as the comparison method to eliminate the
case sensitive noise.

Also we need to return a pre-defined error code order to avoid below
message like:
"FIXME: Got an API for which error does not match any expected type!!!:
failed to reach build target dev in Dockerfile"

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-05-01 10:28:40 +08:00
Kir Kolyshkin
7d62e40f7e Switch from x/net/context -> context
Since Go 1.7, context is a standard package. Since Go 1.9, everything
that is provided by "x/net/context" is a couple of type aliases to
types in "context".

Many vendored packages still use x/net/context, so vendor entry remains
for now.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-23 13:52:44 -07:00
Vincent Demeester
a7999aaa53
Skip some tests requires root uid when run as user
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-23 10:14:39 +02:00
Brian Goff
9d46c4c138 Support cancellation in directory.Size()
Makes sure that if the user cancels a request that the daemon stops
trying to traverse a directory.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-03-29 15:49:15 -04:00
John Stephens
29fc64b590
Merge pull request #35089 from Microsoft/jjh/fromplatformbuilder
LCOW - Change platform parser directive to FROM statement flag
2018-03-26 14:17:49 -07:00
John Howard
14429056d3 Builder: Review feedback
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:36 -07:00
John Howard
317513d698 Builder: Fix CI issues
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:36 -07:00
John Howard
69fa84bc3d Builder: Plumbing through platform in FROM statement
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:36 -07:00
John Howard
7f0c2d23e1 Builder - add --platform to FROM statement
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:36 -07:00
John Howard
9cae03900f Builder - Parser. Remove platform parser directive
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:35 -07:00
John Howard
735e5d22b7 Builder - dockerfile - just use API for now, and unit test fix
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:35 -07:00
John Howard
98dd1fdca1 Builder - parser - remove OS
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-03-19 14:29:35 -07:00
Daniel Nephin
c9e52bd0da Post migration assertion fixes
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-03-16 11:03:46 -04:00
Daniel Nephin
6be0f70983 Automated migration using
gty-migrate-from-testify --ignore-build-tags

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-03-16 11:03:43 -04:00
Vincent Demeester
5b124d7422
Merge pull request #36424 from liubin/fix-typo
Fix typos in builder and client
2018-02-27 10:24:20 +01:00
Vincent Demeester
600475715e
Merge pull request #36338 from tonistiigi/fix-copy-leak
builder: fix layer lifecycle leak
2018-02-26 22:36:40 +01:00
John Howard
8c52560ea4 Windows: Pass back system errors on container exit
Signed-off-by: John Howard <jhoward@microsoft.com>

While debugging #32838, it was found (https://github.com/moby/moby/issues/32838#issuecomment-356005845) that the utility VM in some circumstances was crashing. Unfortunately, this was silently thrown away, and as far as the build step (also applies to docker run) was concerned, the exit code was zero and the error was thrown away. Windows containers operate differently to containers on Linux, and there can be legitimate system errors during container shutdown after the init process exits. This PR handles this and passes the error all the way back to the client, and correctly causes a build step running a container which hits a system error to fail, rather than blindly trying to keep going, assuming all is good, and get a subsequent failure on a commit.

With this change, assuming an error occurs, here's an example of a failure which previous was reported as a commit error:

```
The command 'powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; Install-WindowsFeature -Name Web-App-Dev ;   Install-WindowsFeature -Name ADLDS;   Install-WindowsFeature -Name Web-Mgmt-Compat;   Install-WindowsFeature -Name Web-Mgmt-Service;   Install-WindowsFeature -Name Web-Metabase;   Install-WindowsFeature -Name Web-Lgcy-Scripting;   Install-WindowsFeature -Name Web-WMI;   Install-WindowsFeature -Name Web-WHC;   Install-WindowsFeature -Name Web-Scripting-Tools;   Install-WindowsFeature -Name Web-Net-Ext45;   Install-WindowsFeature -Name Web-ASP;   Install-WindowsFeature -Name Web-ISAPI-Ext;   Install-WindowsFeature -Name Web-ISAPI-Filter;   Install-WindowsFeature -Name Web-Default-Doc;   Install-WindowsFeature -Name Web-Dir-Browsing;   Install-WindowsFeature -Name Web-Http-Errors;   Install-WindowsFeature -Name Web-Static-Content;   Install-WindowsFeature -Name Web-Http-Redirect;   Install-WindowsFeature -Name Web-DAV-Publishing;   Install-WindowsFeature -Name Web-Health;   Install-WindowsFeature -Name Web-Http-Logging;   Install-WindowsFeature -Name Web-Custom-Logging;   Install-WindowsFeature -Name Web-Log-Libraries;   Install-WindowsFeature -Name Web-Request-Monitor;   Install-WindowsFeature -Name Web-Http-Tracing;   Install-WindowsFeature -Name Web-Stat-Compression;   Install-WindowsFeature -Name Web-Dyn-Compression;   Install-WindowsFeature -Name Web-Security;   Install-WindowsFeature -Name Web-Windows-Auth;   Install-WindowsFeature -Name Web-Basic-Auth;   Install-WindowsFeature -Name Web-Url-Auth;   Install-WindowsFeature -Name Web-WebSockets;   Install-WindowsFeature -Name Web-AppInit;   Install-WindowsFeature -Name NET-WCF-HTTP-Activation45;   Install-WindowsFeature -Name NET-WCF-Pipe-Activation45;   Install-WindowsFeature -Name NET-WCF-TCP-Activation45;' returned a non-zero code: 4294967295: container shutdown failed: container ba9c65054d42d4830fb25ef55e4ab3287550345aa1a2bb265df4e5bfcd79c78a encountered an error during WaitTimeout: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106)
```

Without this change, it would be incorrectly reported such as in this comment: https://github.com/moby/moby/issues/32838#issuecomment-309621097

```
Step 3/8 : ADD buildtools C:/buildtools
re-exec error: exit status 1: output: time="2017-06-20T11:37:38+10:00" level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\\\?\\C:\\ProgramData\\docker\\windowsfilter\\b41d28c95f98368b73fc192cb9205700e21
6691495c1f9ac79b9b04ec4923ea2 flavour=1 folder=C:\\Windows\\TEMP\\hcs232661915"
hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\?\C:\ProgramData\docker\windowsfilter\b41d28c95f98368b73fc192cb9205700e216691495c1f9ac79b9b04ec4923ea2 flavour=1 folder=C:\Windows\TEMP\hcs232661915
```
2018-02-22 08:53:43 -08:00
Tonis Tiigi
7ad41d53df builder: fix layer lifecycle leak
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-02-20 11:14:53 -08:00
bin liu
ce09932704 Fix typos in builder and client
Signed-off-by: bin liu <liubin0329@gmail.com>
2018-02-10 19:39:26 +08:00
Akihiro Suda
9769ef333f
Merge pull request #36224 from dnephin/refactor-commit
Refactor Daemon.Commit()
2018-02-08 21:02:30 +09:00
Daniel Nephin
daff039049 Refactor commit
The goal of this refactor is to make it easier to integrate buildkit
and containerd snapshotters.

Commit is used from two places (api and build), each calls it
with distinct arguments. Refactored to pull out the common commit
logic and provide different interfaces for each consumer.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-02-07 15:09:06 -05:00
Daniel Nephin
4f0d95fa6e Add canonical import comment
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-02-05 16:51:57 -05:00
Matt Rickard
a634526d14 Move builder shell parser into subpackage
Moves builder/shell_parser and into its own subpackage at builder/shell since it
has no dependencies other than the standard library. This will make it
much easier to vendor for downstream libraries, without pulling all the
dependencies of builder/.

Fixes #36154

Signed-off-by: Matt Rickard <mrick@google.com>
2018-01-30 17:54:39 -08:00
Brian Goff
47a0dcbcd2
Merge pull request #35986 from thaJeztah/bump-containerd-1.0.1-rc1
Bump containerd to 1.0.1 (9b55aab90508bd389d7654c4baf173a981477d55)
2018-01-19 15:48:36 -05:00
John Howard
0cba7740d4 Address feedback from Tonis
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-01-18 12:30:39 -08:00
John Howard
afd305c4b5 LCOW: Refactor to multiple layer-stores based on feedback
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-01-18 08:31:05 -08:00
John Howard
ce8e529e18 LCOW: Re-coalesce stores
Signed-off-by: John Howard <jhoward@microsoft.com>

The re-coalesces the daemon stores which were split as part of the
original LCOW implementation.

This is part of the work discussed in https://github.com/moby/moby/issues/34617,
in particular see the document linked to in that issue.
2018-01-18 08:29:19 -08:00
Sebastiaan van Stijn
d1c34831e9
Fixes for libcontainer changes
Libcontainer no longer provides placeholders for
unsupported platforms, which cause the Windows
builds to fail.

This patch moves features that are not supported
to platform-specific files.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-01-18 10:08:12 +01:00
Yong Tang
c36274da83
Merge pull request #35638 from cpuguy83/error_helpers2
Add helpers to create errdef errors
2018-01-15 10:56:46 -08:00
Sebastiaan van Stijn
b4a6313969
Golint: remove redundant ifs
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-01-15 00:42:25 +01:00
Brian Goff
d453fe35b9 Move api/errdefs to errdefs
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-01-11 21:21:43 -05:00
Brian Goff
87a12421a9 Add helpers to create errdef errors
Instead of having to create a bunch of custom error types that are doing
nothing but wrapping another error in sub-packages, use a common helper
to create errors of the requested type.

e.g. instead of re-implementing this over and over:

```go
type notFoundError struct {
  cause error
}

func(e notFoundError) Error() string {
  return e.cause.Error()
}

func(e notFoundError) NotFound() {}

func(e notFoundError) Cause() error {
  return e.cause
}
```

Packages can instead just do:

```
  errdefs.NotFound(err)
```

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-01-11 21:21:43 -05:00
John Stephens
fff4232899
Merge pull request #35925 from Microsoft/jjh/builder-default-rw-size-bump
Windows: Bump RW layer size
2018-01-04 22:54:38 -08:00
Yong Tang
50ce546450
Merge pull request #35737 from tonistiigi/fix-git-submodules
gitutils: fix checking out submodules
2018-01-05 10:41:04 +08:00
John Howard
deb335d04f Windows: Bump RW layer size
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-01-03 15:59:06 -08:00
Anthony Sottile
0785836c4b Fix environ substitutions in docker commit --change ...
The building machinery was being handed an uninitialized container
Config.  This changes it to use the target container's Config.

Resolves #30538

Signed-off-by: Anthony Sottile <asottile@umich.edu>
2017-12-20 11:03:38 -08:00
Tonis Tiigi
4ea320cb8e gitutils: remove checkout directory on error
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-08 11:58:13 -08:00
Tonis Tiigi
a602525524 gitutils: fix checking out submodules
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-07 14:25:19 -08:00
Anthony Sottile
b6eed3dca2 Change references from test-integration-cli to test-integration
Signed-off-by: Anthony Sottile <asottile@umich.edu>
2017-11-27 19:01:14 -08:00
Sebastiaan van Stijn
7c53e73253
Merge pull request #35429 from dnephin/build-fails-on-long-label
Fix dockerfile parser failing silently on long tokens
2017-11-15 18:48:50 +01:00
Daniel Nephin
9bcd5d2574 Add deepCopyRunConfig for copying buidler runConfig
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-11-13 14:51:17 -05:00
Daniel Nephin
a74cc83345 Fix remote build target as Dockerfile
The test was passing previously because the preamble was already buffered. After
the change to return Scanner.Err() the final read error on the buffer was no
longer being ignored.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-11-09 15:05:26 -05:00
Daniel Nephin
59ad3a36e2 Fix dockerfile parser failing silently on long tokens
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-11-09 15:03:20 -05:00
John Stephens
a97817b673
Set OS on scratch image and prevent panic if empty
Signed-off-by: John Stephens <johnstep@docker.com>
2017-11-06 18:22:15 -08:00
Yong Tang
48694a3f88
Merge pull request #35244 from joyce/joyce/todo-fixes
fix todo for printing error messages
2017-11-04 13:52:45 -07:00
Tonis Tiigi
c6703b722e builder: fix long stream sync
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-11-03 12:53:17 -07:00
Joyce
883ab41ce8 fix todo for printing error messages
Signed-off-by: Joyce <mail@joycejang.com>
2017-11-03 11:35:41 -07:00
Yong Tang
3ab20a87fa
Merge pull request #34271 from tonistiigi/build-symlink-hash
builder: fix build cache hash for broken symlink
2017-10-31 09:38:57 -07:00
Kenfe-Mickael Laventure
ddae20c032
Update libcontainerd to use containerd 1.0
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-10-20 07:11:37 -07:00
Ri Xu
87e8a936e8 Typo fixed and simple code.
Signed-off-by: Ri Xu <xuri@360.net>
2017-10-18 10:26:58 +08:00
Simon Ferquel
172e73a1df Test & Fix build with rm/force-rm matrix
Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
2017-10-12 10:22:40 +02:00
Tonis Tiigi
d5b15f0d3e Fix tests after Hash() behavior change
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-10 14:23:12 -07:00
Yong Tang
513f0e6c26 Merge pull request #35138 from vdemeester/foo
Carry 35078 : Fix TestErrorCases to run all tests
2017-10-09 12:16:16 -07:00
John Howard
d2c41f06c3
Fix TestErrorCases to run all tests
Signed-off-by: John Howard <jhoward@microsoft.com>
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2017-10-09 15:15:58 +02:00
John Howard
d98ecf2d6c LCOW: API change JSON header to string POST parameter
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-10-06 15:26:48 -07:00
John Howard
0380fbff37 LCOW: API: Add platform to /images/create and /build
Signed-off-by: John Howard <jhoward@microsoft.com>

This PR has the API changes described in https://github.com/moby/moby/issues/34617.
Specifically, it adds an HTTP header "X-Requested-Platform" which is a JSON-encoded
OCI Image-spec `Platform` structure.

In addition, it renames (almost all) uses of a string variable platform (and associated)
methods/functions to os. This makes it much clearer to disambiguate with the swarm
"platform" which is really os/arch. This is a stepping stone to getting the daemon towards
fully multi-platform/arch-aware, and makes it clear when "operating system" is being
referred to rather than "platform" which is misleadingly used - sometimes in the swarm
meaning, but more often as just the operating system.
2017-10-06 11:44:18 -07:00
Tonis Tiigi
793ebdbf4b builder: fix build cache hash for broken symlink
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-29 09:27:51 -07:00
Lorenzo Fontana
dff2a23749
Cleaning dead code out of the builder
Signed-off-by: Lorenzo Fontana <lo@linux.com>
2017-09-29 02:55:35 +02:00
Sebastiaan van Stijn
2fd736ac10
Warn on empty continuation lines only, not for comments
Commit 8d1ae76dcb added
deprecation warnings for empty continuation lines,
but also treated comment-only lines as empty.

This patch distinguishes empty continuation lines
from comment-only lines, and only outputs warnings
for the former.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-09-28 00:11:10 +02:00
Sebastiaan van Stijn
76b5ab6f43 Merge pull request #34918 from thaJeztah/update-copy-add-error
Improve error message for COPY missing destination
2017-09-26 18:43:36 +02:00
Sebastiaan van Stijn
5d05a82913
Improve error message for COPY missing destination
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-09-25 14:17:18 +02:00
Tonis Tiigi
d472974902 builder: updates to session after vendor
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-22 09:51:52 -07:00
Simon Ferquel
669c067798 Introduce a typed command system and 2 phase parse/dispatch build
This is a work base to introduce more features like build time
dockerfile optimisations, dependency analysis and parallel build, as
well as a first step to go from a dispatch-inline process to a
frontend+backend process.

Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
2017-09-18 09:48:56 +02:00
Akash Gupta
7a7357dae1 LCOW: Implemented support for docker cp + build
This enables docker cp and ADD/COPY docker build support for LCOW.
Originally, the graphdriver.Get() interface returned a local path
to the container root filesystem. This does not work for LCOW, so
the Get() method now returns an interface that LCOW implements to
support copying to and from the container.

Signed-off-by: Akash Gupta <akagup@microsoft.com>
2017-09-14 12:07:52 -07:00
Yong Tang
027297a60f Add unit test to cover changes.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
2017-09-14 16:17:38 +00:00
Yong Tang
bea0a072d8 Fix build with ADD urls without any sub path
This fix tries to address the issue raised in #34208 where
in Dockerfile an `ADD` followed by an url without any sub path
will cause an error.

The issue is because the temporary filename relies on the sub path.

An integration test has been added.

This fix fixes #34208.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
2017-09-14 16:17:38 +00:00
Daniel Nephin
f7f101d57e Add gosimple linter
Update gometalinter

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-09-12 12:09:59 -04:00
Daniel Nephin
09652bf878 Add ineffassign linter
Also enable GC in linting to reduce memory usage.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-09-08 18:23:21 -04:00
Yong Tang
cb952bf006 Merge pull request #34625 from dnephin/more-linters
Add interfacer and unconvert linters
2017-09-01 08:46:08 -07:00
Tõnis Tiigi
a1183dda57 Merge pull request #34263 from estesp/chown-flag-add-copy
Add --chown flag to ADD/COPY commands
2017-08-28 09:50:44 -07:00
Daniel Nephin
e62b2d410c Move ErrorContains to an internal package.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-08-25 12:04:58 -04:00
Daniel Nephin
2f5f0af3fd Add unconvert linter
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-08-24 15:08:31 -04:00
Daniel Nephin
709bf8b7bc Add interfacer linter
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-08-24 15:08:26 -04:00
Daniel Nephin
60672382c7 Update tests to use gotestyourself/fs
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-08-23 17:25:00 -04:00
Victor Vieux
59e1565565 Merge pull request #34600 from dnephin/remove-pkg-testutil
Remove pkg/testutil/utils.go
2017-08-23 09:38:20 -07:00
Stephen J Day
ae8dbeaeed
*: normalize the use of normalize
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-22 15:25:31 -07:00
Daniel Nephin
4f304e72a2 Remove testutil.ReadBody
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-08-22 17:15:26 -04:00
Phil Estes
19a29f6fcf
Rebase --chown function for ADD/COPY
Rebases and completes initial PR for (prior: --user) --chown flag for
ADD/COPY commands in Dockerfile.

Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2017-08-22 13:39:25 -04:00
Daniel Nephin
9b47b7b151 Fix golint errors.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-08-18 14:23:44 -04:00
John Howard
9fa449064c LCOW: WORKDIR correct handling
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-08-17 15:29:17 -07:00
Brian Goff
ebcb7d6b40 Remove string checking in API error handling
Use strongly typed errors to set HTTP status codes.
Error interfaces are defined in the api/errors package and errors
returned from controllers are checked against these interfaces.

Errors can be wraeped in a pkg/errors.Causer, as long as somewhere in the
line of causes one of the interfaces is implemented. The special error
interfaces take precedence over Causer, meaning if both Causer and one
of the new error interfaces are implemented, the Causer is not
traversed.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2017-08-15 16:01:11 -04:00
John Howard
8af4db6f00 Merge pull request #34272 from dmcgowan/update-logrus
Update logrus to v1.0.1 (Sirupsen -> sirupsen)
2017-08-01 18:53:33 -07:00
Yong Tang
f7d09a0a9b Merge pull request #33704 from ecnerwala/33701-shallow-clone
Fix shallow git clone in docker-build
2017-08-01 10:17:08 -07:00
Derek McGowan
1009e6a40b
Update logrus to v1.0.1
Fixes case sensitivity issue

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-31 13:16:46 -07:00
Tibor Vass
41445a4745 Remove client/session package, depend on buildkit's session package
gofmt -w -r '"github.com/docker/docker/client/session" -> "github.com/moby/buildkit/session"'
gofmt -w -r '"github.com/docker/docker/client/session/filesync" -> "github.com/moby/buildkit/session/filesync"'

Signed-off-by: Tibor Vass <tibor@docker.com>
2017-07-28 16:34:23 -07:00
Andrew He
85afbbc2ed Fix shallow git clone in docker-build
If the HEAD request fails, use a GET request to properly test if git
server is smart-http.

Signed-off-by: Andrew He <he.andrew.mail@gmail.com>
2017-07-25 13:20:59 -07:00
Kara Alexandra
858fad3795
Add --chown flag to Dockerfile ADD and COPY
Rebased by @estesp

Signed-off-by: Kara Alexandra <kalexandra@us.ibm.com>
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2017-07-24 16:25:04 -04:00
allencloud
87b4dc2002 return prune data when context canceled
Signed-off-by: allencloud <allen.sun@daocloud.io>
2017-07-10 10:06:24 +08:00
Sebastiaan van Stijn
19ee873616 Merge pull request #33859 from tonistiigi/session-includepaths
Add path filtering to build session client
2017-07-05 18:21:48 -07:00
Josh Soref
39bcaee47b
Spelling fixes
* additional
* ambiguous
* anonymous
* anything
* application
* because
* before
* building
* capabilities
* circumstances
* commit
* committer
* compresses
* concatenated
* config
* container
* container's
* current
* definition
* delimiter
* disassociates
* discovery
* distributed
* doesnotexist
* downloads
* duplicates
* either
* enhancing
* enumerate
* escapable
* exactly
* expect
* expectations
* expected
* explicitly
* false
* filesystem
* following
* forbidden
* git with
* healthcheck
* ignore
* independent
* inheritance
* investigating
* irrelevant
* it
* logging
* looking
* membership
* mimic
* minimum
* modify
* mountpoint
* multiline
* notifier
* outputting
* outside
* overridden
* override
* parsable
* plugins
* precedence
* propagation
* provided
* provides
* registries
* repositories
* returning
* settings
* should
* signals
* someone
* something
* specifically
* successfully
* synchronize
* they've
* thinking
* uninitialized
* unintentionally
* unmarshaling
* unnamed
* unreferenced
* verify

Signed-off-by: Josh Soref <jsoref@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-07-03 13:13:09 -07:00
Brian Goff
e066edcfcf Merge pull request #33860 from lixiaobing1/masterFclose
add f.Close()  after dockerignore.ReadAll(f) before return err
2017-06-29 10:19:45 -04:00
lixiaobing10051267
3072fce067
add f.Close() after dockerignore.ReadAll(f) before return err
Signed-off-by: lixiaobing10051267 <li.xiaobing1@zte.com.cn>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-06-29 00:51:31 -07:00
Sebastiaan van Stijn
b7ac1571a3 Merge pull request #33851 from tonistiigi/fix-add-remote
build: fix add from remote url
2017-06-28 23:53:21 -07:00
Tonis Tiigi
ad46348d7c builder: enable path filtering for filesync session
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-28 09:12:30 -07:00
John Howard
950d472c9c Merge pull request #33826 from Microsoft/jjh/lcownits
LCOW: Fix nits from 33241
2017-06-27 22:56:38 -07:00
John Howard
4ec9766a27 LCOW: Fix nits from 33241
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-27 11:59:49 -07:00
Tonis Tiigi
2981667e11 build: fix add from remote url
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-27 10:38:01 -07:00
Sebastiaan van Stijn
d3d1aabcc6
Move IsGitTransport() to gitutils
This function was only used inside gitutils,
and is written specifically for the requirements
there.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-06-26 10:07:04 -07:00
Sebastiaan van Stijn
913eb99fdc Fix handling of remote "git@" notation
`docker build` accepts remote repositories
using either the `git://` notation, or `git@`.

Docker attempted to parse both as an URL, however,
`git@` is not an URL, but an argument to `git clone`.

Go 1.7 silently ignored this, and managed to
extract the needed information from these
remotes, however, Go 1.8 does a more strict
validation, and invalidated these.

This patch adds a different path for `git@` remotes,
to prevent them from being handled as URL (and
invalidated).

A test is also added, because there were no
tests for handling of `git@` remotes.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-06-26 10:02:12 -07:00
Daniel Nephin
b47b375cb8 Add a test for warning on empty continuation lines.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-23 13:15:04 -04:00
Yong Tang
7815c8f875 Warn/deprecate continuing on empty lines in Dockerfile
This fix is related to 29005 and 24693. Currently in `Dockerfile`
empty lines will continue as long as there is line escape before.
This may cause some issues. The issue in 24693 is an example.
A non-empty line after an empty line might be considered to be a
separate instruction by many users. However, it is actually part
of the last instruction under the current `Dockerfile` parsing
rule.

This fix is an effort to reduce the confusion around the parsing
of `Dockerfile`. Even though this fix does not change the behavior
of the `Dockerfile` parsing, it tries to deprecate the empty line
continuation and present a warning for the user. In this case,
at least it prompt users to check for the Dockerfile and avoid
the confusion if possible.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
2017-06-23 13:04:40 -04:00
Tonis Tiigi
5c3d2d552b Implement incremental file sync using client session
Also exposes shared cache and garbage collection/prune
for the source data.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-22 11:52:35 -07:00
Tonis Tiigi
ec7b6238c3 Add long-running client session endpoint
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-22 11:22:41 -07:00
John Stephens
930e689668 Merge pull request #33241 from Microsoft/jjh/multi-layerstore
LCOW: Support most operations excluding remote filesystem
2017-06-21 15:45:23 -07:00
John Howard
fe7b4d8fcd LCOW: Set correct default shell for platform in builder
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-20 19:54:33 -07:00
John Howard
3a09b56a10 LCOW: Rework after 33454 merged which refactored daemon/builder interface
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-20 19:50:13 -07:00
John Howard
f154588226 LCOW: OCI Spec and Environment for container start
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-20 19:50:11 -07:00
John Howard
ba40132366 LCOW: Fix ImageCache to address right store
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-20 19:49:53 -07:00
John Howard
08252bc963 LCOW: Builder plumbing
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-20 19:49:53 -07:00
John Howard
8f53780694 LCOW: commit to use image platform, not host
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-20 19:49:52 -07:00
Tonis Tiigi
c268d9da4b builder: fix copy —from conflict with force pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-20 15:45:33 -07:00
Vincent Demeester
99c72eb268 Merge pull request #33454 from dnephin/refactor-builder-remove-copy-on-build
[Builder] Move file coping from the daemon to the builder
2017-06-20 10:12:57 +02:00
Vincent Demeester
c21c17c642 Merge pull request #32088 from simonferquel/windows-dockerignore-fix
Fix behavior of absolute paths in .dockerignore
2017-06-13 13:36:39 +02:00
Daniel Nephin
5136096520 Fix copy when used with scratch and images with empty RootFS
Commit the rwLayer to get the correct DiffID
Refacator copy in thebuilder
move more code into exportImage
cleanup some windows tests
Release the newly commited layer.
Set the imageID on the buildStage after exporting a new image.
Move archiver to BuildManager.
Have ReleaseableLayer.Commit return a layer
and store the Image from exportImage in the local imageSources cache
Remove NewChild from image interface.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-08 15:07:16 -04:00
Daniel Nephin
ecd44d23cc Rename LazyContext to LazySource.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-08 15:07:15 -04:00
Daniel Nephin
bd5f92d263 Remove CopyOnBuild from the daemon.
Add CreateImage() to the daemon
Refactor daemon.Comit() and expose a Image.NewChild()
Update copy to use IDMappings.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-08 15:06:54 -04:00
Akihiro Suda
09eca591f7 builder: add a test for ENV name (without =value)
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-06-07 05:23:26 +00:00
Daniel Nephin
f1ade82d82 Fix Cache with ONBUILD
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-06 10:52:29 -04:00
Vincent Demeester
34f1cd26b6 Merge pull request #33500 from dnephin/fix-onbuild-copy
Fix ONBUILD COPY
2017-06-05 11:48:57 +02:00
Daniel Nephin
3f26041577 Fix ONBUILD COPY
the source was missing from the second dispatch

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-02 17:47:03 -04:00
Daniel Nephin
b9d85ac58d Move pkg/gitutils to remotecontext/git
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-02 16:54:50 -04:00
Daniel Nephin
4060d6ee0b Remove the last of pkg/httputil
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-02 16:54:36 -04:00
Daniel Nephin
c91521be68 Move httputils/mimtype to the single consumser, and remove unused function.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-06-02 16:10:39 -04:00
John Howard
57ca00240b Merge pull request #33436 from Microsoft/jjh/parserdirective
Builder: Fix parser directive refactoring
2017-06-01 21:21:46 -07:00
John Howard
c97170d618 Builder: Fix parser directive refactoring
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-05-31 18:48:27 -07:00
Sebastiaan van Stijn
7ea0d1ecc6 Merge pull request #33363 from tonistiigi/fix-chunked-build
Fix cancelling builder on chunked requests
2017-05-31 20:18:35 +02:00
Tonis Tiigi
23628bd7ef Fix cancelling builder on chunked requests
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-23 16:29:13 -07:00
Daniel Nephin
19f3b0715c Extract imageProber and ContainerBackend from Builder
Extract a common function for builder.createContainer
Extract imageCache for doing cache probes
Removes the cacheBuested field from Builder
Create a new containerManager class which reduces the interface between the
builder and managing containers to 3 functions (from 6)

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-23 15:12:58 -04:00
Daniel Nephin
213ed02e25 Refactor builder.calcCopyInfo and dependencies
Remove pathCache from imageContexts
Extract validateCopySourcePath
Extract copyWithWildcards
Extract copyInfoForFile and walkSource from calcCopyInfo
Move copy internals to copy.go
remove source from Builder

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-17 09:54:39 -04:00
Josh Hawn
4921171587 Update ContainerWait API
This patch adds the untilRemoved option to the ContainerWait API which
allows the client to wait until the container is not only exited but
also removed.

This patch also adds some more CLI integration tests for waiting for a
created container and waiting with the new --until-removed flag.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Handle detach sequence in CLI

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Update Container Wait Conditions

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Apply container wait changes to API 1.30

The set of changes to the containerWait API missed the cut for the
Docker 17.05 release (API version 1.29). This patch bumps the version
checks to use 1.30 instead.

This patch also makes a minor update to a testfile which was added to
the builder/dockerfile package.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Remove wait changes from CLI

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Address minor nits on wait changes

- Changed the name of the tty Proxy wrapper to `escapeProxy`
- Removed the unnecessary Error() method on container.State
- Fixes a typo in comment (repeated word)

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Use router.WithCancel in the containerWait handler

This handler previously added this functionality manually but now uses
the existing wrapper which does it for us.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Add WaitCondition constants to api/types/container

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Address more ContainerWait review comments

- Update ContainerWait backend interface to not return pointer values
  for container.StateStatus type.
- Updated container state's Wait() method comments to clarify that a
  context MUST be used for cancelling the request, setting timeouts,
  and to avoid goroutine leaks.
- Removed unnecessary buffering when making channels in the client's
  ContainerWait methods.
- Renamed result and error channels in client's ContainerWait methods
  to clarify that only a single result or error value would be sent
  on the channel.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Move container.WaitCondition type to separate file

... to avoid conflict with swagger-generated code for API response

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Address more ContainerWait review comments

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2017-05-16 15:11:39 -07:00
Josh Hawn
cfdf84d5d0 Update Container Wait Backend
This patch consolidates the two WaitStop and WaitWithContext methods
on the container.State type. Now there is a single method, Wait, which
takes a context and a bool specifying whether to wait for not just a
container exit but also removal.

The behavior has been changed slightly so that a wait call during a
Created state will not return immediately but instead wait for the
container to be started and then exited.

The interface has been changed to no longer block, but instead returns
a channel on which the caller can receive a *StateStatus value which
indicates the ExitCode or an error if there was one (like a context
timeout or state transition error).

These changes have been propagated through the rest of the deamon to
preserve all other existing behavior.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2017-05-16 15:09:14 -07:00
Tõnis Tiigi
ad846a136a Merge pull request #32959 from simonferquel/tar-filemode-windows
Respect tar entries modes when rewriting them on Windows
2017-05-16 11:45:42 -07:00
Kenfe-Mickaël Laventure
4e0fdc9098 Merge pull request #33179 from tonistiigi/fix-from-scratch
Fix resetting image metadata between stages for scratch case
2017-05-13 21:17:02 -07:00
Sebastiaan van Stijn
20255ade29 Merge pull request #32700 from jphuynh/more-maintainer-deprecation
MAINTAINER deprecation follow up. Relates to #25466
2017-05-13 17:28:52 +02:00
Tonis Tiigi
b192218503 Fix resetting image metadata between stages for scratch case
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-12 12:00:55 -07:00
Daniel Nephin
6c28e8edd5 Refactor imageContexts into two different structs.
buildStages now tracks the imageID and runConfig for a build stage

imageMounter tracks image mounts so they can released when the build ends.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-10 17:59:30 -04:00
Daniel Nephin
ab3a037a5b Refactor interaction between dispatcher.from and dispatchState
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-10 17:58:57 -04:00
Daniel Nephin
b3bc7b28d0 Expose a smaller interface for the Builder retrieving images from daemon
Removes 3 methods from the builder.Backend interface
Remove the coupling between imageContexts, imageMounts and the builder.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-10 17:58:57 -04:00
Tõnis Tiigi
a3eebaf509 Merge pull request #33090 from tonistiigi/fix-release-mounts
builder: Fix releasing implicit mounts
2017-05-10 14:46:58 -07:00
Vincent Demeester
22a03192fb Merge pull request #33114 from dnephin/fix-builder-warn-on-unused-builtint
Fix warning for unused build args
2017-05-10 09:38:07 +02:00
Daniel Nephin
543dfde637 Merge pull request #32997 from dnephin/fix-builder-scratch-with-name
Fix a rare case where using FROM scratch as NAME would fail
2017-05-09 18:15:33 -04:00
Sebastiaan van Stijn
1a6f8a92b2 Merge pull request #32792 from rogaha/extra_prometheus_metrics
Add extra prometheus metrics
2017-05-09 21:48:34 +02:00
Daniel Nephin
72cc81ee8d Fix warning for unused build args.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-09 11:25:33 -04:00
Roberto Gandolfo Hashioka
a28b173a78 Add extra prometheus metrics
- buildsTriggered
 - buildsFailed
    - valid options:
        metricsDockerfileSyntaxError,
        metricsDockerfileEmptyError,
        metricsCommandNotSupportedError,
        metricsErrorProcessingCommandsError,
        metricsBuildTargetNotReachableError,
        metricsMissingOnbuildArgumentsError,
        metricsUnknownInstructionError,
        metricsBuildCanceled,
- engineInfo

Signed-off-by: Roberto Gandolfo Hashioka <roberto_hashioka@hotmail.com>
2017-05-09 01:04:40 -07:00
Tonis Tiigi
29efb93a19 builder: Fix releasing implicit mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-08 10:32:20 -07:00
Ian Campbell
5894bc1abf Add docker build --iidfile=FILE
This is synonymous with `docker run --cidfile=FILE` and writes the digest of
the newly built image to the named file. This is intended to be used by build
systems which want to avoid tagging (perhaps because they are in CI or
otherwise want to avoid fixed names which can clash) by enabling e.g. Makefile
constructs like:

    image.id: Dockerfile
    	docker build --iidfile=image.id .

    do-some-more-stuff: image.id
    	do-stuff-with <image.id

Currently the only way to achieve this is to use `docker build -q` and capture
the stdout, but at the expense of losing the build output.

In non-silent mode (without `-q`) with API >= v1.29 the caller will now see a
`JSONMessage` with the `Aux` field containing a `types.BuildResult` in the
output stream for each image/layer produced during the build, with the final
one being the end product.  Having all of the intermediate images might be
interesting in some cases.

In silent mode (with `-q`) there is no change, on success the only output will
be the resulting image digest as it was previosuly.

There was no wrapper to just output an Aux section without enclosing it in a
Progress, so add one here.

Added some tests to integration cli tests.

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-05-05 16:35:54 +01:00
Daniel Nephin
28379bd29a Fix a rare case where using FROM scracth as NAME would fail
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-05 11:14:03 -04:00
Akihiro Suda
b61ffbfb52 Merge pull request #33033 from YuPengZTE/devReadAll
check err
2017-05-05 22:33:23 +09:00
yupengzte
0d99cb8629 check err
Signed-off-by: yupengzte <yu.peng36@zte.com.cn>
2017-05-05 13:33:40 +08:00
Daniel Nephin
2f0ebba0e7 Some refactoring of dispatch()
Remove runConfig from Builder and dispatchRequest. It is not only on
dispatchState.

Move dispatch state fields from Builder to dispatchState

Move stageName tracking to dispatchRequest.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-04 17:11:08 -04:00
Daniel Nephin
37be263826 Merge pull request #32749 from dnephin/rm-ContainerUpdateCmdOnBuild
[Builder] Remove ContainerUpdateCmdOnBuild
2017-05-03 14:02:08 -04:00
Aaron Lehmann
08d7fad45d Merge pull request #32949 from dnephin/refactor-streamformatter-and-progress
Refacator pkg/streamformatter
2017-05-03 09:05:26 -07:00
Daniel Nephin
d9371ee807 Fix run with entrypoint in base image
Update a test to use a base image with entrypoint to that the linux build
has at least one test that behaves like all the windows tests.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-03 10:59:38 -04:00
Daniel Nephin
97f8607164 Remove ContainerUpdateCmdOnBuild, it does nothing.
Set a blank entrypoint to preserve the old behaviour.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-03 10:59:38 -04:00
Daniel Nephin
3dcab28982 Fix setting b.runConfig.Image at arbitrary places.
Previously this value was set at some point attrbitrarily between when it was updated and when it was going to be used next.

Instead always set it as the last step of dispatch.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-03 10:59:37 -04:00
Daniel Nephin
c87d67b0ad Refacator pkg/streamformatter
StreamFormatter suffered was two distinct structs mixed into a single struct
without any overlap.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-02 17:38:12 -04:00
Simon Ferquel
41eb61d5c2 Respect tar entries modes when rewriting them on Windows
Previously, only perm-related bits where preserved when rewriting
FileMode in tar entries on Windows. This had the nasty side effect of
having tarsum returning different values when executing from a tar filed
produced on Windows or Linux.

This fix the issue, and pave the way for incremental build context
to work in hybrid contexts.

Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
2017-05-02 11:36:43 +02:00
Daniel Nephin
9f738cc574 Cleanup all the mutate + defer revert of b.runConfig in the builder
Instead of mutating and reverting, just create a copy and pass the copy
around.

Add a unit test for builder dispatcher.run

Fix two test failures

Fix image history by adding a CreatedBy to commit options. Previously the
createdBy field was being created by modifying a reference to the runConfig that
was held from when the container was created.

Fix a test that expected a trailing slash. Previously the runConfig was being
modified by container create. Now that we're creating a copy of runConfig
instead of sharing a reference the runConfig retains the trailing slash.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-01 18:36:53 -04:00
Daniel Nephin
0296797f0f Extract squash and tagging from the Dockerfile builder.
Remove pathCache and replace it with syncmap
Cleanup NewBuilder
Create an api/server/backend/build
Extract BuildTagger

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-05-01 12:07:32 -04:00
Daniel Nephin
721e1736ae Fix cache miss when builtin build args are used.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-28 12:49:50 -04:00
Brian Goff
ec67cea96e Merge pull request #32772 from dnephin/refactor-builder-split-commit
[Builder] Refactor `Builder.commit()`
2017-04-28 11:18:55 -04:00
Brian Goff
9f0ebea335 Merge pull request #32763 from dave-tucker/fix-32744
builder: Make builtin arg pruning work with > 1 arg
2017-04-28 10:24:13 -04:00
Brian Goff
a7519152d9 Merge pull request #32504 from dongluochen/healthcheck_duration
do not allow duration less than 1 ms in healthcheck parameters
2017-04-27 23:54:00 -04:00
Daniel Nephin
0d9e66b98a Extract commitContainer from commit() to separate the distinct branches
Remove unused arguments to commit.
This will allow us to remove all the runConfig mutate+revert code that
is scattered around builder dispatchers/internals

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-27 20:49:38 -04:00
Daniel Nephin
aee2da3bdf Merge pull request #32858 from dnephin/builder-shell-words-interface
[Builder] Remove b.escapeToken, create ShellLex
2017-04-27 13:16:40 -04:00
Brian Goff
ae0f8c7ba1 Merge pull request #31984 from tonistiigi/remote-context
builder: Refactor remote context
2017-04-27 11:50:18 -04:00
Daniel Nephin
2414166e1e Create a new ShellLex struct which provides the expected interface to Builder.
Remove b.escapeToken from builder

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-26 19:39:16 -04:00
Tõnis Tiigi
5eca7f7c5d Merge pull request #31856 from jim-minter/more_races
Resolve a set of race conditions in logging and attaching code
2017-04-26 16:04:02 -07:00
Dave Tucker
1076ab58ec builder: Make builtin arg pruning work with > 1 arg
The previous implementation would error out with "Unexpected EOF" which
was caused by an underlying "array index out-of-bounds" error.
The root cause was deleting items from the same array that was being
iterated over. The iteration was unaware that the array size had
changed, resulting in an error.

The new implementation builds a new array instead of mutating a copy of
the old one.

Fixes: #32744

Signed-off-by: Dave Tucker <dt@docker.com>
2017-04-26 10:24:58 +01:00
Tonis Tiigi
d1faf3df27 Refactor remote context parsing
Redefine a better interface for remote context dependency.

Separate Dockerfile build instruction from remote context.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-04-25 14:35:31 -07:00
Daniel Nephin
52626bb919 Reduce the number of calls to b.image
In preparation for removing it from builder.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-24 14:21:19 -04:00
Daniel Nephin
e59327aaac Change builder.dispatchers to use a dispatchRequest object
This change starts the process of splitting up the Builder into logical
components. Remove builder.flags and move it to the new dispatchRequest
object.

Use runConfig from dispatchRequest instead of from the builder.

More progress removing things from the Builder struct.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-24 14:21:19 -04:00
Tõnis Tiigi
3beb8d9c8b Merge pull request #32500 from dnephin/refactor-from-dispatcher
Refactor from dispatcher
2017-04-24 10:10:42 -07:00
Jim Minter
32ca1214fa Fix race on ContainerAttachRaw
Signed-off-by: Jim Minter <jminter@redhat.com>
2017-04-20 12:27:50 +01:00
Jean-Pierre Huynh
2c435ab8be MAINTAINER deprecation follow up. Relates to #25466
Signed-off-by: Jean-Pierre Huynh <jean-pierre.huynh@ounet.fr>
2017-04-18 16:44:33 -05:00
Aaron Lehmann
6052f2b396 Remove pkg/testutil/assert in favor of testify
I noticed that we're using a homegrown package for assertions. The
functions are extremely similar to testify, but with enough slight
differences to be confusing (for example, Equal takes its arguments in a
different order). We already vendor testify, and it's used in a few
places by tests.

I also found some problems with pkg/testutil/assert. For example, the
NotNil function seems to be broken. It checks the argument against
"nil", which only works for an interface. If you pass in a nil map or
slice, the equality check will fail.

In the interest of avoiding NIH, I'm proposing replacing
pkg/testutil/assert with testify. The test code looks almost the same,
but we avoid the confusion of having two similar but slightly different
assertion packages, and having to maintain our own package instead of
using a commonly-used one.

In the process, I found a few places where the tests should halt if an
assertion fails, so I've made those cases (that I noticed) use "require"
instead of "assert", and I've vendored the "require" package from
testify alongside the already-present "assert" package.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2017-04-14 12:03:21 -07:00
Vincent Demeester
41f4c3cf7e Merge pull request #32601 from dnephin/builder-shell-words-buffer
Use a bytes.Buffer for shell_words string concat
2017-04-13 23:22:18 +02:00
Daniel Nephin
aafd7fa969 remove increment flag on imageContexts.new()
Rename the methods to match my understanding of the behaviour.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-13 14:56:57 -04:00
Daniel Nephin
4d62f67117 Extract two functions from the FROM dispatcher
This helps clarify the difference cases for parsing ctxName, and for
getting an image.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-13 14:56:57 -04:00
Vincent Demeester
700b4807c3 Merge pull request #32580 from dnephin/refactor-builder-parser-directive
Refactor Dockerfile.parser and directive
2017-04-13 20:47:42 +02:00
Daniel Nephin
0055a48277 Use a bytes.Buffer for shell_words string concat
It's much faster

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-13 14:39:34 -04:00
Daniel Nephin
df3c425407 Merge pull request #32328 from duglin/fixEnvVars
Minor tweaks to quotes in env vars
2017-04-13 10:41:28 -04:00
Dong Chen
d8b6a35d02 set 1ms as container duration minimum value
Signed-off-by: Dong Chen <dongluo.chen@docker.com>
2017-04-12 15:45:29 -07:00
Daniel Nephin
4cbc953a5d Refactor dockerfile parser
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-12 18:31:52 -04:00
Daniel Nephin
9c53fa2d0c Cleanup processing of directives.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-12 15:42:35 -04:00
Doug Davis
2fb7c3c4f0 Minor tweaks to quotes in env vars
Addresses part of #32140, in particular:
- this will make it so that double backslashes in double-quoted
strings will result in a single backslash. While in single quotes it remains
a double backslash.
- missing closing " and ' will now generate an error

Signed-off-by: Doug Davis <dug@us.ibm.com>
2017-04-12 12:20:14 -07:00
Daniel Nephin
64c4c1c3d5 Keep parser.Directive internal to parser
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-12 14:48:09 -04:00
Daniel Nephin
bb429da9a9 Hide builder.parser.Directive internals
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-12 14:48:09 -04:00
Tõnis Tiigi
d2ab40e5b9 Merge pull request #32532 from dnephin/refactor-some-builder-parts
[builder] cleanup env dispatcher and Builder.build()
2017-04-12 11:11:06 -07:00
Vincent Demeester
3c07259882 Merge pull request #32370 from AkihiroSuda/improve-build-error
builder/dockerfile: improve error message about build stage name
2017-04-12 16:33:23 +02:00
Akihiro Suda
50a9be4bf5 builder/dockerfile: improve error message about build stage name
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-04-12 05:27:39 +00:00
Daniel Nephin
068f344e03 Remove unused Builder.Cancel()
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-11 20:33:13 -04:00
Daniel Nephin
a6abd57b83 Remove unused id field from Builder.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-11 14:44:32 -04:00
Daniel Nephin
bfcd95817a Factor out functions from builder/dockerfile/builder.go:Builder.build()
Remove the block comment which is stale, and redundant now that the
function is just as readable as the comment.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-11 14:44:32 -04:00
Daniel Nephin
c7fad9b750 Cleanup in dispatcher.env
Remove commented code blocks
Remove some duplication in comparing and restructuring env

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-11 14:44:28 -04:00
Akihiro Suda
b764c471bf Merge pull request #32495 from dnephin/cleanup-builder-dispatcher-env
Remove Builder.dockerfile field
2017-04-11 14:50:55 +09:00
Tonis Tiigi
33e07f41ad builder: add an option for specifying build target
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-04-10 16:21:43 -07:00
Daniel Nephin
f3e205ddc1 Remove the dockerfile field from Builder.
Return dockerfile from parseDockerfile and pass the dockerfile nodes
as an arg

Strip unused arg from builder.NewBuilder.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-10 18:09:17 -04:00
Daniel Nephin
4d9e32a08e Merge pull request #31352 from dnephin/allow-arg-in-fromt
[dockerfile] Allow ARG in FROM
2017-04-10 13:23:40 -04:00
Daniel Nephin
239c53bf83 Refactor BuildArgs
Add MetaArgs for ARG that occur before the first FROM
Integration test for these cases.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-07 17:57:47 -04:00
Daniel Nephin
9b4aa7629c Fix arg in from when arg is not defined
Add mock builder backend
Add tests for ARG in FROM

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-07 17:57:47 -04:00
Daniel Nephin
f0a9c2e3f4 Allow ARG to come before FROM to support variables in FROM.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-04-07 17:57:47 -04:00
Simon Ferquel
b0e7588873 Test and fix forbidden path for COPY --from on Windows
Paths resolving to c:\ or c:\windows are forbidden

Replaced the obscure (and non-working) regex with a simple case
insensitive comparison to the black listed paths (we should forbid c:\,
c:\windows but not d:\)

Also, add a test ensuring paths are case insensitive on windows

Also, made sure existing multi-staged build tests pass on windows

Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
2017-04-07 18:00:36 +02:00
Elias Faxö
e401f63735 Added start period option to health check.
Signed-off-by: Elias Faxö <elias.faxo@gmail.com>
2017-04-06 12:35:34 +02:00
Victor Vieux
2e96c1739b Merge pull request #32063 from tonistiigi/named-contexts
Add support for named build stages
2017-04-04 12:53:59 -07:00
Tõnis Tiigi
6abbc933ae Merge pull request #31750 from dnephin/some-builder-cleanup
Fix `docker build --label` when the label includes single quotes and a space
2017-04-04 09:37:00 -07:00
Tonis Tiigi
672ea30a94 Add support for FROM using named block
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-04-04 09:34:28 -07:00
Tonis Tiigi
73b4b8ed7f Implicit copy-from with reference
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-04-04 09:34:28 -07:00
Tonis Tiigi
87512bbc84 Add named context support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-04-04 09:34:28 -07:00
Simon Ferquel
1cde87c43a Fix behavior of absolute paths in .dockerignore
According to documentation (https://docs.docker.com/engine/reference/builder/#dockerignore-file), absolute paths like `/foo/bar` should have the same effect as `foo/bar`. This is not the case today.

This fix normalize paths when reading the .dockerignore file by removing
leading slashes.

Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
2017-03-31 10:43:59 +02:00
Tonis Tiigi
a1fa59ea12 builder: Fix setting command with custom shell
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-03-30 13:29:21 -07:00
Alexandre Garnier
d005219830 Display created tags on successful build
Signed-off-by: Alexandre Garnier <alexandre.garnier@zenika.com>
Signed-off-by: Alexandre Garnier <zigarn@gmail.com>
2017-03-25 11:46:20 +01:00
Daniel Nephin
434d77bc0e Fix --label being env var expanded.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-03-24 12:29:14 -04:00
Daniel Nephin
726fb269cf Fix --label on docker build when a single quote is used in the value
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-03-24 12:29:14 -04:00
Daniel Nephin
d7807c7316 Refactor some builder code
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-03-24 12:29:14 -04:00
Tõnis Tiigi
2fa8fe400d Merge pull request #31257 from tonistiigi/nested-build
build: add multi-stage build support
2017-03-23 19:14:13 -07:00
Tonis Tiigi
684633f734 Fix relative path on windows for uuid paths
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-03-23 15:12:19 -07:00
Tonis Tiigi
f95f58283b Add support for COPY from previous rootfs
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-03-23 15:12:15 -07:00
Dave Tucker
89a2a885c8 Ignore built-in allowed build-args in image history
Removes the build-args from the image history if they are in the
BuiltinAllowedBuildArgs map unless they are explicitly defined in an ARG
instruction.

Signed-off-by: Dave Tucker <dt@docker.com>
2017-03-21 16:36:48 +00:00
Tonis Tiigi
acad599210 Fix cache for dockerfiles with multiple FROM
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-03-20 11:24:50 -07:00
Tonis Tiigi
09f308ce21 Fix ARG scoping for Dockerfiles with multiple FROM
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-03-20 11:24:44 -07:00
fate-grand-order
2b79dbc89b fix misspell in utils.go
Signed-off-by: Helen Xie <chenjg@harmonycloud.cn>
2017-03-20 19:52:00 +08:00
Daniel Nephin
9ea5e02e92 extract a function
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-03-08 11:46:09 -05:00
Sebastiaan van Stijn
fb1890fb95 Merge pull request #31177 from allencloud/donot-allow-subsecond-in-dockerfile-healthcheck
do not allow sub second in healthcheck options in Dockerfile
2017-03-08 15:18:09 +01:00
Sebastiaan van Stijn
a64ea37753 Merge pull request #30383 from TDAbboud/30096-add-host-docker-build
Add --add-host for docker build
2017-02-27 12:42:18 +01:00
Akihiro Suda
a6576b8a2b Merge pull request #31212 from fate-grand-order/fixed
use t.Fatal() to output the err message and fix typo
2017-02-27 12:26:35 +09:00
Daniel Nephin
6e005fd5cc Refactor to remove duplicate code around BuildArgs.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-02-24 16:22:32 -05:00
fate-grand-order
2a8d6368d4 use t.Fatal() to output the err message where the values used for formatting
text does not appear to contain a placeholder

Signed-off-by: Helen Xie <chenjg@harmonycloud.cn>
2017-02-23 10:16:22 +08:00
Reficul
9b1ceecd8a
fix wrong print format
Signed-off-by: Reficul <xuzhenglun@gmail.com>
2017-02-21 10:26:06 +08:00