Instead of having to create a bunch of custom error types that are doing
nothing but wrapping another error in sub-packages, use a common helper
to create errors of the requested type.
e.g. instead of re-implementing this over and over:
```go
type notFoundError struct {
cause error
}
func(e notFoundError) Error() string {
return e.cause.Error()
}
func(e notFoundError) NotFound() {}
func(e notFoundError) Cause() error {
return e.cause
}
```
Packages can instead just do:
```
errdefs.NotFound(err)
```
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The building machinery was being handed an uninitialized container
Config. This changes it to use the target container's Config.
Resolves#30538
Signed-off-by: Anthony Sottile <asottile@umich.edu>
The test was passing previously because the preamble was already buffered. After
the change to return Scanner.Err() the final read error on the buffer was no
longer being ignored.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
This PR has the API changes described in https://github.com/moby/moby/issues/34617.
Specifically, it adds an HTTP header "X-Requested-Platform" which is a JSON-encoded
OCI Image-spec `Platform` structure.
In addition, it renames (almost all) uses of a string variable platform (and associated)
methods/functions to os. This makes it much clearer to disambiguate with the swarm
"platform" which is really os/arch. This is a stepping stone to getting the daemon towards
fully multi-platform/arch-aware, and makes it clear when "operating system" is being
referred to rather than "platform" which is misleadingly used - sometimes in the swarm
meaning, but more often as just the operating system.
Commit 8d1ae76dcb added
deprecation warnings for empty continuation lines,
but also treated comment-only lines as empty.
This patch distinguishes empty continuation lines
from comment-only lines, and only outputs warnings
for the former.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is a work base to introduce more features like build time
dockerfile optimisations, dependency analysis and parallel build, as
well as a first step to go from a dispatch-inline process to a
frontend+backend process.
Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
This enables docker cp and ADD/COPY docker build support for LCOW.
Originally, the graphdriver.Get() interface returned a local path
to the container root filesystem. This does not work for LCOW, so
the Get() method now returns an interface that LCOW implements to
support copying to and from the container.
Signed-off-by: Akash Gupta <akagup@microsoft.com>
This fix tries to address the issue raised in #34208 where
in Dockerfile an `ADD` followed by an url without any sub path
will cause an error.
The issue is because the temporary filename relies on the sub path.
An integration test has been added.
This fix fixes#34208.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Rebases and completes initial PR for (prior: --user) --chown flag for
ADD/COPY commands in Dockerfile.
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
Use strongly typed errors to set HTTP status codes.
Error interfaces are defined in the api/errors package and errors
returned from controllers are checked against these interfaces.
Errors can be wraeped in a pkg/errors.Causer, as long as somewhere in the
line of causes one of the interfaces is implemented. The special error
interfaces take precedence over Causer, meaning if both Causer and one
of the new error interfaces are implemented, the Causer is not
traversed.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This function was only used inside gitutils,
and is written specifically for the requirements
there.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`docker build` accepts remote repositories
using either the `git://` notation, or `git@`.
Docker attempted to parse both as an URL, however,
`git@` is not an URL, but an argument to `git clone`.
Go 1.7 silently ignored this, and managed to
extract the needed information from these
remotes, however, Go 1.8 does a more strict
validation, and invalidated these.
This patch adds a different path for `git@` remotes,
to prevent them from being handled as URL (and
invalidated).
A test is also added, because there were no
tests for handling of `git@` remotes.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix is related to 29005 and 24693. Currently in `Dockerfile`
empty lines will continue as long as there is line escape before.
This may cause some issues. The issue in 24693 is an example.
A non-empty line after an empty line might be considered to be a
separate instruction by many users. However, it is actually part
of the last instruction under the current `Dockerfile` parsing
rule.
This fix is an effort to reduce the confusion around the parsing
of `Dockerfile`. Even though this fix does not change the behavior
of the `Dockerfile` parsing, it tries to deprecate the empty line
continuation and present a warning for the user. In this case,
at least it prompt users to check for the Dockerfile and avoid
the confusion if possible.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Commit the rwLayer to get the correct DiffID
Refacator copy in thebuilder
move more code into exportImage
cleanup some windows tests
Release the newly commited layer.
Set the imageID on the buildStage after exporting a new image.
Move archiver to BuildManager.
Have ReleaseableLayer.Commit return a layer
and store the Image from exportImage in the local imageSources cache
Remove NewChild from image interface.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Add CreateImage() to the daemon
Refactor daemon.Comit() and expose a Image.NewChild()
Update copy to use IDMappings.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Extract a common function for builder.createContainer
Extract imageCache for doing cache probes
Removes the cacheBuested field from Builder
Create a new containerManager class which reduces the interface between the
builder and managing containers to 3 functions (from 6)
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Remove pathCache from imageContexts
Extract validateCopySourcePath
Extract copyWithWildcards
Extract copyInfoForFile and walkSource from calcCopyInfo
Move copy internals to copy.go
remove source from Builder
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This patch adds the untilRemoved option to the ContainerWait API which
allows the client to wait until the container is not only exited but
also removed.
This patch also adds some more CLI integration tests for waiting for a
created container and waiting with the new --until-removed flag.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Handle detach sequence in CLI
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Update Container Wait Conditions
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Apply container wait changes to API 1.30
The set of changes to the containerWait API missed the cut for the
Docker 17.05 release (API version 1.29). This patch bumps the version
checks to use 1.30 instead.
This patch also makes a minor update to a testfile which was added to
the builder/dockerfile package.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Remove wait changes from CLI
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Address minor nits on wait changes
- Changed the name of the tty Proxy wrapper to `escapeProxy`
- Removed the unnecessary Error() method on container.State
- Fixes a typo in comment (repeated word)
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Use router.WithCancel in the containerWait handler
This handler previously added this functionality manually but now uses
the existing wrapper which does it for us.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Add WaitCondition constants to api/types/container
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Address more ContainerWait review comments
- Update ContainerWait backend interface to not return pointer values
for container.StateStatus type.
- Updated container state's Wait() method comments to clarify that a
context MUST be used for cancelling the request, setting timeouts,
and to avoid goroutine leaks.
- Removed unnecessary buffering when making channels in the client's
ContainerWait methods.
- Renamed result and error channels in client's ContainerWait methods
to clarify that only a single result or error value would be sent
on the channel.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Move container.WaitCondition type to separate file
... to avoid conflict with swagger-generated code for API response
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Address more ContainerWait review comments
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This patch consolidates the two WaitStop and WaitWithContext methods
on the container.State type. Now there is a single method, Wait, which
takes a context and a bool specifying whether to wait for not just a
container exit but also removal.
The behavior has been changed slightly so that a wait call during a
Created state will not return immediately but instead wait for the
container to be started and then exited.
The interface has been changed to no longer block, but instead returns
a channel on which the caller can receive a *StateStatus value which
indicates the ExitCode or an error if there was one (like a context
timeout or state transition error).
These changes have been propagated through the rest of the deamon to
preserve all other existing behavior.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
buildStages now tracks the imageID and runConfig for a build stage
imageMounter tracks image mounts so they can released when the build ends.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Removes 3 methods from the builder.Backend interface
Remove the coupling between imageContexts, imageMounts and the builder.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This is synonymous with `docker run --cidfile=FILE` and writes the digest of
the newly built image to the named file. This is intended to be used by build
systems which want to avoid tagging (perhaps because they are in CI or
otherwise want to avoid fixed names which can clash) by enabling e.g. Makefile
constructs like:
image.id: Dockerfile
docker build --iidfile=image.id .
do-some-more-stuff: image.id
do-stuff-with <image.id
Currently the only way to achieve this is to use `docker build -q` and capture
the stdout, but at the expense of losing the build output.
In non-silent mode (without `-q`) with API >= v1.29 the caller will now see a
`JSONMessage` with the `Aux` field containing a `types.BuildResult` in the
output stream for each image/layer produced during the build, with the final
one being the end product. Having all of the intermediate images might be
interesting in some cases.
In silent mode (with `-q`) there is no change, on success the only output will
be the resulting image digest as it was previosuly.
There was no wrapper to just output an Aux section without enclosing it in a
Progress, so add one here.
Added some tests to integration cli tests.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Remove runConfig from Builder and dispatchRequest. It is not only on
dispatchState.
Move dispatch state fields from Builder to dispatchState
Move stageName tracking to dispatchRequest.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Update a test to use a base image with entrypoint to that the linux build
has at least one test that behaves like all the windows tests.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Previously this value was set at some point attrbitrarily between when it was updated and when it was going to be used next.
Instead always set it as the last step of dispatch.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Previously, only perm-related bits where preserved when rewriting
FileMode in tar entries on Windows. This had the nasty side effect of
having tarsum returning different values when executing from a tar filed
produced on Windows or Linux.
This fix the issue, and pave the way for incremental build context
to work in hybrid contexts.
Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
Instead of mutating and reverting, just create a copy and pass the copy
around.
Add a unit test for builder dispatcher.run
Fix two test failures
Fix image history by adding a CreatedBy to commit options. Previously the
createdBy field was being created by modifying a reference to the runConfig that
was held from when the container was created.
Fix a test that expected a trailing slash. Previously the runConfig was being
modified by container create. Now that we're creating a copy of runConfig
instead of sharing a reference the runConfig retains the trailing slash.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Remove pathCache and replace it with syncmap
Cleanup NewBuilder
Create an api/server/backend/build
Extract BuildTagger
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Remove unused arguments to commit.
This will allow us to remove all the runConfig mutate+revert code that
is scattered around builder dispatchers/internals
Signed-off-by: Daniel Nephin <dnephin@docker.com>
The previous implementation would error out with "Unexpected EOF" which
was caused by an underlying "array index out-of-bounds" error.
The root cause was deleting items from the same array that was being
iterated over. The iteration was unaware that the array size had
changed, resulting in an error.
The new implementation builds a new array instead of mutating a copy of
the old one.
Fixes: #32744
Signed-off-by: Dave Tucker <dt@docker.com>
Redefine a better interface for remote context dependency.
Separate Dockerfile build instruction from remote context.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This change starts the process of splitting up the Builder into logical
components. Remove builder.flags and move it to the new dispatchRequest
object.
Use runConfig from dispatchRequest instead of from the builder.
More progress removing things from the Builder struct.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
I noticed that we're using a homegrown package for assertions. The
functions are extremely similar to testify, but with enough slight
differences to be confusing (for example, Equal takes its arguments in a
different order). We already vendor testify, and it's used in a few
places by tests.
I also found some problems with pkg/testutil/assert. For example, the
NotNil function seems to be broken. It checks the argument against
"nil", which only works for an interface. If you pass in a nil map or
slice, the equality check will fail.
In the interest of avoiding NIH, I'm proposing replacing
pkg/testutil/assert with testify. The test code looks almost the same,
but we avoid the confusion of having two similar but slightly different
assertion packages, and having to maintain our own package instead of
using a commonly-used one.
In the process, I found a few places where the tests should halt if an
assertion fails, so I've made those cases (that I noticed) use "require"
instead of "assert", and I've vendored the "require" package from
testify alongside the already-present "assert" package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Addresses part of #32140, in particular:
- this will make it so that double backslashes in double-quoted
strings will result in a single backslash. While in single quotes it remains
a double backslash.
- missing closing " and ' will now generate an error
Signed-off-by: Doug Davis <dug@us.ibm.com>
Remove the block comment which is stale, and redundant now that the
function is just as readable as the comment.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Return dockerfile from parseDockerfile and pass the dockerfile nodes
as an arg
Strip unused arg from builder.NewBuilder.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Paths resolving to c:\ or c:\windows are forbidden
Replaced the obscure (and non-working) regex with a simple case
insensitive comparison to the black listed paths (we should forbid c:\,
c:\windows but not d:\)
Also, add a test ensuring paths are case insensitive on windows
Also, made sure existing multi-staged build tests pass on windows
Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
According to documentation (https://docs.docker.com/engine/reference/builder/#dockerignore-file), absolute paths like `/foo/bar` should have the same effect as `foo/bar`. This is not the case today.
This fix normalize paths when reading the .dockerignore file by removing
leading slashes.
Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
Removes the build-args from the image history if they are in the
BuiltinAllowedBuildArgs map unless they are explicitly defined in an ARG
instruction.
Signed-off-by: Dave Tucker <dt@docker.com>
Remove forked reference package. Use normalized named values
everywhere and familiar functions to convert back to familiar
strings for UX and storage compatibility.
Enforce that the source repository in the distribution metadata
is always a normalized string, ignore invalid values which are not.
Update distribution tests to use normalized values.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This drops support for migrations from pre-1.10 Docker versions, which
should be done via an external tool or an intermediate upgrade.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This patch fixed below 4 types of code line
1. Remove unnecessary variable assignment
2. Use variables declaration instead of explicit initial zero value
3. Change variable name to underbar when variable not used
4. Add erro check and return for ignored error
Signed-off-by: Daehyeok Mun <daehyeok@gmail.com>
This reverts commit 105bc63295,
which (although correct), resulted in a backward incompatible
change.
We can re-implement this in future, after this changes goes
through a deprecation cycle
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Ke Li <kel@splunk.com>
Add missing changes
Signed-off-by: Ke Li <kel@splunk.com>
User errors.New to create error
Signed-off-by: Ke Li <kel@splunk.com>
This fix tries to fix 29667 where image's `CMD` is modified
after `WORKDIR` in Dockerfile.
The value of `b.runConfig.Cmd` was modified in the processing
of `WORKDIR`, in order to fix 28902. However, the same
`b.runConfig.Cmd` is passed to `commit()`.
This fix restored the `b.runConfig.Cmd` before `commit()`
the image for `WORKDIR`.
A test has been added.
This fix fixes 29667.
This fix is related to 28902, 28909, 28514.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to fix the issue in 29619 where
labels passed from `build --labels` are not sorted.
As a result, if multiple labels have been passed,
each `docker build --labels A=A --labels B=B --labels C=C`
will generate different layers.
This fix fixes the issue by sort the Labels before
they are concatenated to `LABEL ...`.
A unit test has been added to cover the changes
This fix fixes 29619.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to fix the issue raised in 29486 where interrupted
`docker build` leaves some tmp files in `/var/lib/docker/tmp`.
With tmp file name prefixed with `/var/lib/docker/tmp/docker-builderXXXXXX`.
The reason for the issue is that in `MakeTarSumContext()`:
```
if err := chrootarchive.Untar(sum, root, nil); err != nil {
return nil, err
}
```
the `err` is shadowed and caused the clean up function in `defer func()`
not being called.
This fix fixes the issue.
This fix is tested manually, as was specified in 29486:
```
rm -rf /var/lib/docker/tmp
mkdir repro && cd repro
fallocate -l 300M bigfile
cat > Dockerfile <<EOF
FROM scratch
COPY ./bigfile /
EOF
docker build .
{Cancel}
ls -la /var/lib/docker/tmp
```
This fix fixes 29486.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This reverts 26103. 26103 was trying to make it so that if someone did:
docker build --build-arg FOO .
and FOO wasn't set as an env var then it would pick-up FOO from the
Dockerfile's ARG cmd. However, it went too far and removed the ability
to specify a build arg w/o any value. Meaning it required the --build-arg
param to always be in the form "name=value", and not just "name".
This PR does the right fix - it allows just "name" and it'll grab the value
from the env vars if set. If "name" isn't set in the env then it still needs
to send "name" to the server so that a warning can be printed about an
unused --build-arg. And this is why buildArgs in the options is now a
*string instead of just a string - 'nil' == mentioned but no value.
Closes#29084
Signed-off-by: Doug Davis <dug@us.ibm.com>
Allow built images to be squash to scratch.
Squashing does not destroy any images or layers, and preserves the
build cache.
Introduce a new CLI argument --squash to docker build
Introduce a new param to the build API endpoint `squash`
Once the build is complete, docker creates a new image loading the diffs
from each layer into a single new layer and references all the parent's
layers.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
As is seem in the comment of `normaliseWorkdir` for windows:
```
...
// WORKDIR c:\\foo --> C:\foo
// WORKDIR \\foo --> C:\foo
...
```
However, this is not the case in the current implementation because
`filepath.FromSlash` is used and `FromSlash` does not replace multiple
separator with a single one (`file.Clean` does).
So `normaliseWorkdir` does not truly normalize workdir.
This fix changes the implementation of `normaliseWorkdir` and use
`filepath.Clean` instead of `filepath.FromSlash`.
Additional test cases have been added to the unit test.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Just to help the next time someone goes looking for it while debugging.
Like @jhowardmsft and I did while looking at #27545.
Signed-off-by: Doug Davis <dug@us.ibm.com>
