Signed-off-by: John Howard <jhoward@microsoft.com>
The re-coalesces the daemon stores which were split as part of the
original LCOW implementation.
This is part of the work discussed in https://github.com/moby/moby/issues/34617,
in particular see the document linked to in that issue.
Signed-off-by: John Howard <jhoward@microsoft.com>
This PR has the API changes described in https://github.com/moby/moby/issues/34617.
Specifically, it adds an HTTP header "X-Requested-Platform" which is a JSON-encoded
OCI Image-spec `Platform` structure.
In addition, it renames (almost all) uses of a string variable platform (and associated)
methods/functions to os. This makes it much clearer to disambiguate with the swarm
"platform" which is really os/arch. This is a stepping stone to getting the daemon towards
fully multi-platform/arch-aware, and makes it clear when "operating system" is being
referred to rather than "platform" which is misleadingly used - sometimes in the swarm
meaning, but more often as just the operating system.
Use strongly typed errors to set HTTP status codes.
Error interfaces are defined in the api/errors package and errors
returned from controllers are checked against these interfaces.
Errors can be wraeped in a pkg/errors.Causer, as long as somewhere in the
line of causes one of the interfaces is implemented. The special error
interfaces take precedence over Causer, meaning if both Causer and one
of the new error interfaces are implemented, the Causer is not
traversed.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This showed up in a recent profile.
Search doesn't need to take a lock on the store, because digestset has
its own locking.
Some other methods can get by with a read lock instead of an exclusive
lock.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Commit the rwLayer to get the correct DiffID
Refacator copy in thebuilder
move more code into exportImage
cleanup some windows tests
Release the newly commited layer.
Set the imageID on the buildStage after exporting a new image.
Move archiver to BuildManager.
Have ReleaseableLayer.Commit return a layer
and store the Image from exportImage in the local imageSources cache
Remove NewChild from image interface.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Add CreateImage() to the daemon
Refactor daemon.Comit() and expose a Image.NewChild()
Update copy to use IDMappings.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Issue Description:
* 1. Saving more than one images, `docker save -o a.tar aaa bbb`
* 2. Delete the last image which in saving progress. `docker rmi bbb`
Espected:
Saving images operation shouldn't be disturbed. But the real result is that failed to
save image and get an error as below:
`Error response from daemon: open
/var/lib/docker/image/devicemapper/imagedb/content/sha256/7c24e4d533a76e801662ad1b7e6e06bc1204f80110b5623e96ba2877c51479a1:
no such file or directory`
Analysis:
1. While saving more than one images, it will get all the image info from reference/imagestore,
and then using the `cached data` to save the images to a tar file.
2. But this process doesn't have a resource lock, if a deletion operation comes, the image will be deleted,
so saving operation will fail.
Solution:
When begin to save an image, `Get` all the layers first. then the
deletion operation won't delete the layers.
Signed-off-by: Wentao Zhang <zhangwentao234@huawei.com>
Instead of mutating and reverting, just create a copy and pass the copy
around.
Add a unit test for builder dispatcher.run
Fix two test failures
Fix image history by adding a CreatedBy to commit options. Previously the
createdBy field was being created by modifying a reference to the runConfig that
was held from when the container was created.
Fix a test that expected a trailing slash. Previously the runConfig was being
modified by container create. Now that we're creating a copy of runConfig
instead of sharing a reference the runConfig retains the trailing slash.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Mostly, they always have been. Most regular expressions were implemented
as `[\w][\w.-]{0,127]`, which actually allows 128 characters, since we
start with a character in the first expression.
This "fact" has been backported to the existing specifications where
length is mentioned. For the most part, no ill-effects should come of
this, unless someone has optimized to hold the length of a tag in a 7
bit integer.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
I noticed that we're using a homegrown package for assertions. The
functions are extremely similar to testify, but with enough slight
differences to be confusing (for example, Equal takes its arguments in a
different order). We already vendor testify, and it's used in a few
places by tests.
I also found some problems with pkg/testutil/assert. For example, the
NotNil function seems to be broken. It checks the argument against
"nil", which only works for an interface. If you pass in a nil map or
slice, the equality check will fail.
In the interest of avoiding NIH, I'm proposing replacing
pkg/testutil/assert with testify. The test code looks almost the same,
but we avoid the confusion of having two similar but slightly different
assertion packages, and having to maintain our own package instead of
using a commonly-used one.
In the process, I found a few places where the tests should halt if an
assertion fails, so I've made those cases (that I noticed) use "require"
instead of "assert", and I've vendored the "require" package from
testify alongside the already-present "assert" package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Remove forked reference package. Use normalized named values
everywhere and familiar functions to convert back to familiar
strings for UX and storage compatibility.
Enforce that the source repository in the distribution metadata
is always a normalized string, ignore invalid values which are not.
Update distribution tests to use normalized values.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
After https://github.com/docker/docker/pull/28926, if saving multiple images
which have common layers at same time, the common layers can't share in the tar archive
because the hash ID changes because of the Create time. The Create time is used for
pre v1.9 which treat each layer as a image and make no sense for after v1.10.
To make the hash ID consistent and keep the image save from >1.10 working properly
on pre v1.9, using a constant Create time `time.Unix(0,0)`.
Signed-off-by: Lei Jitang <leijitang@huawei.com>
The `digest` data type, used throughout docker for image verification
and identity, has been broken out into `opencontainers/go-digest`. This
PR updates the dependencies and moves uses over to the new type.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
- Use the word letter rather than character to refer to letters ;) when trying to specify that only letters and numbers can be used, and not ANY character...
- Small corrections
Fixes#29821
Signed-off-by: Timothy Hobbs <timothy@hobbs.cz>
This fix tries to address the issue raised in 25529 wehre the
image/load API returns `application/json` for quiet=0 and
`text/plain` for quite=1.
This fix makes the change so that `application/json` is returned
for both quiet=0 and quite=1.
This fix has been tested manually.
This fix fixes 25529.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Windows base layers are no longer the special "layers+base" type, so we can remove all the special handling for that.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
Relative paths are now calculated from a base path rather than from the file path, which gets treated like a directory.
Symlinks will now properly point to the file as "../<layer dir>/layer.tar" rather the incorrect "../../<layer dir>/layer.tar".
Fixes#24951
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The image spec did not match the regexp that validates tags. It
neglected to mention that period and dash characters are allowed in
tags, as long as they are not the first character. It also did not
mention the length limit for tags.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
When calling docker load on an image tar containing a compressed layer,
apply NewProgressReader to the compressed layer (whose size is known), not
the uncompressed stream. This fixes progress reporting to the client in
this case.
Signed-off-by: John Starks <jostarks@microsoft.com>
During a `docker load` there are times when nothing is printed
to the screen, leaving the user with no idea whether something happened.
When something *is* printed, often its just something like:
```
1834950e52ce: Loading layer 1.311 MB/1.311 MB
5f70bf18a086: Loading layer 1.024 kB/1.024 kB
```
which isn't necessarily the same as the image IDs.
This PR will either show:
- all of the tags for the image, or
- all of the image IDs if there are no tags
Sample output:
```
$ docker load -i busybox.tar
Loaded image: busybox:latest
$ docker load -i a.tar
Loaded image ID: sha256:47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb
```
IOW, show the human-friendly stuff first and then only if there are no tags
default back to the image IDs, so they have something to work with.
For me this this is needed because I have lots of images and after a
recent `docker load` I had no idea what image I just imported and had a
hard time figuring it out. This should fix that by telling the user
which images they just imported.
I'll add tests once there's agreement that we want this change.
Signed-off-by: Doug Davis <dug@us.ibm.com>
The image spec in image/spec/v1.md is quite a bit out of date. Not only
is it missing the changes that went into 1.10 for content
addressability, but it has inaccuracies that date back further, such as
mentioning storing tarsum in the image configuration.
This commit creates image/spec/v1.1.md which brings the specification up
to date. It discusses content addressability, new fields in the image
configuration, the repository/tag grammar, and the current mechanism for
exporting an image.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Now that we are checking if the image and host have the same architectures
via #21272, this value should be null so that the test passes on non-x86
machines
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
Previously, Windows only supported running with a OS-managed base image.
With this change, Windows supports normal, Linux-like layered images, too.
Signed-off-by: John Starks <jostarks@microsoft.com>
These fields are needed to specify the exact version of Windows that an
image can run on. They may be useful for other platforms in the future.
This also changes image.store.Create to validate that the loaded image is
supported on the current machine. This change affects Linux as well, since
it now validates the architecture and OS fields.
Signed-off-by: John Starks <jostarks@microsoft.com>
Restores the correct parent chain relationship
between images on docker load if multiple images
have been saved.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Fixes: #20709
As discussed in the issue, we need refine the message to
help user more understood, what happened for non-exist image.
Signed-off-by: Kai Qiang Wu(Kennan) <wkqwu@cn.ibm.com>
Save was failing file integrity checksums due to bugs in both
Windows and Docker. This commit includes fixes to file time handling
in tarexport and system.chtimes that are necessary along with
the Windows platform fixes to correctly support save. With this
change, sysfile_backups for windowsfilter driver are no longer
needed, so that code is removed.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
On migration 2 different images can end up with same
content addressable ID, meaning `SetParent` will be called
multiple times. Previous version did not clear the old
in-memory reference.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Currently, daemonbuilder package (part of daemon) implemented the
builder backend. However, it was a very thin wrapper around daemon
methods and caused an implementation dependency for api/server build
endpoint. api/server buildrouter should only know about the backend
implementing the /build API endpoint.
Removing daemonbuilder involved moving build specific methods to
respective files in the daemon, where they fit naturally.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
- Make the API client library completely standalone.
- Move windows partition isolation detection to the client, so the
driver doesn't use external types.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Removed images were not cleaned up from the
digest-set that is used for the search index.
Fixes#18437
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Moved a defer up to a better spot.
Fixed TestUntarPathWithInvalidDest to actually fail for the right reason
Closes#18170
Signed-off-by: Doug Davis <dug@us.ibm.com>
Currently, the resources associated with the io.Reader returned by
TarStream are only freed when it is read until EOF. This means that
partial uploads or exports (for example, in the case of a full disk or
severed connection) can leak a goroutine and open file. This commit
changes TarStream to return an io.ReadCloser. Resources are freed when
Close is called.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
The image store abstracts image handling. It keeps track of the
available images, and makes it possible to delete existing images or
register new ones. The image store holds references to the underlying
layers for each image.
The image/v1 package provides compatibility functions for interoperating
with older (non-content-addressable) image structures.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Generate a hash chain involving the image configuration, layer digests,
and parent image hashes. Use the digests to compute IDs for each image
in a manifest, instead of using the remotely specified IDs.
To avoid breaking users' caches, check for images already in the graph
under old IDs, and avoid repulling an image if the version on disk under
the legacy ID ends up with the same digest that was computed from the
manifest for that image.
When a calculated ID already exists in the graph but can't be verified,
continue trying SHA256(digest) until a suitable ID is found.
"save" and "load" are not changed to use a similar scheme. "load" will
preserve the IDs present in the tar file.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Move graph related functions in image to graph package.
Consolidating graph functionality is the first step in refactoring graph into an image store model.
Subsequent refactors will involve breaking up graph into multiple types with a strongly defined interface.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
While reading some of the docs I noticed a few errors, so I ran
misspellings (https://pypi.python.org/pypi/misspellings) on markdown files
Signed-off-by: Joe Gordon <joe.gordon0@gmail.com>
The checksum is now being stored in a separate file beside the image
JSON file.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
While checksums are verified when a layer is pulled from v2 registries,
there are known issues where the checksum may change when the layer diff
is computed again. To avoid these issues, the checksum should no longer
be computed and stored until after it has been extracted to the docker
storage driver. The checksums are instead computed lazily before they
are pushed to a v2 registry.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
The title `Image JSON Schema` was used as a header in the section
which describes the layout and fields of the image metadata JSON
file. It was pointed out that `JSON Schema` is its own term for
describing JSON in a machine-and-human-readable format, while the
word "Schema" in this context was used more generically to say that
the section is meant to be an example and outline of the Image JSON.
http://spacetelescope.github.io/understanding-json-schema/
This section now has the title `Image JSON Description` in order
to not cause this confusion.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Updates `image.StoreImage()` to always ensure that images
that are installed in Docker have a tarsum.v1 checksum.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
While the v2 pull operation is writing the body of the layer blob to disk
it now computes the tarsum checksum of the archive before extracting it to
the backend storage driver. If the checksum does not match that from the
image manifest an error is raised.
Also adds more debug logging to the pull operation and fixes existing test
cases which were failing. Adds a reverse lookup constructor to the tarsum
package so that you can get a tarsum object using a checksum label.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
