When building a manifest during a push operation, all layers must have
an associated descriptor. If a layer is missing a descriptor, that leads
to a panic.
A break inside a switch in layerAlreadyExists meant to break from the
loop surrounding the switch, but instead breaks from the switch. This
causes the loop to continue, and can overwrite the descriptor with an
empty one, leading to the panic.
Also, fix layerAlreadyExists not to abort the push when a speculative
stat on a candidate layer digest fails with an error. This could happen
in situations like a potential cross-repository mount where the user
does not have permission to access the source repository.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
(cherry picked from commit e0702e9f37)
Signed-off-by: Victor Vieux <victorvieux@gmail.com>
Translate pull errors to provide a more consistent and user friendly
error message.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Remove the following comment in pullV2Tag:
// NOTE: not using TagService.Get, since it uses HEAD requests
// against the manifests endpoint, which are not supported by
// all registry versions.
This is actually not an issue, because TagService.Get does a fallback to
GET if HEAD fails. It has done this ever since TagService was added to
the distribution API, so this comment was probably based on an early
version of TagService before it was merged, or was always a
misunderstanding.
However, we continue to use ManifestService.Get instead because it
saves a round trip. The manifest can be retrieved directly instead of
resolving the digest first.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Allow built images to be squash to scratch.
Squashing does not destroy any images or layers, and preserves the
build cache.
Introduce a new CLI argument --squash to docker build
Introduce a new param to the build API endpoint `squash`
Once the build is complete, docker creates a new image loading the diffs
from each layer into a single new layer and references all the parent's
layers.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Classify blobs into three categories based on size.
Use a very limited number of mount attempts and no existence check for
small blobs. Use more attempts for bigger blobs.
Also remember blob associations during layer existence check.
Blob digests are now checked in the target repository from newest to
latest. If the blob exists and the metadata entry does not, it will be
created. If the blob is not found, the metadata entry will be removed.
Signed-off-by: Michal Minář <miminar@redhat.com>
Don't fallback back immediately to blob upload if the cross-repo mount
fails and layer upload is initiated by registry.
Instead cancel the upload and re-try cross-repo mount from different
source repository before doing full re-upload.
Signed-off-by: Michal Minář <miminar@redhat.com>
to avoid unnecessary blob re-uploads.
Cross repository mount from particular repo will most probably fail if
the user pushing to the registry is not the same as the one who pulled
or pushed to the source repo.
This PR attempts first to cross-repo mount from the source repositories
associated with the pusher's auth config. Then it falls back to other
repositories sorted from the most similar to the target repo to the
least.
It also prevents metadata deletion in cases where cross-repo mount fails
and the auth config hashes differ.
Signed-off-by: Michal Minář <miminar@redhat.com>
Windows base layers are no longer the special "layers+base" type, so we can remove all the special handling for that.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
Ensure that cancellation of a pull propagates rather than continuing to
container creation. This ensures that the `Prepare` method is properly
re-entrant.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This fixes the hard coded restriction for non-linux platforms to v2 registries. Previously, the check was above the flag parsing, which would overwrite the hard coded value and prevent correct operation. This change also removes the related daemon flag from Windows to avoid confusion, as it has no meaning when the value is going to always be hard coded to true.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
Updates the v1 search endpoint to also support v2 auth when an identity token is given.
Only search v1 endpoint is supported since there is not v2 search currently defined to replace it.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
If the remote registry responds with a NAME_UNKNOWN error, treat this as
a fatal error and don't retry the push.
Tested against an ECR registry.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Always attempt to add digest even when tag already exists.
Ensure digest does not currently exist.
When image id is mismatched, output an error log.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Layer uploads are deduplicated by a "key" made up of the layer DiffID
and the repository name. The repository name being used to form this key
was a remote version of the name that didn't include the name of the
registry. Consequently, pushes of the same layer in a repository with
the same remote name to different registries would wrongly be
deduplicated.
Correct the key by using the full name of the repository, which includes
the registry hostname as well as the image's name.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This patch introduces a new experimental engine-level plugin management
with a new API and command line. Plugins can be distributed via a Docker
registry, and their lifecycle is managed by the engine.
This makes plugins a first-class construct.
For more background, have a look at issue #20363.
Documentation is in a separate commit. If you want to understand how the
new plugin system works, you can start by reading the documentation.
Note: backwards compatibility with existing plugins is maintained,
albeit they won't benefit from the advantages of the new system.
Signed-off-by: Tibor Vass <tibor@docker.com>
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
This fix tries to fix logrus formatting by removing `f` from
`logrus.[Error|Warn|Debug|Fatal|Panic|Info]f` when formatting string
is not present.
This fix fixes#23459.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
… and refactor a little bit some daemon on the way.
- Move `SearchRegistryForImages` to a new file (`daemon/search.go`) as
`daemon.go` is getting pretty big.
- `registry.Service` is now an interface (allowing us to decouple it a
little bit and thus unit test easily).
- Add some unit test for `SearchRegistryForImages`.
- Use UniqueExactMatch for search filters
- And use empty restore id for now in client.ContainerStart.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This fix tries to address issues raised in #20936 and #22443
where `docker pull` or `docker push` fails because of the
concurrent connection failing.
Currently, the number of maximum concurrent connections is
controlled by `maxDownloadConcurrency` and `maxUploadConcurrency`
which are hardcoded to 3 and 5 respectively. Therefore, in
situations where network connections don't support multiple
downloads/uploads, failures may encounter for `docker push`
or `docker pull`.
This fix tries changes `maxDownloadConcurrency` and
`maxUploadConcurrency` to adjustable by passing
`--max-concurrent-uploads` and `--max-concurrent-downloads` to
`docker daemon` command.
The documentation related to docker daemon has been updated.
Additional test case have been added to cover the changes in this fix.
This fix fixes#20936. This fix fixes#22443.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
