Commit graph

92 commits

Author SHA1 Message Date
Brian Goff
a7d4af84bd pull: Validate layer digest format
Otherwise a malformed or empty digest may cause a panic.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2021-01-28 21:33:12 +00:00
Brian Goff
495d623ae5 Add fallback for pull by tag
Some registries seem to be non-conformant and return a not found error
when pulling by digest (which docker now does all the time).
To work around this, fallback when all of the following are true:

1. Image reference is a tag
2. Tag->digest resolution succeeds
3. Fetch by resolved digest fails with a "not found" error.

This is intentionally not caching the manifests to reduce complexity for
this edge case.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-12-04 23:50:52 +00:00
Brian Goff
9ca3bb632e Store image manifests in containerd content store
This allows us to cache manifests and avoid extra round trips to the
registry for content we already know about.

dockerd currently does not support containerd on Windows, so this does
not store manifests on Windows, yet.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-11-05 20:02:18 +00:00
Brian Goff
e4cf1c7336 pull: use tag service for pulling tagged reference
The tag service does a `HEAD` request to get the manifest digest, where
we can then do a `GET` against the digest.

The `GET` by tag is not cacheable, but the `GET` against the digest is.
This allows proxies to work way better.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-10-30 01:57:11 +00:00
Sebastiaan van Stijn
0cc7ad0a7d
distribution: remove use of deprecated os.SEEK_END
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-11-25 18:51:57 +01:00
Brian Goff
e443512ce4 Use ocischema package instead of custom handler
Previously we were re-using schema2.DeserializedManifest to handle oci
manifests. The issue lies in the fact that distribution started
validating the media type string during json deserialization. This
change broke our usage of that type.

Instead distribution now provides direct support for oci schemas, so use
that instead of our custom handlers.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2019-10-11 10:19:29 -07:00
Kir Kolyshkin
a7942baf76
Use io.SeekStart instead of os.SEEK_SET
Fixes linter warnings like this one:

> distribution/pull_v2.go:229:39: SA1019: os.SEEK_SET is deprecated: Use io.SeekStart, io.SeekCurrent, and io.SeekEnd.  (staticcheck)

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2019-09-18 12:57:31 +02:00
Sebastiaan van Stijn
07ff4f1de8
goimports: fix imports
Format the source according to latest goimports.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-18 12:56:54 +02:00
Tibor Vass
647dfe99a5 distribution: modify warning logic when pulling v2 schema1 manifests
The warning on pull was incorrectly asking to contact registry admins.
It is kept on push however.

Pulling manifest lists with v2 schema1 manifests will not be supported thus
there is a warning for those, but wording changed to suggest repository author
to upgrade.

Finally, a milder warning on regular pull is kept ONLY for DockerHub users
in order to incite moving away from schema1.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-09-12 18:52:49 +00:00
Tibor Vass
d35f8f4329 Add deprecation message for schema1
This will add a warning log in the daemon, and will send the message
to be displayed by the CLI.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-06-18 01:40:25 +00:00
Stefan Scherer
4b9db209fe
Improve 'no matching manifest' error
Signed-off-by: Stefan Scherer <scherer_stefan@icloud.com>
2019-01-15 18:24:15 +01:00
Tonis Tiigi
337ba71fc1 distribution: fix passing platform struct to puller
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-27 14:59:31 -07:00
John Howard
35193c0e7d LCOW: Auto-select OS
Signed-off-by: John Howard <jhoward@microsoft.com>

Addresses https://github.com/moby/moby/pull/35089#issuecomment-367802698.
This change enables the daemon to automatically select an image under LCOW
that can be used if the API doesn't specify an explicit platform.

For example:

FROM supertest2014/nyan
ADD Dockerfile /

And docker build . will download the linux image (not a multi-manifest image)

And similarly docker pull ubuntu will match linux/amd64
2018-06-26 11:37:43 -07:00
Sebastiaan van Stijn
f23c00d870
Various code-cleanup
remove unnescessary import aliases, brackets, and so on.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-23 17:50:54 +02:00
Kir Kolyshkin
7d62e40f7e Switch from x/net/context -> context
Since Go 1.7, context is a standard package. Since Go 1.9, everything
that is provided by "x/net/context" is a couple of type aliases to
types in "context".

Many vendored packages still use x/net/context, so vendor entry remains
for now.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-23 13:52:44 -07:00
John Howard
83908836d3 Windows: Block pulling uplevel images
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-02-26 12:33:54 -08:00
Daniel Nephin
4f0d95fa6e Add canonical import comment
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2018-02-05 16:51:57 -05:00
John Howard
ce8e529e18 LCOW: Re-coalesce stores
Signed-off-by: John Howard <jhoward@microsoft.com>

The re-coalesces the daemon stores which were split as part of the
original LCOW implementation.

This is part of the work discussed in https://github.com/moby/moby/issues/34617,
in particular see the document linked to in that issue.
2018-01-18 08:29:19 -08:00
Jake Sanders
305801f58f Disambiguate mirror -> other endpoint fallbacks from V2 -> V1
Signed-off-by: Jake Sanders <jsand@google.com>
2017-11-30 20:01:03 -08:00
John Howard
d98ecf2d6c LCOW: API change JSON header to string POST parameter
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-10-06 15:26:48 -07:00
John Howard
0380fbff37 LCOW: API: Add platform to /images/create and /build
Signed-off-by: John Howard <jhoward@microsoft.com>

This PR has the API changes described in https://github.com/moby/moby/issues/34617.
Specifically, it adds an HTTP header "X-Requested-Platform" which is a JSON-encoded
OCI Image-spec `Platform` structure.

In addition, it renames (almost all) uses of a string variable platform (and associated)
methods/functions to os. This makes it much clearer to disambiguate with the swarm
"platform" which is really os/arch. This is a stepping stone to getting the daemon towards
fully multi-platform/arch-aware, and makes it clear when "operating system" is being
referred to rather than "platform" which is misleadingly used - sometimes in the swarm
meaning, but more often as just the operating system.
2017-10-06 11:44:18 -07:00
Derek McGowan
38aef56e1f
Add support for Windows version filtering on pull
Update logic to choose manifest from manifest list to check
for os version on Windows. Separate the logic for windows
and unix to keep unix logic the same.


Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-05 13:31:58 -07:00
John Howard
b291f5a317 LCOW: Spot fix for multi-os image pulling
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-09-12 23:33:34 -07:00
Daniel Nephin
709bf8b7bc Add interfacer linter
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-08-24 15:08:26 -04:00
Daniel Nephin
9b47b7b151 Fix golint errors.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2017-08-18 14:23:44 -04:00
Brian Goff
ebcb7d6b40 Remove string checking in API error handling
Use strongly typed errors to set HTTP status codes.
Error interfaces are defined in the api/errors package and errors
returned from controllers are checked against these interfaces.

Errors can be wraeped in a pkg/errors.Causer, as long as somewhere in the
line of causes one of the interfaces is implemented. The special error
interfaces take precedence over Causer, meaning if both Causer and one
of the new error interfaces are implemented, the Causer is not
traversed.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2017-08-15 16:01:11 -04:00
Derek McGowan
1009e6a40b
Update logrus to v1.0.1
Fixes case sensitivity issue

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-31 13:16:46 -07:00
John Howard
4ec9766a27 LCOW: Fix nits from 33241
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-27 11:59:49 -07:00
John Howard
b21d9ab595 LCOW: Fix schemav1 pull to extract platform
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-20 19:50:13 -07:00
John Howard
42c5c1a9ec LCOW: Pass platform through into layer store
Signed-off-by: John Howard <jhoward@microsoft.com>
2017-06-20 09:21:37 -07:00
Aaron Lehmann
0bff591bb0 Prefer digest over tag on pull
If a reference passed to the pull code contains both a tag and a digest,
currently the tag is used instead of the digest in the request to the
registry. This is the wrong behavior. Change it to favor the digest.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2017-05-15 17:17:27 -07:00
Derek McGowan
633f9252b8
Use diff ids from image configuration
The diff id resolution currently relies on a stored mapping for
archive digest to diff id. This mapping could be derived from
the image configuration if the image configuration is available.
On linux the image config is pulled in parallel and may not be
available. On windows, however, it is always pulled first and can
be used to supplement the stored mapping for images which may not
have this mapping from being side loaded. This becomes useful when
combined with side loaded foreign layers.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-05-05 10:56:40 -07:00
Tibor Vass
e10f9db26d Merge pull request #30047 from duglin/distError
Add the mediaType to the error
2017-03-10 16:44:49 -08:00
Akihiro Suda
91f11437b6 Merge pull request #30888 from estesp/moar-manifestlist-debug-output
Add debug output to manifest list parsing
2017-03-03 13:09:38 +09:00
Doug Davis
c127d9614f Add the mediaType to the error
Without this fix the error the client might see is:
	target is unknown
which wasn't helpful to me when I saw this today. With this fix I
now see:
	MediaType is unknown: 'text/html'
which helped me track down the issue to the registry I was talking to.

Signed-off-by: Doug Davis <dug@us.ibm.com>
2017-02-28 11:36:32 -08:00
Phil Estes
9a8cb9313c
Add debug output to manifest list parsing
Per request for more debug info on how the engine deals with
multi-platform "manifest list" images, this adds information about the
manifest list entries and whether it found an os/arch match, and the
digest of the match.

Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2017-02-28 14:11:13 -05:00
Sebastiaan van Stijn
8606144159
log fallback errors as "info"
Fallback errors are not an error, but an
informational message.

This changes those errors to be logged
as "Info" instead of "Error".

After this patch, debug logs look like this;

    DEBU[0050] Calling GET /_ping
    DEBU[0050] Calling POST /v1.27/images/create?fromImage=localhost%3A5000%2Ffoo&tag=latest
    DEBU[0050] Trying to pull localhost:5000/foo from https://localhost:5000 v2
    WARN[0050] Error getting v2 registry: Get https://localhost:5000/v2/: http: server gave HTTP response to HTTPS client
    INFO[0050] Attempting next endpoint for pull after error: Get https://localhost:5000/v2/: http: server gave HTTP response to HTTPS client
    DEBU[0050] Trying to pull localhost:5000/foo from http://localhost:5000 v2
    INFO[0050] Attempting next endpoint for pull after error: manifest unknown: manifest unknown
    DEBU[0050] Trying to pull localhost:5000/foo from https://localhost:5000 v1
    DEBU[0050] attempting v1 ping for registry endpoint https://localhost:5000/v1/
    DEBU[0050] Fallback from error: Get https://localhost:5000/v1/_ping: http: server gave HTTP response to HTTPS client
    INFO[0050] Attempting next endpoint for pull after error: Get https://localhost:5000/v1/_ping: http: server gave HTTP response to HTTPS client
    DEBU[0050] Trying to pull localhost:5000/foo from http://localhost:5000 v1
    DEBU[0050] [registry] Calling GET http://localhost:5000/v1/repositories/foo/images
    ERRO[0050] Not continuing with pull after error: Error: image foo:latest not found

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-02-24 14:27:35 +01:00
Vincent Demeester
c31f73abfd Merge pull request #29001 from darrenstahlmsft/WindowsOnLinux
Block pulling Windows images on non-Windows daemons
2017-02-16 19:57:54 +01:00
Derek McGowan
3a1279393f
Use distribution reference
Remove forked reference package. Use normalized named values
everywhere and familiar functions to convert back to familiar
strings for UX and storage compatibility.

Enforce that the source repository in the distribution metadata
is always a normalized string, ignore invalid values which are not.
Update distribution tests to use normalized values.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-07 11:08:37 -08:00
Darren Stahl
d55304065b Block Windows images on Linux
Signed-off-by: Darren Stahl <darst@microsoft.com>
2017-02-02 11:07:30 -08:00
Aaron.L.Xu
40af569164 fix typo
fix typo I found AMAP in integration-cli/*

fix typo mentioned by Allencloud

Signed-off-by: Aaron.L.Xu <likexu@harmonycloud.cn>
2017-01-19 15:52:28 +08:00
Stephen J Day
7a85579917
*: use opencontainers/go-digest package
The `digest` data type, used throughout docker for image verification
and identity, has been broken out into `opencontainers/go-digest`. This
PR updates the dependencies and moves uses over to the new type.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-06 18:48:41 -08:00
Derek McGowan
3c7676a057
Abstract distribution interfaces from image specific types
Move configurations into a single file.
Abstract download manager in pull config.
Add supports for schema2 only and schema2 type checking.
Add interface for providing push layers.
Abstract image store to generically handle configurations.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2016-12-19 10:55:00 -08:00
Victor Vieux
f644e758bd remove old media type compat for plugins
Signed-off-by: Victor Vieux <vieux@docker.com>
2016-12-07 17:35:09 -08:00
Darren Stahl
dee2968560 Stop retrying pull on UnkownBlob error
Signed-off-by: Darren Stahl <darst@microsoft.com>
2016-11-29 15:06:25 -08:00
Derek McGowan
c85eb00841
Update for distribution vendor
Handle updates to reference package.
Updates for refactoring of challenge manager.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-10 17:34:12 -08:00
Victor Vieux
970b23db2e update media_type
Signed-off-by: Victor Vieux <vieux@docker.com>
2016-11-10 15:51:32 -08:00
Aaron Lehmann
99c59d5988 distribution: Remove misleading comment
Remove the following comment in pullV2Tag:

// NOTE: not using TagService.Get, since it uses HEAD requests
// against the manifests endpoint, which are not supported by
// all registry versions.

This is actually not an issue, because TagService.Get does a fallback to
GET if HEAD fails. It has done this ever since TagService was added to
the distribution API, so this comment was probably based on an early
version of TagService before it was merged, or was always a
misunderstanding.

However, we continue to use ManifestService.Get instead because it
saves a round trip. The manifest can be retrieved directly instead of
resolving the digest first.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-11-04 17:10:47 -07:00
Sebastiaan van Stijn
762556c705 Merge pull request #26564 from miminar/prevent-blob-uploads
Avoid unnecessary blob uploads
2016-10-01 12:53:10 +02:00
Michal Minář
d3bd14a4fb
Allow to mock V2MetadataService in unit tests
Signed-off-by: Michal Minář <miminar@redhat.com>
2016-09-29 14:16:00 +02:00