Commit graph

380 commits

Author SHA1 Message Date
Sebastiaan van Stijn
4e4f62b6db
distribution: use registry consts for API version in tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-29 10:58:38 +02:00
Sebastiaan van Stijn
8c4af5dacb
distribution: show image schema deprecation on all registries
When we added this deprecation warning, some registries had not yet
moved away from the deprecated specification, so we made the warning
conditional for pulling from Docker Hub.

That condition was added in 647dfe99a5,
which is over 4 Years ago, which should be time enough for images
and registries to have moved to current specifications.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-02 14:15:34 +02:00
Sebastiaan van Stijn
982bc0e228
distribution: update warning for deprecated image formats
- Use the same warning for both "v1 in manifest-index" and bare "v1" images.
- Update URL to use a "/go/" redirect, which allows the docs team to more
  easily redirect the URL to relevant docs (if things move).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-02 14:14:47 +02:00
Sebastiaan van Stijn
5e2a1195d7
swap logrus types for their containerd/logs aliases
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-01 13:02:55 +02:00
Sebastiaan van Stijn
846bcd7dd1
distribution: format code with gofumpt
Formatting the code with https://github.com/mvdan/gofumpt

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-29 00:30:52 +02:00
Brian Goff
74da6a6363 Switch all logging to use containerd log pkg
This unifies our logging and allows us to propagate logging and trace
contexts together.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-06-24 00:23:44 +00:00
Sebastiaan van Stijn
ab35df454d
remove pre-go1.17 build-tags
Removed pre-go1.17 build-tags with go fix;

    go mod init
    go fix -mod=readonly ./...
    rm go.mod

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-19 20:38:51 +02:00
Jeyanthinath Muthuram
307b09e7eb
fixing consistent aliases for OCI spec imports
Signed-off-by: Jeyanthinath Muthuram <jeyanthinath10@gmail.com>
2023-05-08 15:27:52 +05:30
Sebastiaan van Stijn
81e62af94a
use consistent alias for containerd's errdefs package
The signatures of functions in containerd's errdefs packages are very
similar to those in our own, and it's easy to accidentally use the wrong
package.

This patch uses a consistent alias for all occurrences of this import.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-08 19:30:33 +02:00
Cory Snider
7b3acdff5d registry: return concrete service type
Move interface definitions to the packages which use the registry
service.

https://github.com/golang/go/wiki/CodeReviewComments#interfaces

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-03-10 18:38:08 -05:00
Brian Goff
68fe202f06
Merge pull request #45050 from lmbarros/close-reader
Close DecompressStream after layer is downloaded
2023-02-21 22:57:53 +00:00
Leandro Motta Barros
bb21e0f191 Close DecompressStream after layer is downloaded
Signed-off-by: Leandro Motta Barros <lmb@stackedboxes.org>
2023-02-21 15:03:19 -03:00
Cory Snider
97921915a8 distribution/xfer: fix download fencepost bug
maxDownloadAttempts maps to the daemon configuration flag

    --max-download-attempts int
      Set the max download attempts for each pull (default 5)

and the daemon configuration machinery interprets a value of 0 as "apply
the default value" and not a valid user value (config validation/
normalization bugs notwithstanding). The intention is clearly that this
configuration value should be an upper limit on the number of times the
daemon should try to download a particular layer before giving up. So it
is surprising to have the configuration value interpreted as a _retry_
limit. The daemon will make up to N+1 attempts to download a layer! This
also means users cannot disable retries even if they wanted to.

Fix the fencepost bug so that max attempts really means max attempts,
not max retries. And fix the fencepost bug with the retry-backoff delay
so that the first backoff is 5s, not 10s.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-02-15 12:45:09 -05:00
Cory Snider
0c68b655f6 Fix loop-closure bugs in tests
...which were flagged by golangci-lint v1.51.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-02-15 12:44:52 -05:00
Tianon Gravi
fdcab58ccf Remove reference to "requireRegistryMatch"
This option appears to never have actually existed in the merged implementation. 😄

Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2023-01-05 11:28:22 -08:00
Sebastiaan van Stijn
456ea1bb1d
image: deprecate IDFromDigest()
Having this function hides what it's doing, which is just to type-cast
to an image.ID (which is a digest). Using a cast is more transparent,
so deprecating this function in favor of a regular typecast.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-01-01 12:33:00 +01:00
Sebastiaan van Stijn
a6a539497a
distribution: checkSupportedMediaType: allow additional media-types
This addresses a regression introduced in 407e3a4552,
which turned out to be "too strict", as there's old images that use, for example;

    docker pull python:3.5.1-alpine
    3.5.1-alpine: Pulling from library/python
    unsupported media type application/octet-stream

Before 407e3a4552, such mediatypes were accepted;

    docker pull python:3.5.1-alpine
    3.5.1-alpine: Pulling from library/python
    e110a4a17941: Pull complete
    30dac23631f0: Pull complete
    202fc3980a36: Pull complete
    Digest: sha256:f88925c97b9709dd6da0cb2f811726da9d724464e9be17a964c70f067d2aa64a
    Status: Downloaded newer image for python:3.5.1-alpine
    docker.io/library/python:3.5.1-alpine

This patch copies the additional media-types, using the list of types that
were added in a215e15cb1, which fixed a
similar issue.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-02 01:26:06 +01:00
Sebastiaan van Stijn
85fddc0081
distribution: remove unused RequireSchema2
It's never set, so we can remove it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 20:29:19 +01:00
Sebastiaan van Stijn
8feeaecb84
use ad-hoc libtrust key
This is only used for tests, and the key is not verified anymore, so
instead of creating a key and storing it, we can just use an ad-hoc
one.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 20:29:19 +01:00
Sebastiaan van Stijn
e854b2a459
distribution: use ad-hoc trustkey for tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 20:29:19 +01:00
Sebastiaan van Stijn
0f7c9cd27e
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
Both of these were deprecated in 55f675811a,
but the format of the GoDoc comments didn't follow the correct format, which
caused them not being picked up by tools as "deprecated".

This patch updates uses in the codebase to use the alternatives.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 16:42:13 +01:00
Sebastiaan van Stijn
6bb909f152
Merge pull request #44367 from vvoland/oci-artifacts-error
distribution: Error when pulling OCI artifacts
2022-11-03 19:39:22 +01:00
Sebastiaan van Stijn
9899820a17
Merge pull request #44389 from tianon/debug-typo
Remove errant "runtime.GOARCH" from debug message
2022-11-02 18:23:11 +01:00
Paweł Gronowski
407e3a4552
distribution: Error when pulling OCI artifacts
Currently an attempt to pull a reference which resolves to an OCI
artifact (Helm chart for example), results in a bit unrelated error
message `invalid rootfs in image configuration`.

This provides a more meaningful error in case a user attempts to
download a media type which isn't image related.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2022-11-02 12:51:03 +01:00
Tianon Gravi
5bc17c3e54 Remove long-deprecated "arm" fallback
This fallback is used when we filter the manifest list by the user-provided platform and find no matches such that we match the previous Docker behavior (before it supported variant matching).  This has been deprecated long enough that I think it's time we finally stop supporting this weird fallback, especially since it makes for buggy behavior like `docker pull --platform linux/arm/v5 alpine:3.16` leading to a `linux/arm/v6` image being pulled (I specified a variant, every manifest list entry specifies a variant, so clearly the only behavior I as a user could reasonably expect is an error that `linux/arm/v5` is not supported, but instead I get an explicitly incompatible image despite doing everything I as a user can to prevent that situation).

Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2022-11-01 15:31:07 -07:00
Tianon Gravi
3d71555a47 Remove errant "runtime.GOARCH" from debug message
This debug message already includes a full platform string, so this ends up being something like `linux/arm/v7/amd64` in the end result.

Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2022-11-01 15:28:29 -07:00
Brian Goff
27530efedb
Validate digest in repo for pull by digest
This is accomplished by storing the distribution source in the content
labels. If the distribution source is not found then we check to the
registry to see if the digest exists in the repo, if it does exist then
the puller will use it.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-10-21 01:48:59 +02:00
Sebastiaan van Stijn
8a2e1245d4
runconfig, oci, image, layer, distribution: fix empty-lines (revive)
runconfig/config_test.go:23:46: empty-lines: extra empty line at the start of a block (revive)
    runconfig/config_test.go:75:55: empty-lines: extra empty line at the start of a block (revive)

    oci/devices_linux.go:57:34: empty-lines: extra empty line at the start of a block (revive)
    oci/devices_linux.go:60:69: empty-lines: extra empty line at the start of a block (revive)

    image/fs_test.go:53:38: empty-lines: extra empty line at the end of a block (revive)
    image/tarexport/save.go:88:29: empty-lines: extra empty line at the end of a block (revive)

    layer/layer_unix_test.go:21:34: empty-lines: extra empty line at the end of a block (revive)

    distribution/xfer/download.go:302:9: empty-lines: extra empty line at the end of a block (revive)
    distribution/manifest_test.go:154:99: empty-lines: extra empty line at the end of a block (revive)
    distribution/manifest_test.go:329:52: empty-lines: extra empty line at the end of a block (revive)
    distribution/manifest_test.go:354:59: empty-lines: extra empty line at the end of a block (revive)

    registry/config_test.go:323:42: empty-lines: extra empty line at the end of a block (revive)
    registry/config_test.go:350:33: empty-lines: extra empty line at the end of a block (revive)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-28 01:58:52 +02:00
Sebastiaan van Stijn
04c339a12b
distribution: use types/registry.AuthConfig
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-07-29 23:05:18 +02:00
Sebastiaan van Stijn
968ff5ab44
fix some minor linting issues
libnetwork/firewall_linux.go:11:21: var-declaration: should drop = nil from declaration of var ctrl; it is the zero value (revive)
        ctrl *controller = nil
                           ^
    distribution/pull_v2_test.go:213:4: S1038: should use t.Fatalf(...) instead of t.Fatal(fmt.Sprintf(...)) (gosimple)
                t.Fatal(fmt.Sprintf("expected formatPlatform to show windows platform with a version, but got '%s'", result))
                ^
    integration-cli/docker_cli_build_test.go:5951:3: S1038: should use c.Skipf(...) instead of c.Skip(fmt.Sprintf(...)) (gosimple)
            c.Skip(fmt.Sprintf("Bug fixed in 18.06 or higher.Skipping it for %s", testEnv.DaemonInfo.ServerVersion))
            ^
    integration-cli/docker_cli_daemon_test.go:240:3: S1038: should use c.Skipf(...) instead of c.Skip(fmt.Sprintf(...)) (gosimple)
            c.Skip(fmt.Sprintf("New base device size (%v) must be greater than (%s)", units.HumanSize(float64(newBasesizeBytes)), units.HumanSize(float64(oldBasesizeBytes))))
            ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-07-04 10:15:28 +02:00
Tonis Tiigi
9adad264d2 distribution: match manifest list resolution with containerd
Make finding the correct runtime image from image index
more compliant with OCI spec and match containerd implementation.

Changes:
- Manifest list is allowed to contain manifest lists
- Unknown mediatype inside manifest list is skipped instead of causing an error
- Platform in descriptor is optional 

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-06-02 11:21:07 -07:00
Sebastiaan van Stijn
14fdd97b69
Remove unused distribution/metadata/V1IDService
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-05-10 23:39:22 +02:00
Eng Zer Jun
7873c27cfb
all: replace strings.Replace with strings.ReplaceAll
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-05-09 19:45:40 +08:00
Sebastiaan van Stijn
2b0da89366
distribution: move newPusher() and newPuller() together with definition
Also moving writeStatus() to the puller, which is where it's used, and makes
it slightly easier to consume.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:13:39 +02:00
Sebastiaan van Stijn
566c8db66d
distribution: add GetRepository(), un-export NewV2Repository, ValidateRepoName
These were only exported to facilitate ImageService.GetRepository() (used for
the `GET /distribution/{name:.*}/json` endpoint.

Moving the core functionality of that to the distribution package makes it
more consistent with (e.g.) "pull" operations, and allows us to keep more things
internal.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:12:02 +02:00
Sebastiaan van Stijn
572c7e0184
distribution: remove RootFSFromConfig(), PlatformFromConfig() from ImageConfigStore
These functions did not require the ImageConfigStore, so could just be local
utilities.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:12:00 +02:00
Sebastiaan van Stijn
1e75ab0ab9
distribution: remove Pusher interface, NewPusher(), and redundant V1 checks
It's only used internally, so we can refer to the implementation itself. Given
that RegistryService.LookupPushEndpoints now only returns V2 endpoints, we
no longer need to check if an endpoint is possibly V1.

Also rename some types that had "v2" in their name, now that we only support v2.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:11:54 +02:00
Sebastiaan van Stijn
41999abcbe
distribution: remove Puller interface, remove redundant V1 checks
It's only used internally, so we can refer to the implementation itself. Given
that RegistryService.LookupPullEndpoints now only returns V2 endpoints, we
no longer need to check if an endpoint is possibly V1.

Also rename some types that had "v2" in their name, now that we only support v2.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:11:07 +02:00
Sebastiaan van Stijn
074e41679d
distribution: un-export internal errors and error-utilities
un-exports errors that were only used internally:

- Remove ErrNoSupport as it was not emitted anywhere
- ImageConfigPullError -> imageConfigPullError
- TranslatePullError() -> translatePullError()

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 22:53:02 +02:00
Sebastiaan van Stijn
fb5485f5d0
distribution: un-export ImageTypes, make ImagePullConfig.Schema2Types optional
Use the default list of accepted mediaTypes if none were passed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 22:36:56 +02:00
Tonis Tiigi
482d1d15bf distribution: use the maximum compatible platform by default
When no specific platform is set, pull the platform that
most matches the current host.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-03-31 15:20:59 -07:00
Tonis Tiigi
fcc42d5682 distribution: use correct platform matcher for containerd
This uses the correct comparison with compatibility
checks for variants.

The deprecated arm variant matcher is left as is.
Although it is not needed for valid cases it is not
fully compatible as also matches some invalid
combinations, so should be removed separately.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-03-27 19:46:45 -07:00
Pete Woods
9f3b1a9664
distribution: retry downloading schema config on retryable error
fixes #43267

Signed-off-by: Pete Woods <pete.woods@circleci.com>
2022-03-05 19:36:48 +01:00
Sebastiaan van Stijn
a0230f3d9a
remove unneeded "digest" alias for "go-digest"
I think this was there for historic reasons (may have been goimports expected
this, and we used to have a linter that wanted it), but it's not needed, so
let's remove it (to make my IDE less complaining about unneeded aliases).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-04 14:49:42 +01:00
Akihiro Suda
d809ad98e3
Merge pull request #43183 from thaJeztah/cleanup_distribution
distribution/xfer: refactor to reduce public api/interface
2022-02-26 23:50:03 +09:00
Sebastiaan van Stijn
ff408210da
distribution: remove use of deprecated dial.DualStack
From the field's description [1]:

    DualStack previously enabled RFC 6555 Fast Fallback
    support, also known as "Happy Eyeballs", in which IPv4 is
    tried soon if IPv6 appears to be misconfigured and
    hanging.

    Deprecated: Fast Fallback is enabled by default. To
    disable, set FallbackDelay to a negative value.

This field was deprecated in efc185029b,
which is included in Go 1.12beta1 and up.

[1]: 2ebe77a2fd/src/net/dial.go (L54-L61)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-22 16:34:12 +01:00
Sebastiaan van Stijn
79ea1b16b1
distribution/xfer: un-export DoFunc type
It's only used internally and by non-exported functions, so it doesn't
have to be exported.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:42 +01:00
Sebastiaan van Stijn
4d6c6a5000
distribution/xfer: smaller interface, rename to DigestRegisterer
All uses of this interface already accept a DownloadDescriptor; keeping the
interface small to allow this functionality to be used by other download-descriptors,
while still being able to check for the actual functionality (to be able to register
a digest).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:40 +01:00
Sebastiaan van Stijn
047e032461
distribution/xfer: define DownloadOption type
This is mostly for documentation purposes; defining a type makes
the option(s) show up grouped on pkg.go.dev (and in godoc).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:38 +01:00
Sebastiaan van Stijn
69b0913e1f
distribution/xfer: add TODO for return error on Registered()
There's currently four implementations in our code: one in distribution, two in
BuildKit-related code, and one "mock" for testing:

- 65b8bcc321/builder/builder-next/worker/worker.go (L452-L455)
- 65b8bcc321/builder/builder-next/adapters/containerimage/pull.go (L648-L651)
- 65b8bcc321/distribution/pull_v2.go (L318-L321)
- 65b8bcc321/distribution/xfer/download_test.go (L186-L188)

All of these call a metadata-service to register the DiffID, and all of those
may return an error, which currently gets ignored.

We should consider changing the signature to return possible errors, so that
the callsite can decide how to handle them (which could be just "log" the error
or handle them in other ways).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:36 +01:00