Commit graph

350 commits

Author SHA1 Message Date
Tonis Tiigi
9adad264d2 distribution: match manifest list resolution with containerd
Make finding the correct runtime image from image index
more compliant with OCI spec and match containerd implementation.

Changes:
- Manifest list is allowed to contain manifest lists
- Unknown mediatype inside manifest list is skipped instead of causing an error
- Platform in descriptor is optional 

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-06-02 11:21:07 -07:00
Sebastiaan van Stijn
14fdd97b69
Remove unused distribution/metadata/V1IDService
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-05-10 23:39:22 +02:00
Eng Zer Jun
7873c27cfb
all: replace strings.Replace with strings.ReplaceAll
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-05-09 19:45:40 +08:00
Sebastiaan van Stijn
2b0da89366
distribution: move newPusher() and newPuller() together with definition
Also moving writeStatus() to the puller, which is where it's used, and makes
it slightly easier to consume.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:13:39 +02:00
Sebastiaan van Stijn
566c8db66d
distribution: add GetRepository(), un-export NewV2Repository, ValidateRepoName
These were only exported to facilitate ImageService.GetRepository() (used for
the `GET /distribution/{name:.*}/json` endpoint.

Moving the core functionality of that to the distribution package makes it
more consistent with (e.g.) "pull" operations, and allows us to keep more things
internal.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:12:02 +02:00
Sebastiaan van Stijn
572c7e0184
distribution: remove RootFSFromConfig(), PlatformFromConfig() from ImageConfigStore
These functions did not require the ImageConfigStore, so could just be local
utilities.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:12:00 +02:00
Sebastiaan van Stijn
1e75ab0ab9
distribution: remove Pusher interface, NewPusher(), and redundant V1 checks
It's only used internally, so we can refer to the implementation itself. Given
that RegistryService.LookupPushEndpoints now only returns V2 endpoints, we
no longer need to check if an endpoint is possibly V1.

Also rename some types that had "v2" in their name, now that we only support v2.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:11:54 +02:00
Sebastiaan van Stijn
41999abcbe
distribution: remove Puller interface, remove redundant V1 checks
It's only used internally, so we can refer to the implementation itself. Given
that RegistryService.LookupPullEndpoints now only returns V2 endpoints, we
no longer need to check if an endpoint is possibly V1.

Also rename some types that had "v2" in their name, now that we only support v2.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 23:11:07 +02:00
Sebastiaan van Stijn
074e41679d
distribution: un-export internal errors and error-utilities
un-exports errors that were only used internally:

- Remove ErrNoSupport as it was not emitted anywhere
- ImageConfigPullError -> imageConfigPullError
- TranslatePullError() -> translatePullError()

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 22:53:02 +02:00
Sebastiaan van Stijn
fb5485f5d0
distribution: un-export ImageTypes, make ImagePullConfig.Schema2Types optional
Use the default list of accepted mediaTypes if none were passed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-21 22:36:56 +02:00
Tonis Tiigi
482d1d15bf distribution: use the maximum compatible platform by default
When no specific platform is set, pull the platform that
most matches the current host.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-03-31 15:20:59 -07:00
Tonis Tiigi
fcc42d5682 distribution: use correct platform matcher for containerd
This uses the correct comparison with compatibility
checks for variants.

The deprecated arm variant matcher is left as is.
Although it is not needed for valid cases it is not
fully compatible as also matches some invalid
combinations, so should be removed separately.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-03-27 19:46:45 -07:00
Pete Woods
9f3b1a9664
distribution: retry downloading schema config on retryable error
fixes #43267

Signed-off-by: Pete Woods <pete.woods@circleci.com>
2022-03-05 19:36:48 +01:00
Sebastiaan van Stijn
a0230f3d9a
remove unneeded "digest" alias for "go-digest"
I think this was there for historic reasons (may have been goimports expected
this, and we used to have a linter that wanted it), but it's not needed, so
let's remove it (to make my IDE less complaining about unneeded aliases).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-04 14:49:42 +01:00
Akihiro Suda
d809ad98e3
Merge pull request #43183 from thaJeztah/cleanup_distribution
distribution/xfer: refactor to reduce public api/interface
2022-02-26 23:50:03 +09:00
Sebastiaan van Stijn
ff408210da
distribution: remove use of deprecated dial.DualStack
From the field's description [1]:

    DualStack previously enabled RFC 6555 Fast Fallback
    support, also known as "Happy Eyeballs", in which IPv4 is
    tried soon if IPv6 appears to be misconfigured and
    hanging.

    Deprecated: Fast Fallback is enabled by default. To
    disable, set FallbackDelay to a negative value.

This field was deprecated in efc185029b,
which is included in Go 1.12beta1 and up.

[1]: 2ebe77a2fd/src/net/dial.go (L54-L61)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-22 16:34:12 +01:00
Sebastiaan van Stijn
79ea1b16b1
distribution/xfer: un-export DoFunc type
It's only used internally and by non-exported functions, so it doesn't
have to be exported.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:42 +01:00
Sebastiaan van Stijn
4d6c6a5000
distribution/xfer: smaller interface, rename to DigestRegisterer
All uses of this interface already accept a DownloadDescriptor; keeping the
interface small to allow this functionality to be used by other download-descriptors,
while still being able to check for the actual functionality (to be able to register
a digest).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:40 +01:00
Sebastiaan van Stijn
047e032461
distribution/xfer: define DownloadOption type
This is mostly for documentation purposes; defining a type makes
the option(s) show up grouped on pkg.go.dev (and in godoc).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:38 +01:00
Sebastiaan van Stijn
69b0913e1f
distribution/xfer: add TODO for return error on Registered()
There's currently four implementations in our code: one in distribution, two in
BuildKit-related code, and one "mock" for testing:

- 65b8bcc321/builder/builder-next/worker/worker.go (L452-L455)
- 65b8bcc321/builder/builder-next/adapters/containerimage/pull.go (L648-L651)
- 65b8bcc321/distribution/pull_v2.go (L318-L321)
- 65b8bcc321/distribution/xfer/download_test.go (L186-L188)

All of these call a metadata-service to register the DiffID, and all of those
may return an error, which currently gets ignored.

We should consider changing the signature to return possible errors, so that
the callsite can decide how to handle them (which could be just "log" the error
or handle them in other ways).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:36 +01:00
Sebastiaan van Stijn
3309ea9419
distribution/xfer: slightly more idomatic type assertion
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:35 +01:00
Sebastiaan van Stijn
4c7dc9806c
distribution/xfer: un-export transfer interface accessors
This interface is internal to the package, so there's no need to export
it's methods.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:33 +01:00
Sebastiaan van Stijn
849d8c2d02
distribution/xfer: un-export Watcher
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:31 +01:00
Sebastiaan van Stijn
96a750aab4
distribution/xfer: un-export Transfer interface
The only implementations and uses are internal to this package.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:29 +01:00
Sebastiaan van Stijn
765844e419
distribution/xfer: un-export NewTransfer()
This is also only used internally, so no need to export.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:28 +01:00
Sebastiaan van Stijn
587c474b57
distribution/xfer: use "transferManager" in godoc
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:26 +01:00
Sebastiaan van Stijn
874b11495b
distribution/xfer: un-export transferManager.setConcurrency() and .transfer()
They're only used within the package itself, so no need to have them public.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:24 +01:00
Sebastiaan van Stijn
d746a836fc
distribution/xfer: remove TransferManager interface, un-export newTransferManager
The `TransferManager` interface only had a single implementation, and neither
`LayerDownloadManager`, nor `LayerUploadManager` currently had an option to
provide a custom implementation, so we can un-export this.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:22 +01:00
Sebastiaan van Stijn
cf31aa0fa0
distribution/xfer: fix some nits in tests
- use const instead of a variable
- remove unneeded type-conversion
- use keys in literals

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:19 +01:00
Sebastiaan van Stijn
32e5fe5099
Merge pull request #43182 from thaJeztah/layer_remove_unused_error
layer: remove unused error return from .Size() and .DiffSize()
2022-02-17 20:51:45 +01:00
Akihiro Suda
65b8bcc321
Merge pull request #43174 from thaJeztah/move_platformcheck
distribution: remove RootFSDownloadManager interface, and remove "os" argument from Download()
2022-01-26 14:08:44 +09:00
Sebastiaan van Stijn
f9a1846ca2
distribution: PushLayer.Size(): remove unused error return
None of the implementations returned an error for this function, so removing it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-24 18:45:51 +01:00
Sebastiaan van Stijn
e1ea911aba
layer: remove unused error return from .Size() and .DiffSize()
None of the implementations used return an error, so removing the error
return can simplify using these.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-24 18:45:47 +01:00
Sebastiaan van Stijn
bb66ebd621
distribution: xfer.LayerDownloadManager.Download(): remove "os" argument
This argument was added for LCOW support, but it was only used to verify if
the passed platform (OS) matched the host. Given that all uses of this function
(except for one) passed runtime.GOOS, we may as well move the check to that
location.

We should do more cleaning up after this, and perform such validations early,
instead of passing platform around in too many places where it's only used for
similar validations. This is a first step in that direction.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-21 15:07:55 +01:00
Sebastiaan van Stijn
0b0a995d9d
distribution: remove RootFSDownloadManager interface
This interface only had a single implementation (xfer.LayerDownloadManager),
and all places where it was used already imported the xfer package.
Removing the interface, also makes it a closer match to the "upload" part,
as `xfer.LayerUploadManager()` did not use an interface.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-21 13:53:36 +01:00
Ziheng Liu
3dda7311cd xfer: fix a simple goroutine leak in unit test
Signed-off-by: Ziheng Liu <lzhfromustc@gmail.com>
2022-01-07 20:28:37 -08:00
Samuel Karp
7c216bcabe
distribution: validate blob type
Signed-off-by: Samuel Karp <skarp@amazon.com>
2021-11-23 17:19:01 -08:00
Sebastiaan van Stijn
1cdad98a7a
distribution: do not retry unsupported layer media types
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-10-27 12:33:30 +02:00
Sebastiaan van Stijn
8684f482e4
Merge pull request #42782 from dkkb/McGrady/code_improvement
Remove platform argument from Puller interface.
2021-08-30 13:19:08 +02:00
Eng Zer Jun
c55a4ac779
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-27 14:56:57 +08:00
Da McGrady
6f5beea146
Remove platform argument from Puller interface.
The platform argument is unneeded because ImagePullConfig is already in the v2Puller struct.

Signed-off-by: Da McGrady <dabkb@aol.com>
2021-08-27 09:34:10 +08:00
Sebastiaan van Stijn
686be57d0a
Update to Go 1.17.0, and gofmt with Go 1.17
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-24 23:33:27 +02:00
Sebastiaan van Stijn
dc70e55c9b
distribution: remove LCOW code
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 13:36:17 +02:00
Sebastiaan van Stijn
2773f81aa5
Merge pull request #42445 from thaJeztah/bump_golang_ci
[testing] ~update~ fix linting issues found by golangci-lint v1.40.1
2021-06-16 22:15:01 +02:00
Sebastiaan van Stijn
dc7cbb9b33
remove layerstore indexing by OS (used for LCOW)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 17:49:11 +02:00
Sebastiaan van Stijn
d13997b4ba
gosec: G601: Implicit memory aliasing in for loop
plugin/v2/plugin.go:141:50: G601: Implicit memory aliasing in for loop. (gosec)
                    updateSettingsEnv(&p.PluginObj.Settings.Env, &s)
                                                                 ^
    libcontainerd/remote/client.go:572:13: G601: Implicit memory aliasing in for loop. (gosec)
                cpDesc = &m
                         ^
    distribution/push_v2.go:400:34: G601: Implicit memory aliasing in for loop. (gosec)
                (metadata.CheckV2MetadataHMAC(&mountCandidate, pd.hmacKey) ||
                                              ^
    builder/dockerfile/builder.go:261:84: G601: Implicit memory aliasing in for loop. (gosec)
            currentCommandIndex = printCommand(b.Stdout, currentCommandIndex, totalCommands, &meta)
                                                                                             ^
    builder/dockerfile/builder.go:278:46: G601: Implicit memory aliasing in for loop. (gosec)
            if err := initializeStage(dispatchRequest, &stage); err != nil {
                                                       ^
    daemon/container.go:283:40: G601: Implicit memory aliasing in for loop. (gosec)
            if err := parser.ValidateMountConfig(&cfg); err != nil {
                                                 ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:29 +02:00
Tibor Vass
588da41f52
distribution: fix errors tests
Signed-off-by: Tibor Vass <tibor@docker.com>
2021-02-23 09:49:49 +01:00
Tibor Vass
7a50fe8a52
Remove more of registry v1 code.
Signed-off-by: Tibor Vass <tibor@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-23 09:49:46 +01:00
Brian Goff
c747d9f8ee
pull: Validate layer digest format
Otherwise a malformed or empty digest may cause a panic.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit a7d4af84bd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-02 13:37:24 +01:00
Brian Goff
495d623ae5 Add fallback for pull by tag
Some registries seem to be non-conformant and return a not found error
when pulling by digest (which docker now does all the time).
To work around this, fallback when all of the following are true:

1. Image reference is a tag
2. Tag->digest resolution succeeds
3. Fetch by resolved digest fails with a "not found" error.

This is intentionally not caching the manifests to reduce complexity for
this edge case.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-12-04 23:50:52 +00:00