Commit graph

340 commits

Author SHA1 Message Date
Tonis Tiigi
482d1d15bf distribution: use the maximum compatible platform by default
When no specific platform is set, pull the platform that
most matches the current host.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-03-31 15:20:59 -07:00
Tonis Tiigi
fcc42d5682 distribution: use correct platform matcher for containerd
This uses the correct comparison with compatibility
checks for variants.

The deprecated arm variant matcher is left as is.
Although it is not needed for valid cases it is not
fully compatible as also matches some invalid
combinations, so should be removed separately.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-03-27 19:46:45 -07:00
Pete Woods
9f3b1a9664
distribution: retry downloading schema config on retryable error
fixes #43267

Signed-off-by: Pete Woods <pete.woods@circleci.com>
2022-03-05 19:36:48 +01:00
Sebastiaan van Stijn
a0230f3d9a
remove unneeded "digest" alias for "go-digest"
I think this was there for historic reasons (may have been goimports expected
this, and we used to have a linter that wanted it), but it's not needed, so
let's remove it (to make my IDE less complaining about unneeded aliases).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-03-04 14:49:42 +01:00
Akihiro Suda
d809ad98e3
Merge pull request #43183 from thaJeztah/cleanup_distribution
distribution/xfer: refactor to reduce public api/interface
2022-02-26 23:50:03 +09:00
Sebastiaan van Stijn
ff408210da
distribution: remove use of deprecated dial.DualStack
From the field's description [1]:

    DualStack previously enabled RFC 6555 Fast Fallback
    support, also known as "Happy Eyeballs", in which IPv4 is
    tried soon if IPv6 appears to be misconfigured and
    hanging.

    Deprecated: Fast Fallback is enabled by default. To
    disable, set FallbackDelay to a negative value.

This field was deprecated in efc185029b,
which is included in Go 1.12beta1 and up.

[1]: 2ebe77a2fd/src/net/dial.go (L54-L61)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-22 16:34:12 +01:00
Sebastiaan van Stijn
79ea1b16b1
distribution/xfer: un-export DoFunc type
It's only used internally and by non-exported functions, so it doesn't
have to be exported.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:42 +01:00
Sebastiaan van Stijn
4d6c6a5000
distribution/xfer: smaller interface, rename to DigestRegisterer
All uses of this interface already accept a DownloadDescriptor; keeping the
interface small to allow this functionality to be used by other download-descriptors,
while still being able to check for the actual functionality (to be able to register
a digest).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:40 +01:00
Sebastiaan van Stijn
047e032461
distribution/xfer: define DownloadOption type
This is mostly for documentation purposes; defining a type makes
the option(s) show up grouped on pkg.go.dev (and in godoc).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:38 +01:00
Sebastiaan van Stijn
69b0913e1f
distribution/xfer: add TODO for return error on Registered()
There's currently four implementations in our code: one in distribution, two in
BuildKit-related code, and one "mock" for testing:

- 65b8bcc321/builder/builder-next/worker/worker.go (L452-L455)
- 65b8bcc321/builder/builder-next/adapters/containerimage/pull.go (L648-L651)
- 65b8bcc321/distribution/pull_v2.go (L318-L321)
- 65b8bcc321/distribution/xfer/download_test.go (L186-L188)

All of these call a metadata-service to register the DiffID, and all of those
may return an error, which currently gets ignored.

We should consider changing the signature to return possible errors, so that
the callsite can decide how to handle them (which could be just "log" the error
or handle them in other ways).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:36 +01:00
Sebastiaan van Stijn
3309ea9419
distribution/xfer: slightly more idomatic type assertion
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:35 +01:00
Sebastiaan van Stijn
4c7dc9806c
distribution/xfer: un-export transfer interface accessors
This interface is internal to the package, so there's no need to export
it's methods.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:33 +01:00
Sebastiaan van Stijn
849d8c2d02
distribution/xfer: un-export Watcher
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:31 +01:00
Sebastiaan van Stijn
96a750aab4
distribution/xfer: un-export Transfer interface
The only implementations and uses are internal to this package.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:29 +01:00
Sebastiaan van Stijn
765844e419
distribution/xfer: un-export NewTransfer()
This is also only used internally, so no need to export.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:28 +01:00
Sebastiaan van Stijn
587c474b57
distribution/xfer: use "transferManager" in godoc
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:26 +01:00
Sebastiaan van Stijn
874b11495b
distribution/xfer: un-export transferManager.setConcurrency() and .transfer()
They're only used within the package itself, so no need to have them public.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:24 +01:00
Sebastiaan van Stijn
d746a836fc
distribution/xfer: remove TransferManager interface, un-export newTransferManager
The `TransferManager` interface only had a single implementation, and neither
`LayerDownloadManager`, nor `LayerUploadManager` currently had an option to
provide a custom implementation, so we can un-export this.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:22 +01:00
Sebastiaan van Stijn
cf31aa0fa0
distribution/xfer: fix some nits in tests
- use const instead of a variable
- remove unneeded type-conversion
- use keys in literals

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-02-18 16:58:19 +01:00
Sebastiaan van Stijn
32e5fe5099
Merge pull request #43182 from thaJeztah/layer_remove_unused_error
layer: remove unused error return from .Size() and .DiffSize()
2022-02-17 20:51:45 +01:00
Akihiro Suda
65b8bcc321
Merge pull request #43174 from thaJeztah/move_platformcheck
distribution: remove RootFSDownloadManager interface, and remove "os" argument from Download()
2022-01-26 14:08:44 +09:00
Sebastiaan van Stijn
f9a1846ca2
distribution: PushLayer.Size(): remove unused error return
None of the implementations returned an error for this function, so removing it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-24 18:45:51 +01:00
Sebastiaan van Stijn
e1ea911aba
layer: remove unused error return from .Size() and .DiffSize()
None of the implementations used return an error, so removing the error
return can simplify using these.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-24 18:45:47 +01:00
Sebastiaan van Stijn
bb66ebd621
distribution: xfer.LayerDownloadManager.Download(): remove "os" argument
This argument was added for LCOW support, but it was only used to verify if
the passed platform (OS) matched the host. Given that all uses of this function
(except for one) passed runtime.GOOS, we may as well move the check to that
location.

We should do more cleaning up after this, and perform such validations early,
instead of passing platform around in too many places where it's only used for
similar validations. This is a first step in that direction.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-21 15:07:55 +01:00
Sebastiaan van Stijn
0b0a995d9d
distribution: remove RootFSDownloadManager interface
This interface only had a single implementation (xfer.LayerDownloadManager),
and all places where it was used already imported the xfer package.
Removing the interface, also makes it a closer match to the "upload" part,
as `xfer.LayerUploadManager()` did not use an interface.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-01-21 13:53:36 +01:00
Ziheng Liu
3dda7311cd xfer: fix a simple goroutine leak in unit test
Signed-off-by: Ziheng Liu <lzhfromustc@gmail.com>
2022-01-07 20:28:37 -08:00
Samuel Karp
7c216bcabe
distribution: validate blob type
Signed-off-by: Samuel Karp <skarp@amazon.com>
2021-11-23 17:19:01 -08:00
Sebastiaan van Stijn
1cdad98a7a
distribution: do not retry unsupported layer media types
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-10-27 12:33:30 +02:00
Sebastiaan van Stijn
8684f482e4
Merge pull request #42782 from dkkb/McGrady/code_improvement
Remove platform argument from Puller interface.
2021-08-30 13:19:08 +02:00
Eng Zer Jun
c55a4ac779
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-27 14:56:57 +08:00
Da McGrady
6f5beea146
Remove platform argument from Puller interface.
The platform argument is unneeded because ImagePullConfig is already in the v2Puller struct.

Signed-off-by: Da McGrady <dabkb@aol.com>
2021-08-27 09:34:10 +08:00
Sebastiaan van Stijn
686be57d0a
Update to Go 1.17.0, and gofmt with Go 1.17
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-24 23:33:27 +02:00
Sebastiaan van Stijn
dc70e55c9b
distribution: remove LCOW code
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 13:36:17 +02:00
Sebastiaan van Stijn
2773f81aa5
Merge pull request #42445 from thaJeztah/bump_golang_ci
[testing] ~update~ fix linting issues found by golangci-lint v1.40.1
2021-06-16 22:15:01 +02:00
Sebastiaan van Stijn
dc7cbb9b33
remove layerstore indexing by OS (used for LCOW)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 17:49:11 +02:00
Sebastiaan van Stijn
d13997b4ba
gosec: G601: Implicit memory aliasing in for loop
plugin/v2/plugin.go:141:50: G601: Implicit memory aliasing in for loop. (gosec)
                    updateSettingsEnv(&p.PluginObj.Settings.Env, &s)
                                                                 ^
    libcontainerd/remote/client.go:572:13: G601: Implicit memory aliasing in for loop. (gosec)
                cpDesc = &m
                         ^
    distribution/push_v2.go:400:34: G601: Implicit memory aliasing in for loop. (gosec)
                (metadata.CheckV2MetadataHMAC(&mountCandidate, pd.hmacKey) ||
                                              ^
    builder/dockerfile/builder.go:261:84: G601: Implicit memory aliasing in for loop. (gosec)
            currentCommandIndex = printCommand(b.Stdout, currentCommandIndex, totalCommands, &meta)
                                                                                             ^
    builder/dockerfile/builder.go:278:46: G601: Implicit memory aliasing in for loop. (gosec)
            if err := initializeStage(dispatchRequest, &stage); err != nil {
                                                       ^
    daemon/container.go:283:40: G601: Implicit memory aliasing in for loop. (gosec)
            if err := parser.ValidateMountConfig(&cfg); err != nil {
                                                 ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:29 +02:00
Tibor Vass
588da41f52
distribution: fix errors tests
Signed-off-by: Tibor Vass <tibor@docker.com>
2021-02-23 09:49:49 +01:00
Tibor Vass
7a50fe8a52
Remove more of registry v1 code.
Signed-off-by: Tibor Vass <tibor@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-23 09:49:46 +01:00
Brian Goff
c747d9f8ee
pull: Validate layer digest format
Otherwise a malformed or empty digest may cause a panic.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit a7d4af84bd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-02 13:37:24 +01:00
Brian Goff
495d623ae5 Add fallback for pull by tag
Some registries seem to be non-conformant and return a not found error
when pulling by digest (which docker now does all the time).
To work around this, fallback when all of the following are true:

1. Image reference is a tag
2. Tag->digest resolution succeeds
3. Fetch by resolved digest fails with a "not found" error.

This is intentionally not caching the manifests to reduce complexity for
this edge case.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-12-04 23:50:52 +00:00
Brian Goff
9ca3bb632e Store image manifests in containerd content store
This allows us to cache manifests and avoid extra round trips to the
registry for content we already know about.

dockerd currently does not support containerd on Windows, so this does
not store manifests on Windows, yet.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-11-05 20:02:18 +00:00
Brian Goff
e4cf1c7336 pull: use tag service for pulling tagged reference
The tag service does a `HEAD` request to get the manifest digest, where
we can then do a `GET` against the digest.

The `GET` by tag is not cacheable, but the `GET` against the digest is.
This allows proxies to work way better.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-10-30 01:57:11 +00:00
Sebastiaan van Stijn
6302dbbf46
Disable manifest v2 schema 1 push
For CI, a temporary `DOCKER_ALLOW_SCHEMA1_PUSH_DONOTUSE` environment
variable was added while we work out a solution for testing schema 1
pulls (which currently require pushing them to a local registry first
for testing).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-07-31 20:13:58 +02:00
Sebastiaan van Stijn
c88a830507
distribution/xfer: use "main" instead of "master" for progress channel
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-07-14 10:48:30 +02:00
zhipengzuo
e07a88c95f fix code note when push all tags
Signed-off-by: zhipengzuo <zuozhipeng@baidu.com>
2020-06-16 21:05:10 +08:00
Brian Goff
6859bc7eee Move schema1 deprecation notice
Currently we show this deprecation notice for any error returned by a
registry.
Registries can return an error for any number of reasons.
Instead let's show the deprecation notice only if the fallback was
successful.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2020-03-12 11:17:20 -07:00
Sebastiaan van Stijn
9f0b3f5609
bump gotest.tools v3.0.1 for compatibility with Go 1.14
full diff: https://github.com/gotestyourself/gotest.tools/compare/v2.3.0...v3.0.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-11 00:06:42 +01:00
Sebastiaan van Stijn
0cc7ad0a7d
distribution: remove use of deprecated os.SEEK_END
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-11-25 18:51:57 +01:00
Sebastiaan van Stijn
6b91ceff74
Use hcsshim osversion package for Windows versions
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-22 02:53:00 +02:00
Sebastiaan van Stijn
ec1fd4b1b0
distribution: SA4021: x = append(y) is equivalent to x = y (staticcheck)
```
distribution/push_v2_test.go:552:29: SA4021: x = append(y) is equivalent to x = y (staticcheck)
	return nil, errcode.Errors(append([]error{errcode.ErrorCodeUnauthorized.WithMessage("unauthorized")}))
	                           ^
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-10-18 00:45:31 +02:00