The --allow-nondistributable-artifacts daemon option specifies
registries to which foreign layers should be pushed. (By default,
foreign layers are not pushed to registries.)
Additionally, to make this option effective, foreign layers are now
pulled from the registry if possible, falling back to the URLs in the
image manifest otherwise.
This option is useful when pushing images containing foreign layers to a
registry on an air-gapped network so hosts on that network can pull the
images without connecting to another server.
Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
This fix is based on:
https://github.com/docker/docker/issues/29936#issuecomment-277494885
Currently the insecure registry is only checked to see if it contains
scheme (`http(s)://`) or not. No fully validation is done and this
caused many confusions like in #29936.
This fix tries to address the issue.
This fix adds additional validation so that an insecure registry
is validated to make sure it is in `host:port` format where host
could be IPv4/IPv6 or a host name, and port could be an integer
between 0-65535.
Additional unit tests have been added.
This fix is related to #29936.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Remove forked reference package. Use normalized named values
everywhere and familiar functions to convert back to familiar
strings for UX and storage compatibility.
Enforce that the source repository in the distribution metadata
is always a normalized string, ignore invalid values which are not.
Update distribution tests to use normalized values.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Use resolving to repo info as the split point between the
legitimate reference package and forked reference package.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
While investigating 29936 I noticed one potential issue
in `LoadInsecureRegistries`.
The implementation of the func assumes that the format
of insecure registry should be `host:port` if not CIDR.
However, it is very common that user may incorrectly
provide a registry with a scheme (e.g, `http://myregistry.com:5000`)
Such a registry format with a scheme will cause docker pull to
always try https endpoint.
The reason is that the func of `isSecureIndex()` actually will
check for the map of the index server for `myregistry.com:5000`
while the insecure registry only has a record of `http://myregistry.com:5000`.
As a consequence, docker assumes that `myregistry.com:5000` is not
a insecure registry and will go ahead with https endpoint.
This fix addresses the issue by error out insecure registries with scheme.
A unit test has been added.
This fix is related to 29936.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This patch fixed below 4 types of code line
1. Remove unnecessary variable assignment
2. Use variables declaration instead of explicit initial zero value
3. Change variable name to underbar when variable not used
4. Add erro check and return for ignored error
Signed-off-by: Daehyeok Mun <daehyeok@gmail.com>
This removes some very old vestigial code that really should have been
removed during the content addressability transition. It implements
something called "reference" but it behaves differently from the actual
reference package. This was only used by client-side content trust code,
and is relatively easy to extricate.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
All images in the default registry (AKA docker.io, index.docker.io, and
registry-1.docker.io) are available via the v2 protocol, so there's no
reason to use the v1 protocol. Disabling it prevents useless fallbacks.
Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
Also consolidate the leftover packages under cli.
Remove pkg/mflag.
Make manpage generation work with new cobra layout.
Remove remaining mflag and fix tests after rebase with master.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This fixes the hard coded restriction for non-linux platforms to v2 registries. Previously, the check was above the flag parsing, which would overwrite the hard coded value and prevent correct operation. This change also removes the related daemon flag from Windows to avoid confusion, as it has no meaning when the value is going to always be hard coded to true.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
This adds the `--live-restore` option to the documentation.
Also synched usage description in the documentation
with the actual description, and re-phrased some
flag descriptions to be a bit more consistent.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Updates the v1 search endpoint to also support v2 auth when an identity token is given.
Only search v1 endpoint is supported since there is not v2 search currently defined to replace it.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
This fix tries to fix logrus formatting by removing `f` from
`logrus.[Error|Warn|Debug|Fatal|Panic|Info]f` when formatting string
is not present.
This fix fixes#23459.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in #23055.
Currently `docker search` result caps at 25 and there is
no way to allow getting more results (if exist).
This fix adds the flag `--limit` so that it is possible
to return more results from the `docker search`.
Related documentation has been updated.
Additional tests have been added to cover the changes.
This fix fixes#23055.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
… and refactor a little bit some daemon on the way.
- Move `SearchRegistryForImages` to a new file (`daemon/search.go`) as
`daemon.go` is getting pretty big.
- `registry.Service` is now an interface (allowing us to decouple it a
little bit and thus unit test easily).
- Add some unit test for `SearchRegistryForImages`.
- Use UniqueExactMatch for search filters
- And use empty restore id for now in client.ContainerStart.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Use sockets.DialerFromEnvironment, as is done in other places,
to transparently support SOCKS proxy config from ALL_PROXY
environment variable.
Requires the *engine* have the ALL_PROXY env var set, which
doesn't seem ideal. Maybe it should be a CLI option somehow?
Only tested with push and a v2 registry so far. I'm happy to look
further into testing more broadly, but I wanted to get feedback on
the general idea first.
Signed-off-by: Brett Higgins <brhiggins@arbor.net>
This fix tries to address the issue in #22244 where the remote
API `/auth` will not set the default value of `serveraddress`
if not provided. This behavior happens after only in 1.11.0
and is a regression as in 1.10.3 `serveraddress` will be assigned
with `IndexServer` if no value is provided.
The default value `IndexServer` is assigned to `serveraddress` if
no value provided in this fix.
An integration test `TestAuthApi` has been added to cover this change
This fix fixes#22244.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Going forward, Docker won't use a different default registry on Windows.
This changes Windows to use the standard Docker Hub registry as the
default registry.
There is a plan in place to migrate existing images from the Windows
registry to Hub's normal registry, in advance of the 1.11 release. In
the mean time, images on the Windows registry can be accessed by
prefixing them with `registry-win-tp3.docker.io/`.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Use token handler options for initialization.
Update auth endpoint to set identity token in response.
Update credential store to match distribution interface changes.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Further differentiate the APIEndpoint used with V2 with the endpoint type which is only used for v1 registry interactions
Rename Endpoint to V1Endpoint and remove version ambiguity
Use distribution token handler for login
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This removes the email prompt when you use docker login, and also removes the ability to register via the docker cli. Docker login, will strictly be used for logging into a registry server.
Signed-off-by: Ken Cochrane <kencochrane@gmail.com>
This allows easier URL handling in code that uses APIEndpoint.
If we continued to store the URL unparsed, it would require redundant
parsing whenver we want to extract information from it. Also, parsing
the URL earlier should give improve validation.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Several improvements to error handling:
- Introduce ImageConfigPullError type, wrapping errors related to
downloading the image configuration blob in schema2. This allows for a
more descriptive error message to be seen by the end user.
- Change some logrus.Debugf calls that display errors to logrus.Errorf.
Add log lines in the push/pull fallback cases to make sure the errors
leading to the fallback are shown.
- Move error-related types and functions which are only used by the
distribution package out of the registry package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
The daemon uses two similar filename extensions to identify different
kinds of certificates. ".crt" files are interpreted as CA certificates,
and ".cert" files are interprted as client certificates. If a CA
certificate is accidentally given the extension ".cert", it will lead to
the following error message:
Missing key ca.key for certificate ca.cert
To make this slightly less confusing, clarify the error message with a
note that CA certificates should use the extension ".crt".
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
- Use the ones provided by docker/go-connections, they are a drop in replacement.
- Remove pkg/sockets from docker.
- Keep pkg/tlsconfig because libnetwork still needs it and there is a
circular dependency issue.
Signed-off-by: David Calavera <david.calavera@gmail.com>
This is a followup to #18839. That PR relaxed the fallback logic so that
if a manifest doesn't exist on v2, or the user is unauthorized to access
it, we try again with the v1 protocol. A similar special case is needed
for "pull all tags" (docker pull -a). If the v2 registry doesn't
recognize the repository, or doesn't allow the user to access it, we
should fall back to v1 and try to pull all tags from the v1 registry.
Conversely, if the v2 registry does allow us to list the tags, there
should be no fallback, even if there are errors pulling those tags.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
PR #18590 caused compatibility issues with registries such as gcr.io
which support both the v1 and v2 protocols, but do not provide the same
set of images over both protocols. After #18590, pulls from these
registries would never use the v1 protocol, because of the
Docker-Distribution-Api-Version header indicating that v2 was supported.
Fix the problem by making an exception for the case where a manifest is
not found. This should allow fallback to v1 in case that image is
exposed over the v1 protocol but not the v2 protocol.
This avoids the overly aggressive fallback behavior before #18590 which
would allow protocol fallback after almost any error, but restores
interoperability with mixed v1/v2 registry setups.
Fixes#18832
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
If we detect a Docker-Distribution-Api-Version header indicating that
the registry speaks the V2 protocol, no fallback to V1 should take
place.
The same applies if a V2 registry operation succeeds while attempting a
push or pull.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
registry.ResolveAuthConfig() only needs the AuthConfigs from the ConfigFile, so
this change passed just the AuthConfigs.
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
This commit adds a transfer manager which deduplicates and schedules
transfers, and also an upload manager and download manager that build on
top of the transfer manager to provide high-level interfaces for uploads
and downloads. The push and pull code is modified to use these building
blocks.
Some benefits of the changes:
- Simplification of push/pull code
- Pushes can upload layers concurrently
- Failed downloads and uploads are retried after backoff delays
- Cancellation is supported, but individual transfers will only be
cancelled if all pushes or pulls using them are cancelled.
- The distribution code is decoupled from Docker Engine packages and API
conventions (i.e. streamformatter), which will make it easier to split
out.
This commit also includes unit tests for the new distribution/xfer
package. The tests cover 87.8% of the statements in the package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
ContinueOnError assumes that something of type errcode.Errors contains
at least one error. This is generally true, but might not be true if the
remote registry returns an empty error body or invalid JSON. Add the
bounds check, and in the case where it fails, allow fallbacks to v1.
Fixes#18481
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Add distribution package for managing pulls and pushes. This is based on
the old code in the graph package, with major changes to work with the
new image/layer model.
Add v1 migration code.
Update registry, api/*, and daemon packages to use the reference
package's types where applicable.
Update daemon package to use image/layer/tag stores instead of the graph
package
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This fix avoids overwritting the previous error messages, ensures the client gets the correct error messages and not just the most recent message during the pull request.
For this `var lastErr` replaced with a slice which acts as a temp place holder for the list of returned error messages for every attempt.
The slice is later joined and returned to the caller function after searching for the image with diffirent versions(v2,v1,v0).
Updated the code with check for no space left on device error occurance and prevent the
daemon on falling back to v1,v0.
Incorporated the comments from @calavera, @RichardScothern, @cpuguy83
Signed-off-by: Anil Belur <askb23@gmail.com>
This updates the vendored docker/distribution to the current master
branch.
Note the following changes:
- The manifest package was split into manifest/schema1. Most references
to the manifest package in the engine needed to be updated to use
schema1 instead.
- Validation functions in api/v2 were replaced by the
distribution/reference package. The engine code has been updated to
use the reference package for validation where necessary. A future PR
will change the engine to use the types defined in
distribution/reference more comprehensively.
- The reference package explicitly allows double _ characters in
repository names. registry_test.go was updated for this.
- TestPullFailsWithAlteredManifest was corrupting the manifest JSON, now
that the schema1 package unmarshals the correct payload. The test is
being changed to modify the JSON without affecting its length, which
allows the pull to succeed to the point where digest validation
happens.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This reverts commit d5cd032a86.
Commit caused issues on systems with case-insensitive filesystems.
Revert for now
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
- Move autogen/dockerversion to version
- Update autogen and "builds" to use this package and a build flag
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Pick up name regexp change in distribution to allow matching of hostnames as a valid component of a repository.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Search terms shouldn't be restricted to only full valid repository
names. It should be perfectly valid to search using a part of a name,
even if it ends with a period, dash or underscore.
Signed-off-by: Hu Keping <hukeping@huawei.com>
Add a daemon flag to control this behaviour. Add a warning message when pulling
an image from a v1 registry. The default order of pull is slightly altered
with this changset.
Previously it was:
https v2, https v1, http v2, http v1
now it is:
https v2, http v2, https v1, http v1
Prevent login to v1 registries by explicitly setting the version before ping to
prevent fallback to v1.
Add unit tests for v2 only mode. Create a mock server that can register
handlers for various endpoints. Assert no v1 endpoints are hit with legacy
registries disabled for the following commands: pull, push, build, run and
login. Assert the opposite when legacy registries are not disabled.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
This patch splits LookupEndpoints into LookupPullEndpoints and
LookupPushEndpoints so that mirrors added with --registry-mirror are
skipped in the list returned by LookupPushEndpoints.
Fixes https://github.com/docker/distribution/issues/823
Signed-off-by: Tibor Vass <tibor@docker.com>
Some structures use int for sizes and UNIX timestamps. On some
platforms, int is 32 bits, so this can lead to the year 2038 issues and
overflows when dealing with large containers or layers.
Consistently use int64 to store sizes and UNIX timestamps in
api/types/types.go. Update related to code accordingly (i.e.
strconv.FormatInt instead of strconv.Itoa).
Use int64 in progressreader package to avoid integer overflow when
dealing with large quantities. Update related code accordingly.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This patch fixes a bug where a user specifies a v1 mirror for
--registry-mirror and pull an image from the Hub.
It used to not fallback because of an unexpected error returned when
trying to JSON marshal nginx output.
We now ensure that any unexpected error falls back to the next endpoint
in the list.
Signed-off-by: Tibor Vass <tibor@docker.com>
Currently login and search do not load per registry certificates.
This is a regression caused by the last refactor since this was recently fixed.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Add a trusted flag to force the cli to resolve a tag into a digest via the notary trust library and pull by digest.
On push the flag the trust flag will indicate the digest and size of a manifest should be signed and push to a notary server.
If a tag is given, the cli will resolve the tag into a digest and pull by digest.
After pulling, if a tag is given the cli makes a request to tag the image.
Use certificate directory for notary requests
Read certificates using same logic used by daemon for registry requests.
Catch JSON syntax errors from Notary client
When an uncaught error occurs in Notary it may show up in Docker as a JSON syntax error, causing a confusing error message to the user.
Provide a generic error when a JSON syntax error occurs.
Catch expiration errors and wrap in additional context.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
* Add godoc documentation where it was missing
* Change identifier names that don't match Go style, such as INDEX_NAME
* Rename RegistryInfo to PingResult, which more accurately describes
what this structure is for. It also has the benefit of making the name
not stutter if used outside the package.
Updates #14756
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This patch creates a new cli package that allows to combine both client
and daemon commands (there is only one daemon command: docker daemon).
The `-d` and `--daemon` top-level flags are deprecated and a special
message is added to prompt the user to use `docker daemon`.
Providing top-level daemon-specific flags for client commands result
in an error message prompting the user to use `docker daemon`.
This patch does not break any old but correct usages.
This also makes `-d` and `--daemon` flags, as well as the `daemon`
command illegal in client-only binaries.
Signed-off-by: Tibor Vass <tibor@docker.com>
If a registry mirror is using TLS, ensure that certs for it
are picked up from /etc/docker/certs.d
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
The only uses of RequestAuthorization and its associated functions were
removed in 19515a7ad8 ("Update graph to
use vendored distribution client for the v2 codepath")
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
- fully capitalize HTTP in HTTPHeaders
- comment for CONFIGFILE
- camelcase and privatize oldConfigfile, defaultIndexserver
- remove unused var errConfigFileMissing
- comments for methods and functions throughout
- external references to renamed variables changed
Signed-off-by: Morgan Bauer <mbauer@us.ibm.com>
Currently canonical name gets set to the local name and displayed in the errors.
Canonical name should be the unique and canonical name for an image.
Use docker.io as the canonical domain for images on the public registry.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The docker/distribution dependency was updated in the previous commit to allow
repository name components to only consist of a single letter. The unit tests
have been updated to cement this change.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
By adding this header AuthTransport will add Basic authentication to the request and allow 'docker search' results to include private images.
Signed-off-by: Matt Moore <mattmoor@google.com>
The registry client's TLS configuration used the
default cipher list, including RC4. This change
copies the default cipher list from Golang 1.4 and
removes RC4 from that list. RC4 ciphers are considered
weak and vulnerable to a number of attacks.
Uses the tlsconfig package to define allowed ciphers.
Signed-off-by: Eric Windisch <eric@windisch.us>
Today, endpoints implementing v2 cannot properly fallback to v1 because the underlying transport that deals with authentication (Basic / Token) doesn't get annotated.
This doesn't affect DockerHub because the DockerHub endpoint appears as 'https://index.docker.io/v1/' (in .dockercfg), and the 'v1' tricks this logic just long enough that the transport is always annotated for DockerHub accesses.
Signed-off-by: Matt Moore <mattmoor@google.com>
This patch ensures no auth headers are set for v1 registries if there
was a 302 redirect.
This also ensures v2 does not use authTransport.
Signed-off-by: Tibor Vass <tibor@docker.com>
Refactoring in Docker 1.7 changed the behavior to add this header where as Docker <= 1.6 wouldn't emit this Header on a HTTP 302 redirect.
This closes#13649
Signed-off-by: Jeffrey van Gogh <jvg@google.com>
It should not print to STDOUT so that it only prints the debugTransport
output if there was an error in one of the registry tests.
Signed-off-by: Tibor Vass <tibor@docker.com>
