Commit graph

99 commits

Author SHA1 Message Date
Aaron Lehmann
c5769cf53b Pull: only close temporary file once
Close could be called twice on a temporary download file, which could
have bad side effects.

This fixes the problem by setting to ld.tmpFile to nil when the download
completes sucessfully. Then the call to ld.Close will have no effect,
and only the download manager will close the temporary file when it's
done extracting the layer from it. ld.Close will be responsible for
closing the file if we hit the retry limit and there is still a partial
download present.

Fixes #21675

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
(cherry picked from commit 930ae3dbcb)
2016-03-31 18:35:38 -04:00
Anusha Ragunathan
1818ca9d75 Update mount state of live containers after a daemon crash.
Fix unmount issues in the daemon crash and restart lifecycle, w.r.t
graph drivers. This change sets a live container RWLayer's activity
count to 1, so that the RWLayer is aware of the mount. Note that
containerd has experimental support for restore live containers.

Added/updated corresponding tests.

Signed-off-by: Anusha Ragunathan <anusha@docker.com>
(cherry picked from commit 511a70583f)
2016-03-31 18:35:37 -04:00
Aaron Lehmann
c44e7a3e63 Pass upstream client's user agent through to registry on operations beyond pulls
This adds support for the passthrough on build, push, login, and search.

Revamp the integration test to cover these cases and make it more
robust.

Use backticks instead of quoted strings for backslash-heavy string
contstands.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-21 14:31:47 -07:00
Arnaud Porterie
278d3962a8 Merge pull request #21306 from mgoelzer/issue20001-nodeps
Pass upstream client's user agent through to registry on image pulls
2016-03-21 14:26:59 -07:00
Alexander Morozov
a05fdd6806 Merge pull request #21283 from runcom/update-test
vendor docker/distribution d06d6d3b093302c02a93153ac7b06ebc0ffd1793
2016-03-21 09:46:28 -07:00
Mike Goelzer
d1502afb63 Pass upstream client's user agent through to registry on image pulls
Changes how the Engine interacts with Registry servers on image pull.
Previously, Engine sent a User-Agent string to the Registry server
that included only the Engine's version information.  This commit
appends to that string the fields from the User-Agent sent by the
client (e.g., Compose) of the Engine.  This allows Registry server
operators to understand what tools are actually generating pulls on
their registries.

Signed-off-by: Mike Goelzer <mgoelzer@docker.com>
2016-03-20 04:30:09 -07:00
Antonio Murdaca
264b5b6083 distribution: errors: do not retry if no token in response
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-19 12:20:14 +01:00
Tonis Tiigi
9c4570a958 Replace execdrivers with containerd implementation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
2016-03-18 13:38:32 -07:00
Antonio Murdaca
1b5c2e1d72 distribution: errors: do not access the errors slice if it's empty
- cherry-pick from 1.10.3 branch: 0186f4d422
- add token service test suite
- add integration test (missing in 1.10.3 branch)

Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-16 09:00:39 +01:00
Phil Estes
2b1b235acf Merge pull request #21222 from aaronlehmann/pull-with-no-layers
Fix pulling images that contain no layers at all
2016-03-15 23:03:42 -07:00
Aaron Lehmann
2f4aa96584 Fix flaky test TestTransfer
This test was checking that it received every progress update that was
produced. But delivery of these intermediate progress updates is not
guaranteed. A new update can overwrite the previous one if the previous
one hasn't been sent to the channel yet.

The call to t.Fatalf exited the current goroutine which was consuming
the channel, which caused a deadlock and eventual test timeout rather
than a proper failure message.

Failure seen here:
https://jenkins.dockerproject.org/job/Docker-PRs-experimental/16400/console

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-15 16:26:16 -07:00
Aaron Lehmann
7cf894ce10 Fix pulling images that contain no layers at all
The download manager assumed there was at least one layer involved in
all images. This can be false if the image is essentially a copy of
`scratch`.

Fix a nil pointer dereference that happened in this case. Add
integration tests that involve schema1 and schema2 manifests.

Fixes #21213

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-15 11:10:03 -07:00
Vincent Demeester
90ce409e35 Merge pull request #20949 from imjching/master
Fix typo for download and upload retry messages
2016-03-14 21:08:49 +01:00
Jay
f8d14bd4c6 Fix typo for download and upload retry messages
Signed-off-by: Jay <jay@imjching.com>
2016-03-15 00:56:19 +08:00
Aaron Lehmann
0b962f72a9 Merge pull request #21159 from runcom/fix-retry-push-bug
distribution: errors: do not retry if no credentials provided
2016-03-14 09:47:13 -07:00
Antonio Murdaca
497d545093 distribution: errors: do not retry if no credentials provided
Fix and add test for case c) in #21054

Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-12 20:41:47 +01:00
Derek McGowan
e896d1d7c4 Add support for identity token with token handler
Use token handler options for initialization.
Update auth endpoint to set identity token in response.
Update credential store to match distribution interface changes.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-09 13:47:57 -08:00
Sebastiaan van Stijn
17156ba98f Merge pull request #20832 from aaronlehmann/login-endpoint-refactor
Update login to use token handling code from distribution
2016-03-03 19:04:42 +01:00
Aaron Lehmann
613d364e23 Merge pull request #20868 from LK4D4/unused_ctx
Remove unused ctx from v1Pusher
2016-03-02 10:05:46 -08:00
Alexander Morozov
a4dbbe7d89 Remove unused ctx from v1Pusher
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2016-03-02 07:59:12 -08:00
Derek McGowan
f2d481a299 Login update and endpoint refactor
Further differentiate the APIEndpoint used with V2 with the endpoint type which is only used for v1 registry interactions
Rename Endpoint to V1Endpoint and remove version ambiguity
Use distribution token handler for login

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-01 11:52:32 -08:00
Aaron Lehmann
5c99eebe81 Fix concurrent uploads that share layers
Concurrent uploads which share layers worked correctly as of #18353,
but unfortunately #18785 caused a regression. This PR removed the logic
that shares digests between different push sessions. This overlooked the
case where one session was waiting for another session to upload a
layer.

This commit adds back the ability to propagate this digest information,
using the distribution.Descriptor type because this is what is received
from stats and uploads, and also what is ultimately needed for building
the manifest.

Surprisingly, there was no test covering this case. This commit adds
one. It fails without the fix.

See recent comments on #9132.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-01 11:14:44 -08:00
Tibor Vass
8ffec1fca5 Merge pull request #20772 from Microsoft/jjh/testunitdistribution-xfer
Windows CI Unit Test: Distribution\xfer turn off failing tests
2016-02-29 10:45:57 -05:00
Sebastiaan van Stijn
da4f2ce629 Merge pull request #20771 from Microsoft/jjh/disableunittestsdistribution
Windows CI Unit Test: Distribution turn off failing tests
2016-02-29 12:56:10 +01:00
John Howard
dd2ff281bf Windows CI Unit Test: Distribution\xfer turn off failing tests
Signed-off-by: John Howard <jhoward@microsoft.com>
2016-02-28 20:08:34 -08:00
John Howard
621a1b9aca Windows CI Unit Test: Distribution turn off failing tests
Signed-off-by: John Howard <jhoward@microsoft.com>
2016-02-28 20:02:51 -08:00
Brian Bland
1d3480f9ba Improve fallback behavior for cross-repository push
Attempt layer mounts from up to 3 source repositories, possibly
falling back to a standard blob upload for cross repository pushes.
Addresses compatiblity issues with token servers which do not grant
multiple repository scopes, resulting in an authentication failure for
layer mounts, which would otherwise cause the push to terminate with an
error.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-24 19:13:35 -08:00
Aaron Lehmann
79db131a35 Change APIEndpoint to contain the URL in a parsed format
This allows easier URL handling in code that uses APIEndpoint.
If we continued to store the URL unparsed, it would require redundant
parsing whenver we want to extract information from it. Also, parsing
the URL earlier should give improve validation.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-17 17:48:15 -08:00
Aaron Lehmann
5e8af46fda Smarter push/pull TLS fallback
With the --insecure-registry daemon option (or talking to a registry on
a local IP), the daemon will first try TLS, and then try plaintext if
something goes wrong with the push or pull. It doesn't make sense to try
plaintext if a HTTP request went through while using TLS. This commit
changes the logic to keep track of host/port combinations where a TLS
attempt managed to do at least one HTTP request (whether the response
code indicated success or not). If the host/port responded to a HTTP
using TLS, we won't try to make plaintext HTTP requests to it.

This will result in better error messages, which sometimes ended up
showing the result of the plaintext attempt, like this:

    Error response from daemon: Get
    http://myregistrydomain.com:5000/v2/: malformed HTTP response
    "\x15\x03\x01\x00\x02\x02"

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-12 13:47:48 -08:00
Vincent Demeester
e17afedfe0 Merge pull request #20247 from aaronlehmann/distribution-errors
Push/pull errors improvement and cleanup
2016-02-12 11:57:25 +01:00
David Calavera
5ca6d3bafd Merge pull request #20250 from vieux/fix_misspell
fix common misspells
2016-02-11 17:22:06 -08:00
Aaron Lehmann
8f26fe4f59 Push/pull errors improvement and cleanup
Several improvements to error handling:

- Introduce ImageConfigPullError type, wrapping errors related to
  downloading the image configuration blob in schema2. This allows for a
  more descriptive error message to be seen by the end user.

- Change some logrus.Debugf calls that display errors to logrus.Errorf.
  Add log lines in the push/pull fallback cases to make sure the errors
  leading to the fallback are shown.

- Move error-related types and functions which are only used by the
  distribution package out of the registry package.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-11 16:28:56 -08:00
Victor Vieux
99a396902f fix common misspell
Signed-off-by: Victor Vieux <vieux@docker.com>
2016-02-11 15:49:36 -08:00
Aaron Lehmann
7b81bc147c Fall back to V1 when there are no basic auth credentials
This makes the behavior consistent with having incorrect credentials.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-11 10:48:11 -08:00
Aaron Lehmann
056bf9f25e Attempt to resume downloads after certain errors
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-09 14:12:28 -08:00
Aaron Lehmann
f425529e7e Move temporary download file to download descriptor scope
This will allow it to be reused between download attempts in a
subsequent commit.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-09 14:12:28 -08:00
Alexander Morozov
9f0095f14a Merge pull request #19887 from aaronlehmann/vendor-resumable-downloads
Vendor updated distribution for resumable downloads
2016-02-05 12:58:53 -08:00
Aaron Lehmann
4d437a29d2 Vendor updated distribution for resumable downloads
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-04 17:33:37 -08:00
Tibor Vass
66a4e557f9 Merge pull request #19702 from tiborvass/host-specific-passthru-token
Add test to make sure the new registry pass-thru token is only sent to the intended hosts
2016-02-03 21:57:32 -05:00
Tibor Vass
3fa0d09e74 Merge pull request #19971 from aaronlehmann/revert-19743
Revert "Set idle timeouts for HTTP reads and writes in communications with the registry"
2016-02-03 15:33:29 -05:00
Alexander Morozov
bce70cdc2c Merge pull request #18680 from aaronlehmann/duplicate-pull-complete-message
Avoid outputting last progress item twice
2016-02-03 11:01:45 -08:00
Aaron Lehmann
cbda80aaff Revert "Set idle timeouts for HTTP reads and writes in communications with the registry"
This reverts commit 84b2162c1a.

The intent of this commit was to set an idle timeout on a HTTP
connection. If a read took more than 60 seconds to complete, or a write
took more than 60 seconds to complete, the connection would be
considered dead.

This doesn't work properly, because the HTTP internals apparently read
from the connection concurrently while writing. An upload that doesn't
complete in 60 seconds leads to a timeout.

Fixes #19967

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-03 09:59:56 -08:00
Aaron Lehmann
e273445dd4 Fix panic on network timeout during push
`Upload` already closes the reader returned by `compress` and the
progressreader passed into it, before returning. But even so, the
io.Copy inside compress' goroutine needs to attempt a read from the
progressreader to notice that it's closed, and this read has a side
effect of outputting a progress message. If this happens after `Upload`
returns, it can result in a write to a closed channel. Change `compress`
to return a channel that allows the caller to wait for its goroutine to
finish before freeing any resources connected to the reader that was
passed to it.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-29 14:34:50 -08:00
Tibor Vass
f3df50d115 distribution: add test to ensure pass-thru registry token is host-bound
Signed-off-by: Tibor Vass <tibor@docker.com>
2016-01-28 16:23:06 -08:00
Aaron Lehmann
84b2162c1a Set idle timeouts for HTTP reads and writes in communications with the registry
Otherwise, some operations can get stuck indefinitely when the remote
side is unresponsive.

Fixes #12823

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-26 12:40:39 -08:00
Aaron Lehmann
fde2329eaa Avoid outputting last progress item twice
A watcher would output the current progress item when it was detached,
in case it missed that item earlier, which would leave the user seeing
some intermediate step of the operation. This commit changes it to only
output it on detach if it didn't already output the same item.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-26 10:03:28 -08:00
Vincent Demeester
4158b671f3 Merge pull request #19701 from aaronlehmann/remove-downloads-on-error
Remove temporary layer download file on error
2016-01-26 10:55:43 +01:00
Tibor Vass
07e2dedecb Merge pull request #19057 from dnephin/remove_version_from_registry_pkg
Remove dockerversion from registry package
2016-01-25 22:45:06 -08:00
Aaron Lehmann
5a363ce60b Remove temporary layer download file on error
Currently, the temporary file storing downloaded layer data is only
removed after a successful download or a digest verification error. A
transport-level error does not cause it to be removed. This is a
regression from 1.9 that could cause disk usage to grow until the Docker
daemon is restarted.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-25 18:20:18 -08:00
Aaron Lehmann
3e2b50ccaa Fix watching a released transfer
Things could go wrong if Watch was called after the last existing
watcher was released. The call to Watch would succeed even though it was
not really adding a watcher, and the corresponding call to Release would
close hasWatchers a second time.

The fix for this is twofold:

1. We allow transfers to gain new watchers after the watcher count has
touched zero. This means that the channel returned by Released should
not be closed until all watchers have been released AND the transfer is
no longer tracked by the transfer manager, meaning it won't be possible
for additional calls to Watch to race with closing the channel returned
by Released.

The Transfer interface has a new method called Close so the transfer can
know when the transfer manager no longer references it.

Remove the Cancel method. It's not used and should not be exported.

2. Even though (1) makes it possible to add watchers after all the
previous watchers have been released, we want to avoid doing this in
practice. A transfer that has had all its watchers released is in the
process of being cancelled, and attaching to one of these will never be
the correct behavior. Add a check if a watcher is attaching to a
cancelled transfer.  In this case, wait for the transfer to be removed
from the map and try again. This will ensure correct behavior when a
watcher tries to attach during the race window.

Either (1) or (2) should be sufficient to fix the race involved here,
but the combination is the most correct approach. (1) fixes the
low-level plumbing to be resilient to the race condition, and (2) avoids
using it in a racy way.

Fixes #19606

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-25 13:28:02 -08:00