Fixes case where shutdown occurs before content is synced to disked
on layer creation. This case can leave the layer store in an bad
state and require manual recovery. This change ensures all files
are synced to disk before a layer is committed. Any shutdown that
occurs will only cause the layer to not show up but will allow it to
be repulled or recreated without error.
Added generic io logic to ioutils package to abstract it out of
the layer store package.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Perform chmod before rename with the atomic file writer.
Ensure writeErr is set on short write and file is removed on write error.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Creates a `fixedBuffer` type that is used to encapsulate functionality
for reading/writing from the underlying byte slices.
Uses lazily-loaded set of sync.Pools for storing buffers that are no
longer needed so they can be re-used.
```
benchmark old ns/op new ns/op delta
BenchmarkBytesPipeWrite-8 138469 48985 -64.62%
BenchmarkBytesPipeRead-8 130922 56601 -56.77%
benchmark old allocs new allocs delta
BenchmarkBytesPipeWrite-8 18 8 -55.56%
BenchmarkBytesPipeRead-8 0 0 +0.00%
benchmark old bytes new bytes delta
BenchmarkBytesPipeWrite-8 66903 1649 -97.54%
BenchmarkBytesPipeRead-8 0 1 +Inf%
```
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This commit adds a transfer manager which deduplicates and schedules
transfers, and also an upload manager and download manager that build on
top of the transfer manager to provide high-level interfaces for uploads
and downloads. The push and pull code is modified to use these building
blocks.
Some benefits of the changes:
- Simplification of push/pull code
- Pushes can upload layers concurrently
- Failed downloads and uploads are retried after backoff delays
- Cancellation is supported, but individual transfers will only be
cancelled if all pushes or pulls using them are cancelled.
- The distribution code is decoupled from Docker Engine packages and API
conventions (i.e. streamformatter), which will make it easier to split
out.
This commit also includes unit tests for the new distribution/xfer
package. The tests cover 87.8% of the statements in the package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Turn BytesPipe's Read and Write functions into blocking, goroutine-safe
functions. Add a CloseWithError function to propagate an error code to
the Read function.
Adjust tests to work with the blocking Read and Write functions.
Remove BufReader, since now its users can use BytesPipe directly.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
By adding a (*WriteFlusher).Close, we limit the Write calls to possibly
deallocated http response buffers to the lifetime of an http request.
Typically, this is seen as a very confusing panic, the cause is usually a
situation where an http.ResponseWriter is held after request completion. We
avoid the panic by disallowing further writes to the response writer after the
request is completed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This patch creates interfaces in builder/ for building Docker images.
It is a first step in a series of patches to remove the daemon
dependency on builder and later allow a client-side Dockerfile builder
as well as potential builder plugins.
It is needed because we cannot remove the /build API endpoint, so we
need to keep the server-side Dockerfile builder, but we also want to
reuse the same Dockerfile parser and evaluator for both server-side and
client-side.
builder/dockerfile/ and api/server/builder.go contain implementations
of those interfaces as a refactoring of the current code.
Signed-off-by: Tibor Vass <tibor@docker.com>
