Finish the refactor which was partially completed with commit
34536c498d, passing around IdentityMapping structs instead of pairs of
[]IDMap slices.
Existing code which uses []IDMap relies on zero-valued fields to be
valid, empty mappings. So in order to successfully finish the
refactoring without introducing bugs, their replacement therefore also
needs to have a useful zero value which represents an empty mapping.
Change IdentityMapping to be a pass-by-value type so that there are no
nil pointers to worry about.
The functionality provided by the deprecated NewIDMappingsFromMaps
function is required by unit tests to to construct arbitrary
IdentityMapping values. And the daemon will always need to access the
mappings to pass them to the Linux kernel. Accommodate these use cases
by exporting the struct fields instead. BuildKit currently depends on
the UIDs and GIDs methods so we cannot get rid of them yet.
Signed-off-by: Cory Snider <csnider@mirantis.com>
This removes the `setOS()` / `getOS()` functions from the layer store, which were
added in fc21bf280b and 0380fbff37
in support of LCOW.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I think this was there for historic reasons (may have been goimports expected
this, and we used to have a linter that wanted it), but it's not needed, so
let's remove it (to make my IDE less complaining about unneeded aliases).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This function was abstracting things a bit too much; the layerStore had a
exported `.Get()` which called `.getWithoutLock()`, but also a non-exported
`.get()`, which also called `.getWithoutLock()`.
While it's common to have a non-exported variant (without locking), the naming
of `.get()` could easily be confused for that variant (which it wasn't).
All locations where `.get()` was called were already handling locks for
`releaseLayer()`, so moving the actual locking inline for `.get()` makes it
more visible where locking happens.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was added in commits fc21bf280b and
0380fbff37 in support of LCOW, but was
now always set to runtime.GOOS.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
None of the implementations used return an error, so removing the error
return can simplify using these.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This code was originally written for v1.10 migration where it signified that layers
were recomputed from pre 1.8 layout and could possibly change the tarball checksums.
It's now being repurposed in the BuildKit adapter but there it doesn't have any
warn condition as all data generated by builder is new anyway.
Currently, debug log entries as the one below may appear in the daemon logs:
[2019-10-02T10:00:06.690674253Z] could not get checksum for "x128nsj79yzfx4j5h6em2w2on" with tar-split: "no tar-split file"
This patch removes the debug log, as it may confuse users ("we couldn't validate
what we downloaded, but we're gonna run it anyway?")
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
opts/env_test: suppress a linter warning
this one:
> opts/env_test.go:95:4: U1000: field `err` is unused (unused)
> err error
> ^
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Format the source according to latest goimports.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
1. Reduce complexity due to nested if blocks by using early
return/continue
2. Improve logging
Changes suggested as a part of code review comments in 39748
Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>
Refactored exiting logic on way that layers are first marked to be under
removal so if actual removal fails they can be found from disk and
cleaned up.
Full garbage collector will be implemented as part of containerd
migration.
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
As pointed out by Tonis, there's a race between ReleaseRWLayer()
and GetRWLayer():
```
----- goroutine 1 ----- ----- goroutine 2 -----
ReleaseRWLayer()
m := ls.mounts[l.Name()]
...
m.deleteReference(l)
m.hasReferences()
... GetRWLayer()
... mount := ls.mounts[id]
ls.driver.Remove(m.mountID)
ls.store.RemoveMount(m.name) return mount.getReference()
delete(ls.mounts, m.Name())
----------------------- -----------------------
```
When something like this happens, GetRWLayer will return
an RWLayer without a storage. Oops.
There might be more races like this, and it seems the best
solution is to lock by layer id/name by using pkg/locker.
With this in place, name collision could not happen, so remove
the part of previous commit that protected against it in
CreateRWLayer (temporary nil assigmment and associated rollback).
So, now we have
* layerStore.mountL sync.Mutex to protect layerStore.mount map[]
(against concurrent access);
* mountedLayer's embedded `sync.Mutex` to protect its references map[];
* layerStore.layerL (which I haven't touched);
* per-id locker, to avoid name conflicts and concurrent operations
on the same rw layer.
The whole rig seems to look more readable now (mutexes use is
straightforward, no nested locks).
Reported-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This is an additon to commit 1fea38856a ("Remove v1.10 migrator")
aka PR #38265. Since that one, CreateRWLayerByGraphID() is not
used anywhere, so let's drop it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Goroutine stack analisys shown some lock contention
while doing massively (100 instances of `docker rm`)
parallel image removal, with many goroutines waiting
for the mountL mutex. Optimize it.
With this commit, the above operation is about 3x
faster, with no noticeable change to container
creation times (tested on aufs and overlay2).
kolyshkin@:
- squashed commits
- added description
- protected CreateRWLayer against name collisions by
temporary assiging nil to ls.mounts[name], and treating
nil as "non-existent" in all the other functions.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In applyTar, if the driver's ApplyDiff returns an error, the function
returns early without calling io.Copy.
As a consequence, the resources (a goroutine and some buffers holding
the uncompressed image, the digest, etc...) allocated or referenced by
NewInputTarStream above aren't released, as the worker goroutine only
finishes when it finds EOF or a closed pipe.
Signed-off-by: Sergio Lopez <slp@redhat.com>
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.
NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.
The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>
On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.
Signed-off-by: Salahuddin Khan <salah@docker.com>
Layer metadata storage has not been implemented outside of the layer
store and will be deprecated by containerd metadata storage. To prepare
for this and freeze the current metadata storage, remove the exported
interface and make it internal to the layer store.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: John Howard <jhoward@microsoft.com>
The re-coalesces the daemon stores which were split as part of the
original LCOW implementation.
This is part of the work discussed in https://github.com/moby/moby/issues/34617,
in particular see the document linked to in that issue.
This change makes the VFS graphdriver use the kernel-accelerated
(copy_file_range) mechanism of copying files, which is able to
leverage reflinks.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Signed-off-by: John Howard <jhoward@microsoft.com>
This PR has the API changes described in https://github.com/moby/moby/issues/34617.
Specifically, it adds an HTTP header "X-Requested-Platform" which is a JSON-encoded
OCI Image-spec `Platform` structure.
In addition, it renames (almost all) uses of a string variable platform (and associated)
methods/functions to os. This makes it much clearer to disambiguate with the swarm
"platform" which is really os/arch. This is a stepping stone to getting the daemon towards
fully multi-platform/arch-aware, and makes it clear when "operating system" is being
referred to rather than "platform" which is misleadingly used - sometimes in the swarm
meaning, but more often as just the operating system.
This enables docker cp and ADD/COPY docker build support for LCOW.
Originally, the graphdriver.Get() interface returned a local path
to the container root filesystem. This does not work for LCOW, so
the Get() method now returns an interface that LCOW implements to
support copying to and from the container.
Signed-off-by: Akash Gupta <akagup@microsoft.com>
