In some cases, if a user specifies `-f` when disabling a plugin mounts
can still exist on the plugin rootfs.
This can cause problems during upgrade where the rootfs is removed and
may cause data loss.
To resolve this, ensure the rootfs is unmounted
before performing an upgrade.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
While restoring plugins during daemon restart, some plugins can fail to
respond to net.Dial. These plugins should be explicitly set to disabled,
else they will retain their original state of enabled, which is
incorrect.
Tested with a plugin that fails to restart and observed that the state
was set to disabled.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
When the daemon is configured to run with an authorization-plugin and if
the plugin is disabled, the daemon continues to send API requests to the
plugin and expect it to respond. But the plugin has been disabled. As a
result, all API requests are blocked. Fix this behavior by removing the
disabled plugin from the authz middleware chain.
Tested using riyaz/authz-no-volume-plugin and observed that after
disabling the plugin, API request/response is functional.
Fixes#31836
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
Embedding DockerVersion in plugin config when the plugin is created,
enables users to do a docker plugin inspect and know which version
the plugin was built on. This is helpful in cases where users are
running a new plugin on older docker releases and confused at
unexpected behavior.
By embedding DockerVersion in the config, we claim that there's no
guarantee that if the plugin config's DockerVersion is greater that
the version of the docker engine the plugin is executed against, the
plugin will work as expected.
For example, lets say:
- in 17.03, a plugin was released as johndoe/foo:v1
- in 17.05, the plugin uses the new ipchost config setting and author
publishes johndoe/foo:v2
In this case, johndoe/foo:v2 was built on 17.05 using ipchost, but is
running on docker-engine version 17.03. Since 17.05 > 17.03, there's
no guarantee that the plugin will work as expected. Ofcourse, if the
plugin did not use newly added config settings (ipchost in this case)
in 17.05, it would work fine in 17.03.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
Tested using global-net-plugin-ipc which sets PidHost in config.json.
Plugins might need access to host pid namespace. Add support for that.
Tested using aragunathan/global-net-plugin-ipc which sets "pidhost" in
config.json. Observed using `readlink /proc/self/ns/pid` that plugin and
host have the same ns.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
Plugins might need access to host ipc namespace. A good usecase is
a volume plugin running iscsi multipath commands that need access to
host kernel locks.
Tested with a custom plugin (aragunathan/global-net-plugin-full) that's
built with `"ipchost" : true` in config.json. Observed using
`readlink /proc/self/ns/ipc` that plugin and host have the same ns.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
TestPluginTrustedInstall revealed a race in the plugin shutdown logic,
where the exit channel signal was sent even before the propagated mounts
were unmounted. If the same plugin was enabled, it would try to setup
propagated mounts *before* it was unmounted resulting in errors.
This change fixes the behavior by waiting until the unmount completes on
disable before marking the plugin as disabled.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
When a plugin fails to start, we still incorrectly mark it as enabled.
This change verifies that we can dial to the plugin socket to confirm that
the plugin is functional and only then mark the plugin as enabled. Also,
dont delete the plugin on install, if only the enable fails.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
Remove forked reference package. Use normalized named values
everywhere and familiar functions to convert back to familiar
strings for UX and storage compatibility.
Enforce that the source repository in the distribution metadata
is always a normalized string, ignore invalid values which are not.
Update distribution tests to use normalized values.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This persists the "propagated mount" for plugins outside the main
rootfs. This enables `docker plugin upgrade` to not remove potentially
important data during upgrade rather than forcing plugin authors to hard
code a host path to persist data to.
Also migrates old plugins that have a propagated mount which is in the
rootfs on daemon startup.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This allows a plugin to be upgraded without requiring to
uninstall/reinstall a plugin.
Since plugin resources (e.g. volumes) are tied to a plugin ID, this is
important to ensure resources aren't lost.
The plugin must be disabled while upgrading (errors out if enabled).
This does not add any convenience flags for automatically
disabling/re-enabling the plugin during before/after upgrade.
Since an upgrade may change requested permissions, the user is required
to accept permissions just like `docker plugin install`.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fix adds `--filter capability=[volumedriver|authz]` to `docker plugin ls`.
The related docs has been updated.
An integration test has been added.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix adds `--filter enabled=true` to `docker plugin ls`,
as was specified in 28624.
The related API and docs has been updated.
An integration test has been added.
This fix fixes 28624.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Use resolving to repo info as the split point between the
legitimate reference package and forked reference package.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The `digest` data type, used throughout docker for image verification
and identity, has been broken out into `opencontainers/go-digest`. This
PR updates the dependencies and moves uses over to the new type.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Go style calls for mixed caps instead of all caps:
https://golang.org/doc/effective_go.html#mixed-caps
Change LOOKUP, ACQUIRE, and RELEASE to Lookup, Acquire, and Release.
This vendors a fork of libnetwork for now, to deal with a cyclic
dependency issue. The change will be upstream to libnetwork once this is
merged.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Move plugins to shared distribution stack with images.
Create immutable plugin config that matches schema2 requirements.
Ensure data being pushed is same as pulled/created.
Store distribution artifacts in a blobstore.
Run init layer setup for every plugin start.
Fix breakouts from unsafe file accesses.
Add support for `docker plugin install --alias`
Uses normalized references for default names to avoid collisions when using default hosts/tags.
Some refactoring of the plugin manager to support the change, like removing the singleton manager and adding manager config struct.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
When plugins have a positive refcount, they were not allowed to be
removed. However, plugins could still be disabled when volumes
referenced it and containers using them were running.
This change fixes that by enforcing plugin refcount during disable.
A "force" disable option is also added to ignore reference refcounting.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
The current GetAll handles both V2 and legacy plugins. Also due to the
nature of V1 plugins, it also loads them. This causes problems when
loading is not required. Hence adding an independent API that will
return only the plugins that are loaded using v2 mangaed plugins.
Signed-off-by: Madhu Venugopal <madhu@docker.com>
This fix is a follow up for comment:
https://github.com/docker/docker/pull/29186/files#r91277345
While #29186 addresses the issue of `docker inspect <unknown object>`
on Windows, it actually makes `docker plugin inspect <unknown object>`
out `object not found` on Windows as well. This is actually misleading
as plugin is not supported on Windows.
This fix reverted the change in #29186 while at the same time,
checks `not supported` in `docker inspect <unknown object>` so that
- `docker plugin inspect <unknown object>` returns `not supported` on Windows
- `docker inspect <unknown object>` returns `not found` on Windows
This fix is related to #29186 and #29185.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Fixes a case where removing the plugin from disk would
fail silently. Also moves pluginStore remove after we
remove from disk, so 'docker plugin ls' doesn't isn't
empty in case it errors out.
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
Fixes an issue when starting the daemon with live-restore
where previously it was not set, plugins are not running.
Fixes an issue when starting the daemon with live-restore, the plugin
client (for interacting with the plugins HTTP interface) is not set,
causing a panic when the plugin is called.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Adds 2 new methods to v2 plugin `Acquire` and `Release` which allow
refcounting directly at the plugin level instead of just the store.
Since a graphdriver is initialized exactly once, and is really managed
by a separate object, it didn't really seem right to call
`getter.Get()` to refcount graphdriver plugins.
On shutdown it was particularly weird where we'd either need to keep a
driver reference in daemon, or keep a reference to the pluggin getter in
the layer store, and even then still store extra details on if the
graphdriver is a plugin or not.
Instead the plugin proxy itself will handle calling the neccessary
refcounting methods directly on the plugin object.
Also adds a new interface in `plugingetter` to account for these new
functions which are not going to be implemented by v1 plugins.
Changes terms `plugingetter.CREATE` and `plugingetter.REMOVE` to
`ACQUIRE` and `RELEASE` respectively, which seems to be better
adjectives for what we're doing.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Legacy plugins expect host-relative paths (such as for Volume.Mount).
However, a containerized plugin cannot respond with a host-relative
path. Therefore, this commit modifies new volume plugins' paths in Mount
and List to prepend the container's rootfs path.
This introduces a new PropagatedMount field in the Plugin Config.
When it is set for volume plugins, RootfsPropagation is set to rshared
and the path specified by PropagatedMount is bind-mounted with rshared
prior to launching the container. This is so that the daemon code can
access the paths returned by the plugin from the host mount namespace.
Signed-off-by: Tibor Vass <tibor@docker.com>
v2/Plugin struct had fields that were
- purely used by the manager.
- unsafely exposed without proper locking.
This change fixes this, by moving relevant fields to the manager as well
as making remaining fields as private and providing proper accessors for
them.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
This fix tries to address the issue raised on 29185 where
`docker inspect <unknown object>` on Windows will return:
```
Error response from daemon: plugins are not supported on this platform
```
The reason was that in case `--type` is not specified, `docker inspect`
will iterate through different types `container`, `image`, `network`,
`plugin` etc. The `plugin` object is the last type to check.
However, as `plugin` is not supported on Windows yet, the error message
is not very informative for `plugins are not supported on this platform`.
This fix tries to fix the issue by return a `not found` error on unsupported
platforms as well.
An integration test has been added to cover the changes for Windows/Linux.
This fix fixes 29185.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the proposal raised in 28946
to support plugins in `docker inspect`.
The command `docker inspect` already supports
"container", "image", "node", "network", "service", "volume", "task".
However, `--type plugin` is not supported yet at the moment.
This fix address this issue by adding the support of `--type plugin`
for `docker inspect`.
An additional integration test has been added to cover the changes.
This fix fixes 28946.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in discussion of
PR 28735 where it was not possible to manage plugin based on
plugin ID. Previously it was not possible to invoke
`docker plugin inspect` with a plugin ID (or ID prefix).
This fix updates the implementation of `docker plugin inspect`
so that it is possbile to search based on a plugin name, or a
plugin ID. A short format of plugin ID (prefix) is also possible,
as long as there is no ambiguity.
Previously the check of `docker plugin inspect` was mostly done
on the client side. This could potentially cause inconsistency
between API and CMD. This fix move all the checks to daemon side
so that API and CMD will be consistent.
An integration test has been added to cover the changes.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix is a follow up to the comment:
https://github.com/docker/docker/pull/28717#discussion_r90040589
Currently, the collision checking is done at the last step `Add()` of
plugin creation. However, at this stage the context such as plugin
directories have already been creation. In case of name collision,
rollback is needed which could be expensive.
This fix performs the check at the beginning of CreateFromContext using
GetByName. In this way, collision fails fast and no context creation
or rollback is needed.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
A plugin has an `ExitChan` channel which is used to signal the exit of
the plugin process. In a recent change, the initialization was
incorrectly moved to the daemon Shutdown path.
Fix this by initializing the channel during plugin enable.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
This fix tries to fix the issue raised in 28684:
1. Duplicate plugin create with the same name will override the old plugin reference
2. In case an error happens in the middle of the plugin creation, plugin directories
in `/var/lib/docker/plugins` are not cleaned up.
This fix update the plugin store so that `Add()` will return an error if a plugin
with the same name already exist.
This fix also will clean up the directory in `/var/lib/docker/plugins` in case
an error happens in the middle of the plugin creation.
This fix fixes 28684.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
In the plugin manifest, Capabilities has been moved to
Linux.Capabilities to avoid confusion with Interface.Types[i].Capability
A DeviceCreation boolean has also been added to the manifest. This could
be changed in the future to be specific to a major number.
Signed-off-by: Tibor Vass <tibor@docker.com>
During error cases, we dont cleanup correctly. This commit takes care
of removing the plugin, if there are errors after the pull passed. It
also shuts down the plugin, if there are errors after the plugin in the
enable path.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
This fix tries to fix logrus formatting by removing `f` from
`logrus.[Error|Warn|Debug|Fatal|Panic|Info]f` when formatting string
is not present.
Fixed issue #23459
Signed-off-by: Daehyeok Mun <daehyeok@gmail.com>
fmt package formats an error by calling its Error() method.
This results in recursion. Fix this. Also remove dead code.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
This version is across all plugin types and it versions the actual RPC
API between the daemon and the plugin.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Currently the plugins pkg allows a single handler. This assumption
breaks down if there are mutiple listeners to a plugin of a certain
Manifest such as NetworkDriver or IpamDriver when swarm-mode is enabled.
Signed-off-by: Madhu Venugopal <madhu@docker.com>
As part of making graphdrivers support pluginv2, a PluginGetter
interface was necessary for cleaner separation and avoiding import
cycles.
This commit creates a PluginGetter interface and makes pluginStore
implement it. Then the pluginStore object is created in the daemon
(rather than by the plugin manager) and passed to plugin init as
well as to the different subsystems (eg. graphdrivers, volumedrivers).
A side effect of this change was that some code was moved out of
experimental. This is good, since plugin support will be stable soon.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Legacy plugins (aka pluginv1) calls in libnetwork are replaced with
calls using the new plugin model (aka pluginv2). pkg/plugins is still
used for managing the http client connections to the plugin.
This commit makes the necessary changes in docker/docker. Part 2 will
will take care of the libnetwork changes.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Split plugin package into `store` and `v2/plugin`. Now the functionality
is clearly delineated:
- Manager: Manages the global state of the plugin sub-system.
- PluginStore: Manages a collection of plugins (in memory and on-disk)
- Plugin: Manages the single plugin unit.
This also facilitates splitting the global PluginManager lock into:
- PluginManager lock to protect global states.
- PluginStore lock to protect store states.
- Plugin lock to protect individual plugin states.
Importing "github.com/docker/docker/plugin/store" will provide access
to plugins and has lesser dependencies when compared to importing the
original monolithic `plugin package`.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
handleLegacy is a flag to indicate whether daemon is supporting legacy
plugins. When the time comes to remove support for legacy plugins,
flipping this bool is all that will be needed to remove legacy plugin
support. This can be a global variable rather than be embedded in the
manager, thereby cleaning up code.
Also rename to allowV1PluginsFallback for clarity.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
The main intent of handling plugin exit is for graceful shutdown
of plugins during daemon shutdown. So avoid plugin lookup during
plugin exits caused by other reasons (eg. force remove)
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Volumes and other content created under a bind mount should be
recursively propagated using rshared, not shared. This could be
the reason for EBUSY during removal. Override options with rbind,
rshared and see if CI errors are fixed.
May fix#25511
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Legacy plugin model maintained a map of plugins. This is
not used by the new model. Using this map in the new model
causes incorrect lookup of plugins. This change uses adds
a plugin to the map only if its legacy.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Unix sockets are limited to 108 bytes. As a result, we need to be
careful in not using exec-root as the parent directory for pluginID
(which is already 64 bytes), since it can result in socket path names
longer than 108 bytes. Use /tmp instead. Before this change, setting:
- dockerd --exec-root=/go/src/github.com/do passes
- dockerd --exec-root=/go/src/github.com/doc fails
After this change, there's no failure.
Also, write a volume plugins test to verify that the plugins socket
responds.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
This ensures that:
- The in-memory plugin store is populated with all the plugins
- Plugins which were active before daemon restart are active after.
This utilizes the liverestore feature when available, otherwise it
manually starts the plugin.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This patch introduces a new experimental engine-level plugin management
with a new API and command line. Plugins can be distributed via a Docker
registry, and their lifecycle is managed by the engine.
This makes plugins a first-class construct.
For more background, have a look at issue #20363.
Documentation is in a separate commit. If you want to understand how the
new plugin system works, you can start by reading the documentation.
Note: backwards compatibility with existing plugins is maintained,
albeit they won't benefit from the advantages of the new system.
Signed-off-by: Tibor Vass <tibor@docker.com>
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
