Commit graph

35009 commits

Author SHA1 Message Date
Yong Tang
5c233cf431
Merge pull request #36913 from vdemeester/test-skip-non-root
Skip some tests requires root uid when run as user…
2018-04-23 11:42:42 -07:00
Sebastiaan van Stijn
20b524bf2e
Merge pull request #36920 from kolyshkin/cancel-func
context.WithTimeout: do call the cancel func
2018-04-23 20:14:25 +02:00
Sebastiaan van Stijn
3dd4be9d89
Merge pull request #36927 from wdhif/patch-1
Fix version mismatch in API the documentation
2018-04-23 19:40:57 +02:00
Wassim Dhif
2058928edc Fix version mismatch in API documentation
Fix a version mismatch in the API documentation.

Signed-off-by: Wassim DHIF <wassimdhif@gmail.com>
2018-04-23 18:57:35 +02:00
Vincent Demeester
a7999aaa53
Skip some tests requires root uid when run as user
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-23 10:14:39 +02:00
Vincent Demeester
5d8b88b114
Fix typo in idtools tests
It should check `os.Geteuid` with `uid` instead of `os.Getegid`.
On the container (where the tests run), the uid and gid seems to be
the same, thus this doesn't fail.

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-23 10:14:07 +02:00
Yong Tang
57142e89be
Merge pull request #36922 from yongtang/04212018-restart_test.go
Some improvement in restart_test.go
2018-04-22 09:34:14 -07:00
Yong Tang
67535921b3 Some improvement in restart_test.go
This fix consists of some improvement in restart_test.go
by replacing Fatal with assert, so that they are consistent
with other tests in integration/container.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
2018-04-21 19:42:20 +00:00
Vincent Demeester
53982e3fc1
Merge pull request #36091 from kolyshkin/mount
pkg/mount improvements
2018-04-21 11:03:54 +02:00
Sebastiaan van Stijn
cf9c48bb3e
Merge pull request #36879 from cpuguy83/extra_unmount_check
Extra check before unmounting on shutdown
2018-04-20 17:08:11 -07:00
Brian Goff
c028b6d089
Merge pull request #36912 from vdemeester/api-no-daemon-dep
Remove daemon dependency on api packages
2018-04-20 16:04:26 -04:00
Kir Kolyshkin
05e2f7e2fa context.WithTimeout: do call the cancel func
govet complains (when using standard "context" package):

> the cancel function returned by context.WithTimeout should be called,
> not discarded, to avoid a context leak (vet)

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-20 12:27:13 -07:00
Vincent Demeester
ba02880786
Merge pull request #36888 from vdemeester/mark-test-helpers
Make testing helpers as such…
2018-04-20 17:00:17 +02:00
Vincent Demeester
7d9cbfa96f
Merge pull request #36744 from kolyshkin/t-enospc
TestDaemonNoSpaceLeftOnDeviceError: simplify
2018-04-20 15:40:39 +02:00
Vincent Demeester
c4ba5be6af
Merge pull request #36492 from alejgh/36478-log-standardization
36478 storage-drivers log format standardization
2018-04-20 12:11:14 +02:00
Vincent Demeester
f4106b46db
Remove daemon dependency on api packages
We are using interface in the api routers to not explicitely depend on
the daemon struct (`daemon.Daemon`), but somehow, we do depend on the
`daemon` package for the cluster functionalities.

This removes this dependency by defining the correct interfaces.

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-20 11:06:40 +02:00
Vincent Demeester
cb8db44395
Make testing helpers as such…
That way, those lines won't be reported in the failure.

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-20 10:38:43 +02:00
Sebastiaan van Stijn
d6706dddd5
Merge pull request #36887 from alexjomin/master
pkg/term add build tag for netbsd
2018-04-19 19:41:36 -07:00
Brian Goff
a9bd60d674
Merge pull request #36896 from cpuguy83/move_mount_parsing
Move mount parsing to separate package.
2018-04-19 20:31:53 -04:00
Brian Goff
4da83efdfb
Merge pull request #36875 from vdemeester/integration-cli-some-fixme
[test/integration-cli] small cleanups of FIXME(s)
2018-04-19 20:31:27 -04:00
Kir Kolyshkin
ce468f0ad0 volume/local/TestCreateWithOpts(): use mount filter
This is not for the sake of test to run faster of course;
this is to simplify the code as well as have some more
testing for mount.SingleEntryFilter().

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-19 14:50:03 -07:00
Kir Kolyshkin
ac39a95ea6 volume/local: call umount unconditionally
There is no need to parse mount table and iterate through the list of
mounts, and then call Unmount() which again parses the mount table and
iterates through the list of mounts.

It is totally OK to call Unmount() unconditionally.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-19 14:49:54 -07:00
Kir Kolyshkin
a1d095199d mount.Unmount(): don't look into /proc/self/mountinfo
Now, every Unmount() call takes a burden to parse the whole nine yards
of /proc/self/mountinfo to figure out whether the given mount point is
mounted or not (and returns an error in case parsing fails somehow).

Instead, let's just call umount() and ignore EINVAL, which results
in the same behavior, but much better performance.

Note that EINVAL is returned from umount(2) not only in the case when
`target` is not mounted, but also for invalid flags. As the flags are
hardcoded here, it can't be the case.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-19 14:49:50 -07:00
Kir Kolyshkin
c611f18a7f pkg/mount/mountinfo_linux: parser speed up
The mountinfo parser implemented via `fmt.Sscanf()` is slower than the one
using `strings.Split()` and `strconv.Atoi()`. This rewrite helps to speed it
up to a factor of 8x, here is a result from go bench:

> BenchmarkParsingScanf-4      	     300	  22294112 ns/op
> BenchmarkParsingSplit-4      	    3000	   2780703 ns/op

I tried other approaches, such as using `fmt.Sscanf()` for the first
three (integer) fields and `strings.Split()` for the rest, but it slows
things down considerably:

> BenchmarkParsingMixed-4      	    1000	   8827058 ns/op

Note the old code uses `fmt.Sscanf`, when a linear search for '-' field,
when a split for the last 3 fields. The new code relies on a single
split.

I have also added more comments to aid in future development.

Finally, the test data is fixed to now have white space before the first field.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-19 14:49:42 -07:00
Kir Kolyshkin
871c957242 getSourceMount(): simplify
The flow of getSourceMount was:
 1 get all entries from /proc/self/mountinfo
 2 do a linear search for the `source` directory
 3 if found, return its data
 4 get the parent directory of `source`, goto 2

The repeated linear search through the whole mountinfo (which can have
thousands of records) is inefficient. Instead, let's just

 1 collect all the relevant records (only those mount points
   that can be a parent of `source`)
 2 find the record with the longest mountpath, return its data

This was tested manually with something like

```go
func TestGetSourceMount(t *testing.T) {
	mnt, flags, err := getSourceMount("/sys/devices/msr/")
	assert.NoError(t, err)
	t.Logf("mnt: %v, flags: %v", mnt, flags)
}
```

...but it relies on having a specific mount points on the system
being used for testing.

[v2: add unit tests for ParentsFilter]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-19 14:49:17 -07:00
Kir Kolyshkin
d3ebcde82a daemon.cleanupMounts(): use mount.SingleEntryFilter
Use mount.SingleEntryFilter as we're only interested in a single entry.

Test case data of TestShouldUnmountRoot is modified accordingly, as
from now on:

1. `info` can't be nil;

2. the mountpoint check is not performed (as SingleEntryFilter
   guarantees it to be equal to daemon.root).

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-19 14:48:25 -07:00
Kir Kolyshkin
bb934c6aca pkg/mount: implement/use filter for mountinfo parsing
Functions `GetMounts()` and `parseMountTable()` return all the entries
as read and parsed from /proc/self/mountinfo. In many cases the caller
is only interested only one or a few entries, not all of them.

One good example is `Mounted()` function, which looks for a specific
entry only. Another example is `RecursiveUnmount()` which is only
interested in mount under a specific path.

This commit adds `filter` argument to `GetMounts()` to implement
two things:
 1. filter out entries a caller is not interested in
 2. stop processing if a caller is found what it wanted

`nil` can be passed to get a backward-compatible behavior, i.e. return
all the entries.

A few filters are implemented:
 - `PrefixFilter`: filters out all entries not under `prefix`
 - `SingleEntryFilter`: looks for a specific entry

Finally, `Mounted()` is modified to use `SingleEntryFilter()`, and
`RecursiveUnmount()` is using `PrefixFilter()`.

Unit tests are added to check filters are working.

[v2: ditch NoFilter, use nil]
[v3: ditch GetMountsFiltered()]
[v4: add unit test for filters]
[v5: switch to gotestyourself]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-19 14:48:09 -07:00
Brian Goff
e396b27b7f
Merge pull request #36886 from vdemeester/experimental-plugins-migration
Migrate test-integration-cli experimental plugin tests to integration
2018-04-19 16:13:05 -04:00
Brian Goff
6a70fd222b Move mount parsing to separate package.
This moves the platform specific stuff in a separate package and keeps
the `volume` package and the defined interfaces light to import.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-19 06:35:54 -04:00
Vincent Demeester
69bab3832c
Migrate test-integration-cli experimental plugin tests to integration
All `plugins` tests that require an `ExperimentalDaemon` are migrated
to `integration/plugin/*` and start an experimental daemon to test on
it.

The end goal being to remove the `experimental` build.

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-19 11:57:28 +02:00
Vincent Demeester
8bb5a28eed
Merge pull request #36684 from cpuguy83/bump_containerd_client
Bump containerd client
2018-04-19 10:38:43 +02:00
Sebastiaan van Stijn
f1783da736
Merge pull request #36865 from vdemeester/integration-move-swarm-plugin-test
Migrate TestAPISwarmServicesPlugin to integration
2018-04-18 21:15:59 -07:00
Sebastiaan van Stijn
f0b2f9362f
Merge pull request #36894 from cpuguy83/bump_x_net
Bump golang.org/x/net to go1.10 release commit
2018-04-18 18:04:51 -07:00
Brian Goff
c403f0036b Extra check before unmounting on shutdown
This makes sure that if the daemon root was already a self-binded mount
(thus meaning the daemonc only performed a remount) that the daemon does
not try to unmount.

Example:

```
$ sudo mount --bind /var/lib/docker /var/lib/docker
$ sudo dockerd &
```

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-18 20:43:42 -04:00
Sebastiaan van Stijn
69a5611dde
Merge pull request #36848 from cpuguy83/libcontainerd_client_locking
Relax some libcontainerd client locking
2018-04-18 16:47:23 -07:00
Brian Goff
6cd806aa53
Merge pull request #36637 from cpuguy83/no_global_driver_store
No global volume driver store
2018-04-18 16:32:23 -04:00
Brian Goff
848ef1da8b Bump golang.org/x/net to go1.10 release commit
The primary need for this is x/net/context now is just a type alias to
the stdlib context package.
This makes issues with conflicts between "golang.org/x/net/context" and
the stdib "context" go away (primarily a concern in interface
definitions/implementations).

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-18 15:57:24 -04:00
Vincent Demeester
5d2afe4f51
Remove daemon.BuildImageWithOut and use cli helpers function
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-18 16:45:55 +02:00
Vincent Demeester
aa0db6f9e1
Migrate TestAPISwarmServicesPlugin to integration
Also starts to create more "poll/check" function to `internal/test/daemon`.

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-18 16:18:53 +02:00
Vincent Demeester
3b01e92c9a
Add a Cleanup function that cleans exec root and swarm files
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-04-18 16:14:50 +02:00
Vincent Demeester
25ff68a80f
Merge pull request #36832 from vdemeester/integration-daemon-api-helpers
More integration-cli/integration refactoring + request package
2018-04-18 16:12:14 +02:00
Alexandre Jomin
0d82bc142c pkg/term add build tag for netbsd
Signed-off-by: Alexandre Jomin <alexandrejomin@gmail.com>
2018-04-18 15:16:06 +02:00
Brian Goff
4923f8cc63
Merge pull request #36858 from j-haj/36414-standardize_build_paths
Standardize artefact path for all build stages
2018-04-18 06:58:38 -04:00
Akihiro Suda
5dae4828f3
Merge pull request #36880 from thaJeztah/bump-swarmkit
Bump SwarmKit to 9c2aa152c3054371b833483a7ddad8d15052ec4f
2018-04-18 19:10:55 +09:00
Sebastiaan van Stijn
333b2f28fe
Bump SwarmKit to 9c2aa152c3054371b833483a7ddad8d15052ec4f
Relevant changes:

- docker/swarmkit#2551 RoleManager will remove deleted nodes from the cluster membership
- docker/swarmkit#2574 Scheduler/TaskReaper: handle unassigned tasks marked for shutdown
- docker/swarmkit#2561 Avoid predefined error log
- docker/swarmkit#2557 Task reaper should delete tasks with removed slots that were not yet assigned
- docker/swarmkit#2587 [fips] Agent reports FIPS status
- docker/swarmkit#2603 Fix manager/state/store.timedMutex

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-04-17 13:44:49 -07:00
Brian Goff
977109d808 Remove use of global volume driver store
Instead of using a global store for volume drivers, scope the driver
store to the caller (e.g. the volume store). This makes testing much
simpler.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-17 14:07:08 -04:00
Brian Goff
b047f675ea Rename volumedrivers package to drivers
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-17 14:06:53 -04:00
Brian Goff
0023abbad3 Remove old/uneeded volume migration from vers 1.7
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-17 14:06:53 -04:00
Brian Goff
63826e291b Move direct volume driver interaction to store
Since the volume store already provides this functionality, we should
just use it rather than duplicating it.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-17 14:06:53 -04:00
Brian Goff
806700e410 Relax global client lock on containerd restore.
This unblocks the client to take other restore requests and makes sure
that a long/stuck request can't block the client forever.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-04-17 12:07:06 -04:00