From 29ff2af2d374b0a5b15fa8d01a43b140910bfff6 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sat, 3 Apr 2021 19:24:12 +0200 Subject: [PATCH] Fix flaky TestInspect This test has been flaky for a long time, failing with: --- FAIL: TestInspect (12.04s) inspect_test.go:39: timeout hit after 10s: waiting for tasks to enter run state. task failed with error: task: non-zero exit (1) While looking through logs, noticed tasks were started, entering RUNNING stage, and then exited, to be started again. state.transition="STARTING->RUNNING" ... msg="fatal task error" error="task: non-zero exit (1)" ... state.transition="RUNNING->FAILED" Looking for possible reasons, first considering network issues (possibly we ran out of IP addresses or networking not cleaned up), then I spotted the issue. The service is started with; Command: []string{"/bin/top"}, Args: []string{"-u", "root"}, The `-u root` is not an argument for the service, but for `/bin/top`. While the Ubuntu/Debian/GNU version `top` has a -u/-U option; docker run --rm ubuntu:20.04 top -h 2>&1 | grep '\-u' top -hv | -bcEHiOSs1 -d secs -n max -u|U user -p pid(s) -o field -w [cols] The *busybox* version of top does not: docker run --rm busybox top --help 2>&1 | grep '\-u' So running `top -u root` would cause the task to fail; docker run --rm busybox top -u root top: invalid option -- u ... echo $? 1 As a result, the service went into a crash-loop, and because the `poll.WaitOn()` was running with a short interval, in many cases would _just_ find the RUNNING state, perform the `service inspect`, and pass, but in other cases, it would not be that lucky, and continue polling untill we reached the 10 seconds timeout, and mark the test as failed. Looking for history of this option (was it previously using a different image?) I found this was added in 6cd6d8646a90fa2013416bc8f11bd78d72c4180d, but probably just missed during review. Given that the option is only set to have "something" to inspect, I replaced the `-u root` with `-d 5`, which makes top refresh with a 5 second interval. Note that there is another test (`TestServiceListWithStatuses) that uses the same spec, however, that test is skipped based on API version of the test-daemon, and (to be looked into), when performing that check, no API version is known, causing the test to (always?) be skipped: === RUN TestServiceListWithStatuses --- SKIP: TestServiceListWithStatuses (0.00s) list_test.go:34: versions.LessThan(testEnv.DaemonInfo.ServerVersion, "1.41") Signed-off-by: Sebastiaan van Stijn (cherry picked from commit 00cb3073f4bfe7a24302aba8df87b37a33c99697) Signed-off-by: Sebastiaan van Stijn --- integration/service/inspect_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/integration/service/inspect_test.go b/integration/service/inspect_test.go index 7b6d6ff116..5d8f12dbc3 100644 --- a/integration/service/inspect_test.go +++ b/integration/service/inspect_test.go @@ -88,7 +88,7 @@ func fullSwarmServiceSpec(name string, replicas uint64) swarmtypes.ServiceSpec { Image: "busybox:latest", Labels: map[string]string{"container-label": "container-value"}, Command: []string{"/bin/top"}, - Args: []string{"-u", "root"}, + Args: []string{"-d", "5"}, Hostname: "hostname", Env: []string{"envvar=envvalue"}, Dir: "/work",