Commit graph

37141 commits

Author SHA1 Message Date
Kir Kolyshkin
1e13f66fbb logger: fix follow logs for max-file=1
In case jsonlogfile is used with max-file=1 and max-size set,
the log rotation is not perfomed; instead, the log file is closed
and re-open with O_TRUNC.

This situation is not handled by the log reader in follow mode,
leading to an issue of log reader being stuck forever.

This situation (file close/reopen) could be handled in waitRead(),
but fsnotify library chose to not listen to or deliver this event
(IN_CLOSE_WRITE in inotify lingo).

So, we have to handle this by checking the file size upon receiving
io.EOF from the log reader, and comparing the size with the one received
earlier. In case the new size is less than the old one, the file was
truncated and we need to seek to its beginning.

Fixes #39235.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 9cd24ba605)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2019-09-23 11:29:52 -07:00
Andrew Hsu
9a9ff44418
Merge pull request #296 from thaJeztah/19.03_backport_exec_hang
[19.03 backport] Handle blocked I/O of exec'd processes
2019-09-23 10:05:26 -07:00
Andrew Hsu
c5754a7329
Merge pull request #281 from thaJeztah/19.03_backport_fix_golint_again
[19.03 backport] Integration: change signatures to fix golint warnings
2019-09-23 09:37:43 -07:00
Andrew Hsu
c27f11fa2e
Merge pull request #340 from thaJeztah/19.03_backport_bump_grpc
[19.03 backport] bump google.golang.org/grpc v1.23.0 (CVE-2019-9512, CVE-2019-9514, CVE-2019-9515)
2019-09-23 09:32:43 -07:00
Andrew Hsu
09b72e0be4
Merge pull request #335 from thaJeztah/19.03_backport_dev190815
[19.03 backport] fix docker rmi stucking
2019-09-23 09:30:56 -07:00
Sebastiaan van Stijn
0d6d5b392a
Revert "Fixing integration test"
This reverts commit 8fca769bd5.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-23 18:22:42 +02:00
Tibor Vass
51f390dd79
integration: get tests to compile again
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit a281289515)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-23 18:20:56 +02:00
Sebastiaan van Stijn
168132b632
integration: change testGraphDriver signature to fix linting
Line 441: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dac5710b68)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-23 18:20:54 +02:00
Sebastiaan van Stijn
3f4338cf04
integration: change createAmbiguousNetworks signature to fix linting
Line 30: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 123e29f44a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-23 18:20:52 +02:00
Sebastiaan van Stijn
3ade7ca12b
integration: change container.Run signature to fix linting
Line 59: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9f9b4290b9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-23 18:20:50 +02:00
Sebastiaan van Stijn
9c49308cce
integration: change container.Create signature to fix linting
```
Line 25: warning: context.Context should be the first parameter of a function (golint)
Line 44: warning: context.Context should be the first parameter of a function (golint)
Line 52: warning: context.Context should be the first parameter of a function (golint)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b4c46b0dac)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-23 18:20:48 +02:00
Sebastiaan van Stijn
91757722a9
integration: change network.CreateNoError signature to fix linting
Line 30: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit caec45a37f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-23 18:20:46 +02:00
Sebastiaan van Stijn
235fa0eee8
Revert "integration: have container.Create call compile"
This reverts commit 8f4b96f19e.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-23 18:20:43 +02:00
Andrew Hsu
08af35b250
Merge pull request #375 from thaJeztah/19.03_revert_bump_swarmkit
[19.03] Revert "bump swarmkit to f35d9100f2c6ac810cc8d7de6e8f93dcc7a42d29"
2019-09-23 09:18:46 -07:00
Sebastiaan van Stijn
ef4366ee89
Revert "[19.03] bump swarmkit to f35d9100f2c6ac810cc8d7de6e8f93dcc7a42d29"
This reverts commit 02465c9f9d.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-21 10:32:58 +02:00
Kir Kolyshkin
e489130717
daemon/ProcessEvent: make sure to cancel the contexts
Reported by govet linter:

> daemon/monitor.go:57:9: lostcancel: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak (govet)
> 			ctx, _ := context.WithTimeout(context.Background(), 2*time.Second)
> 			     ^
> daemon/monitor.go:128:9: lostcancel: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak (govet)
> 			ctx, _ := context.WithTimeout(context.Background(), 2*time.Second)
> 			     ^

Fixes: b5f288 ("Handle blocked I/O of exec'd processes")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 53cbf1797b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-20 19:10:55 +02:00
Michael Crosby
34b31d0ee0
Handle blocked I/O of exec'd processes
This is the second part to
https://github.com/containerd/containerd/pull/3361 and will help process
delete not block forever when the process exists but the I/O was
inherited by a subprocess that lives on.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit b5f28865ef)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-20 19:10:52 +02:00
Haichao Yang
8306f1e31e
fix docker rmi stucking
Signed-off-by: Haichao Yang <yang.haichao@zte.com.cn>
(cherry picked from commit d3f64846a2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-20 19:06:32 +02:00
Sebastiaan van Stijn
879fba29d5
Merge pull request #358 from thaJeztah/19.03_backport_exec_failure_event
[19.03 backport] Send exec exit event on failures
2019-09-20 19:05:06 +02:00
Andrew Hsu
421a3aa737
Merge pull request #368 from tiborvass/19.03-remove-warning-on-v2schema1-pull
[19.03 backport] distribution: modify warning logic when pulling v2 schema1 manifests
2019-09-19 18:35:13 -07:00
Andrew Hsu
23c7134bad
Merge pull request #371 from andrewhsu/clean
[19.03] Jenkinsfile: ensure all containers are cleaned up
2019-09-19 17:56:39 -07:00
Andrew Hsu
2399b7a91b
Merge pull request #369 from thaJeztah/19.03_bump_swarmkit
[19.03] bump swarmkit to f35d9100f2c6ac810cc8d7de6e8f93dcc7a42d29
2019-09-19 17:48:36 -07:00
Andrew Hsu
7cb08ca538
Merge pull request #334 from thaJeztah/19.03_backport_switch_creack_pty
[19.03 backport] switch kr/pty to creack/pty v1.1.7
2019-09-19 17:46:32 -07:00
Andrew Hsu
2aa5322638
Merge pull request #352 from thaJeztah/19.03_backport_detect_invalid_linked_container
[19.03 backport] Return "invalid parameter" when linking to non-existing container
2019-09-19 17:45:09 -07:00
Andrew Hsu
3f7e68e894
Merge pull request #370 from andrewhsu/skip-rs1
[19.03] skip win-RS1 on PRs
2019-09-19 17:37:00 -07:00
Tibor Vass
4b5c535be9 Jenkinsfile: ensure all containers are cleaned up
By convention, containers spawned by jenkins jobs have the name:
docker-pr${BUILD_NUMBER}

That works fine for jobs with a single container. This commit cleans up
when multiple containers are spawned with the convention that their names
share the same "docker-pr${BUILD_NUMBER}-" prefix.

Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit f470698c2c)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2019-09-19 23:42:20 +00:00
Andrew Hsu
80727e5a92
Merge pull request #365 from thaJeztah/19.03_backport_pull_platform_regression
[19.03 backport] Fix error handling of incorrect --platform values
2019-09-19 16:31:27 -07:00
Andrew Hsu
ff3dc8a7c4
Merge pull request #354 from thaJeztah/19.03_backport_revert_remove_TestSearchCmdOptions
[19.03 backport] Revert "Remove TestSearchCmdOptions test"
2019-09-19 16:24:58 -07:00
Andrew Hsu
b2823e4609
Merge pull request #366 from thaJeztah/19.03_backport_update_to_go_1.12.9
[19.03 backport] Bump Golang 1.12.9
2019-09-19 16:22:21 -07:00
Andrew Hsu
a18eea2702 run integration-cli stages on s390x and ppc64le if not a PR check
Essentially, run on merge to target branch which may or may not be
master branch. Could be 19.03 branch, for example.

See: https://jenkins.io/doc/book/pipeline/syntax/

Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit e653943e8b)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2019-09-19 21:41:36 +00:00
Andrew Hsu
36fc8f5809 skip win-RS1 on PRs unless the checkbox is checked
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 039eb05ac8)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2019-09-19 21:41:10 +00:00
Sebastiaan van Stijn
ca22ec44ba Jenkinsfile: shorten stage names for consistency and to fit Jenkins UI
The Blue Ocean UI truncates names, which makes it possible to distinguish
which Windows stage is RS1 or RS5. This patch shortens those names so that they
fit in the Blue Ocean UI.

Other stages and parameters were renamed as well to better reflect what they're running;

Before             | After
-------------------|--------------------------------
janky              | amd64
power              | ppc64le
power-master       | ppc64le integration-cli
windowsRS1         | win-RS1
windowsRS5-process | win-RS5
z                  | s390x
z-master           | x390x integration-cli

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

WIP renames

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

(cherry picked from commit c18f793f40)

Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2019-09-19 21:41:05 +00:00
Sebastiaan van Stijn
02465c9f9d
[19.03] bump swarmkit to f35d9100f2c6ac810cc8d7de6e8f93dcc7a42d29
full diff: bbe341867e...f35d9100f2

changes included:

- docker/swarmkit#2891 [19.03 backport] Remove hardcoded IPAM config subnet value for ingress network
  - backport of docker/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network
  - fixes [ENGORC-2651] Specifying --default-addr-pool for docker swarm init is not picked up by ingress network

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-19 09:21:46 +02:00
Tibor Vass
f9232e3f11 distribution: modify warning logic when pulling v2 schema1 manifests
The warning on pull was incorrectly asking to contact registry admins.
It is kept on push however.

Pulling manifest lists with v2 schema1 manifests will not be supported thus
there is a warning for those, but wording changed to suggest repository author
to upgrade.

Finally, a milder warning on regular pull is kept ONLY for DockerHub users
in order to incite moving away from schema1.

Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 647dfe99a5)
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-09-19 00:46:14 +00:00
Sebastiaan van Stijn
6de2bd28df
Merge pull request #360 from thaJeztah/19.03_backport_fix_missing_dir_cleanup_file
[19.03 backport] Ensure parent dir exists for mount cleanup file
2019-09-18 13:55:38 +02:00
Kirill Kolyshkin
2b10608f16
Merge pull request #363 from thaJeztah/19.03_backport_64align
[19.03 backport] atomic: patch 64bit alignment on 32bit systems
2019-09-18 11:30:26 +03:00
Andrew Hsu
c416072ced
Merge pull request #330 from thaJeztah/19.03_backport_bump_libnetwork
[19.03 backport] bump libnetwork to 92d1fbe1eb0883cf11d283cea8e658275146411d
2019-09-16 16:12:22 -07:00
Sebastiaan van Stijn
5196dc65e7
bump hashicorp/go-sockaddr v1.0.2
full diff: 6d291a969b...v1.0.2

Relevant changes:
  - hashicorp/go-sockaddr#25 Add android os
  - hashicorp/go-sockaddr#28 Add go.mod

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 492945c2d5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 18:34:24 +02:00
Sebastiaan van Stijn
8abb005598
bump hashicorp/go-multierror v1.0.0, add errwrap v1.0.0
full diff: fcdddc395d...v1.0.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 720b66ee1f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 18:34:22 +02:00
Sebastiaan van Stijn
44ca36c7cf
bump lib network to 92d1fbe1eb0883cf11d283cea8e658275146411d
full diff: 09cdcc8c0e...92d1fbe1eb

relevant changes included (omitting some changes that were added _and_ reverted in this bump):

- docker/libnetwork#2433 Fix parseIP error when parseIP before get AddressFamily
  - fixes docker/libnetwork#2431 parseIP Error ip=[172 17 0 2 0 0 0 0 0 0 0 0 0 0 0 0]
  - https://github.com/docker/libnetwork/issues/2289
  - this was a regression introduced in docker/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling
- docker/libnetwork#2440 Bump hashicorp go-sockaddr v1.0.2, go-multierror v1.0.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit bab58c1924)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 18:34:20 +02:00
Sebastiaan van Stijn
b6190c2713
bump libnetwork to 09cdcc8c0eab3946c2d70e8f6225b05baf1e90d1
full diff: 83d30db536...09cdcc8c0e

changes included:

- docker/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling
- docker/libnetwork#2411 Macvlan network handles netlabel.Internal wrong
  - fixes docker/libnetwork#2410 Macvlan network handles netlabel.Internal wrong
- docker/libnetwork#2414 Allow network with --config-from to be --internal
  - fixes docker/libnetwork#2413 Network with --config-from does not honor --internal
- docker/libnetwork#2351 Use fewer modprobes
  - relates to moby/moby#38930 Use fewer modprobes
- docker/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks
  - carry of docker/libnetwork#2295 phantom ip/mac vlan network after a powercycle
  - fixes docker/libnetwork#1743 Phantom docker network

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6f234db9fe)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 18:34:18 +02:00
CarlosEDP
ca89db221f
Update modules to support riscv64
Signed-off-by: CarlosEDP <me@carlosedp.com>
(cherry picked from commit 9eaab0425b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 18:34:16 +02:00
Sebastiaan van Stijn
f3e1aff81d
bump libnetwork. vishvananda/netlink 1.0, vishvananda/netns
full diffs:

- fc5a7d91d5...62a13ae87c
- b2de5d10e3...v1.0.0
- 604eaf189e...13995c7128ccc8e51e9a6bd2b551020a27180abd

notable changes in libnetwork:

- docker/libnetwork#2366 Bump vishvananda/netlink to 1.0.0
- docker/libnetwork#2339 controller: Check if IPTables is enabled for arrangeUserFilterRule
  - addresses docker/libnetwork#2158 dockerd when run with --iptables=false modifies iptables by adding DOCKER-USER
  - addresses moby/moby#35777 With iptables=false dockerd still creates DOCKER-USER chain and rules
  - addresses docker/for-linux#136 dockerd --iptables=false adds DOCKER-USER chain and modify FORWARD chain anyway
- docker/libnetwork#2394 Make DNS records and queries case-insensitive
  - addresses moby/moby#28689 Embedded DNS is case-sensitive
  - addresses moby/moby#21169 hostnames with new networking are case-sensitive

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 344b093258)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 18:34:13 +02:00
Jintao Zhang
ad1e6bae4f
Bump Golang 1.12.9
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
(cherry picked from commit 01d6a56699)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 16:20:10 +02:00
Sebastiaan van Stijn
b262d40daf
Merge pull request #345 from thaJeztah/19.03_backport_swarm_flaky
[19.03 backport] integration-cli: fix swarm tests flakiness
2019-09-16 16:00:57 +02:00
Sebastiaan van Stijn
8ab5e2a004
Add regression tests for invalid platform status codes
Before we handled containerd errors, using an invalid platform produced a 500 status:

```bash
curl -v \
  -X POST \
  --unix-socket /var/run/docker.sock \
  "http://localhost:2375/v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest" \
  -H "Content-Type: application/json"
```

```
* Connected to localhost (docker.sock) port 80 (#0)
> POST /v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest HTTP/1.1
> Host: localhost:2375
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.40
< Content-Length: 85
< Content-Type: application/json
< Date: Mon, 15 Jul 2019 15:25:44 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/19.03.0-rc2 (linux)
<
{"message":"\"foobar\": unknown operating system or architecture: invalid argument"}
```

That problem is now fixed, and the API correctly returns a 4xx status:

```bash
curl -v \
  -X POST \
  --unix-socket /var/run/docker.sock \
  "http://localhost:2375/v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest" \
  -H "Content-Type: application/json"
```

```
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> POST /v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest HTTP/1.1
> Host: localhost:2375
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 400 Bad Request
< Api-Version: 1.41
< Content-Type: application/json
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Mon, 15 Jul 2019 15:13:42 GMT
< Content-Length: 85
<
{"message":"\"foobar\": unknown operating system or architecture: invalid argument"}
* Curl_http_done: called premature == 0
```

This patch adds tests to validate the behaviour

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9d1b4f5fc3)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 15:37:47 +02:00
Sebastiaan van Stijn
7b2d5556d5
errdefs: convert containerd errors to the correct status code
In situations where the containerd error is consumed directly
and not received over gRPC, errors were not translated.

This patch converts containerd errors to the correct HTTP
status code.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4a516215e2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 15:37:37 +02:00
Tonis Tiigi
776c2bd113
atomic: patch 64bit alignment on 32bit systems
causes panic on armv7

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit af2e82d054)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 15:32:14 +02:00
Brian Goff
c67edc5d61
Ensure parent dir exists for mount cleanup file
While investigating a test failure, I found this in the logs:

```
time="2019-07-04T15:06:32.622506760Z" level=warning msg="Error while setting daemon root propagation, this is not generally critical but may cause some functionality to not work or fallback to less desirable behavior" dir=/go/src/github.com/docker/docker/bundles/test-integration/d1285b8250308/root error="error writing file to signal mount cleanup on shutdown: open /tmp/dxr/d1285b8250308/unmount-on-shutdown: no such file or directory"
```

This path is generated from the daemon's exec-root, which appears to not
exist yet. This change just makes sure it exists before we try to write
a file.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 7725b88edc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 15:22:31 +02:00
Michael Crosby
1920db0267
Send exec exit event on failures
Fixes #39427

This always sends the exec exit events even when the exec fails to find
the binary.  A standard 127 exit status is sent in this situation.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit c08d4da6e5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-16 14:56:59 +02:00