Commit graph

44743 commits

Author SHA1 Message Date
Bjorn Neergaard
def679d0e0
daemon/config: ignore UTF-8 BOM in config JSON
[RFC 8259] allows for JSON implementations to optionally ignore a BOM
when it helps with interoperability; do so in Moby as Notepad (the only
text editor available out of the box in many versions of Windows Server)
insists on writing UTF-8 with a BOM.

  [RFC 8259]: https://tools.ietf.org/html/rfc8259#section-8.1

Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
(cherry picked from commit bb19265ba8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-01-10 10:51:06 +01:00
Samuel Karp
8f5bbc24ef
Merge pull request #44769 from thaJeztah/23.0_backport_vendor_containerd_1.6.15 2023-01-09 13:24:40 -08:00
Samuel Karp
f6093fab79
Merge pull request #44766 from thaJeztah/23.0_containerd_binary_1.6.15 2023-01-09 13:23:22 -08:00
Sebastiaan van Stijn
f3761a5f38
Merge pull request #44727 from thaJeztah/23.0_backport_fix_42127
[23.0 backport] Check iptables options before looking for ip6tables binary
2023-01-09 18:10:02 +01:00
Sebastiaan van Stijn
691b44b4c3
vendor: github.com/containerd/containerd v1.6.15
no changes to vendored files

full diff: https://github.com/containerd/containerd/compare/v1.6.14...v1.6.15

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit bffe6d5670)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-01-09 13:28:10 +01:00
Sebastiaan van Stijn
5aa0c2cae2
[23.0] update containerd binary to v1.6.15
release notes: https://github.com/containerd/containerd/releases/tag/v1.6.15

> Welcome to the v1.6.15 release of containerd!
>
> The fifteenth patch release for containerd 1.6 fixes an issue with CNI in the CRI plugin
>
> Notable Updates
>
> - Fix no CNI info for pod sandbox on restart in CRI plugin

full diff: https://github.com/containerd/containerd/compare/v1.6.14...v1.6.15

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-01-09 13:12:29 +01:00
Sebastiaan van Stijn
0f2e972234
Merge pull request #44764 from vvoland/fix-test-race-logs-goroutines-23
[23.0 backport] integration-cli: Fix hanging TestLogsFollowGoroutines*
2023-01-09 13:00:41 +01:00
Paweł Gronowski
6acc1701ea
integration-cli: Fix hanging TestLogsFollowGoroutines*
cmd.Wait is called twice from different goroutines which can cause the
test to hang completely. Fix by calling Wait only once and sending its
return value over a channel.

In TestLogsFollowGoroutinesWithStdout also added additional closes and
process kills to ensure that we don't leak anything in case test returns
early because of failed test assertion.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit deb4910c5b)
2023-01-09 10:29:39 +01:00
Sebastiaan van Stijn
6a9d742f12
Merge pull request #44758 from crazy-max/23.0_fix-binary-daemon
[23.0 backport] hack: restore copy_binaries func
2023-01-06 15:24:59 +01:00
CrazyMax
8dfd907478
hack: restore copy_binaries func
This function is still useful for the dev environment.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 7e7b0622a6)
2023-01-06 08:38:33 +01:00
Sebastiaan van Stijn
ba94fff321
Merge pull request #44753 from vvoland/buildflags-nopie-whenrace-23
[23.0 backport] hack/make: Don't add -buildmode=pie with -race
2023-01-05 22:09:55 +01:00
Cory Snider
7b692e1041
Merge pull request #44752 from corhere/backport-23.0/fix-44688
[23.0 backport] Clear conntrack entries for published UDP ports
2023-01-05 15:29:27 -05:00
Paweł Gronowski
0d1aba547e
hack/make: Don't add -buildmode=pie with -race
Make it possible to add `-race` to the BUILDFLAGS without making the
build fail with error:
"-buildmode=pie not supported when -race is enabled"

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit bbe6e9e8d1)
2023-01-05 20:27:08 +01:00
Albin Kerouanton
4da19e2dca Clear conntrack entries for published UDP ports
Conntrack entries are created for UDP flows even if there's nowhere to
route these packets (ie. no listening socket and no NAT rules to
apply). Moreover, iptables NAT rules are evaluated by netfilter only
when creating a new conntrack entry.

When Docker adds NAT rules, netfilter will ignore them for any packet
matching a pre-existing conntrack entry. In such case, when
dockerd runs with userland proxy enabled, packets got routed to it and
the main symptom will be bad source IP address (as shown by #44688).

If the publishing container is run through Docker Swarm or in
"standalone" Docker but with no userland proxy, affected packets will
be dropped (eg. routed to nowhere).

As such, Docker needs to flush all conntrack entries for published UDP
ports to make sure NAT rules are correctly applied to all packets.

- Fixes #44688
- Fixes #8795
- Fixes #16720
- Fixes #7540
- Fixes moby/libnetwork#2423
- and probably more.

As a precautionary measure, those conntrack entries are also flushed
when revoking external connectivity to avoid those entries to be reused
when a new sandbox is created (although the kernel should already
prevent such case).

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
(cherry picked from commit b37d34307d)
Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-01-05 14:22:53 -05:00
Sebastiaan van Stijn
cb13848acd
Merge pull request #44725 from thaJeztah/23.0_backport_fix_volume_createdat
[23.0 backport] Fix volume CreatedAt being altered on initialization
2023-01-03 21:42:58 +01:00
Sebastiaan van Stijn
fe097e018c
Fix volume CreatedAt being altered on initialization
The CreatedAt date was determined from the volume's `_data`
directory (`/var/lib/docker/volumes/<volumename>/_data`).
However, when initializing a volume, this directory is updated,
causing the date to change.

Instead of using the `_data` directory, use its parent directory,
which is not updated afterwards, and should reflect the time that
the volume was created.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 01fd23b625)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-01-03 17:55:26 +01:00
Bjorn Neergaard
a166d959a0
Merge pull request #44717 from neersighted/backport/44668/23.0
[23.0 backport] api: can marshal and unmarshal when args.fields is empty
2023-01-03 08:43:30 -07:00
Sebastiaan van Stijn
0109eac486
Merge pull request #44736 from crazy-max/23.0_dockerfile-cross
[23.0 backport] Dockerfile: use TARGETPLATFORM to build Docker
2023-01-03 10:42:08 +01:00
CrazyMax
d778bbab84
docs: contributing docs update for cross comp
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 83280d44d0)
2023-01-02 18:46:04 +01:00
CrazyMax
2dd577955c
Dockerfile: add "all" stage to build binaries and extra tools
We still need a stage that build binaries and extra tools as well for
docker-ce-packaging repo: ff110508ff/static/Makefile (L41-L57)

This could be removed if we create a package for each project
like it's done in docker-packaging repo: https://github.com/docker/packaging/tree/main/pkg

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit e8a82ed24d)
2023-01-02 18:46:03 +01:00
CrazyMax
3ef71f9085
ci: add extra steps to check artifacts
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 113e95ec02)
2023-01-02 18:46:03 +01:00
CrazyMax
8fbfc2676f
Dockerfile: rename dev stages
Also remove DEV_IMAGE bake var that is not used

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit b565b496ba)
2023-01-02 18:46:03 +01:00
CrazyMax
7e286efcd3
ci: use GITHUB_REF and GITHUB_SHA to set version and commit
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 27ba1766e7)
2023-01-02 18:46:03 +01:00
CrazyMax
f29fe2daec
bake: define additional vars
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 0b3af098e1)
2023-01-02 18:46:03 +01:00
CrazyMax
3b90bb6ad3
bake: define DESTDIR
Keep the same output dir format in the bake definition
as the one used in make scripts.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 9bcf5bed05)
2023-01-02 18:46:02 +01:00
CrazyMax
171471b613
hack: update buildmode pie condition
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 491467b0b9)
2023-01-02 18:46:02 +01:00
CrazyMax
7d8c689923
hack: do not set buildmode pie for statically linked binary
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 347a5f68da)
2023-01-02 18:46:02 +01:00
CrazyMax
75596bee28
hack: use PKG_CONFIG var when checking libdevmapper
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit e37985f590)
2023-01-02 18:46:02 +01:00
CrazyMax
29758a067f
IAmStatic not used anymore
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 84ea9ee0f9)
2023-01-02 18:46:02 +01:00
CrazyMax
5d4ddce5ed
hack: remove containerutility script
this script is not used anymore. containerutility is
built in the Dockerfile.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 04c90b8cf5)
2023-01-02 18:46:01 +01:00
CrazyMax
a85bdbe1ca
Dockerfile: use TARGETPLATFORM to build Docker
Better support for cross compilation so we can fully rely
on `--platform` flag of buildx for a seamless integration.

This removes unnecessary extra cross logic in the Dockerfile,
DOCKER_CROSSPLATFORMS and CROSS vars and some hack scripts as well.

Non-sandboxed build invocation is still supported and dev stages
in the Dockerfile have been updated accordingly.

Bake definition and GitHub Actions workflows have been updated
accordingly as well.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 8086f40123)
2023-01-02 18:46:01 +01:00
Sebastiaan van Stijn
63e861c87e
Merge pull request #44726 from thaJeztah/23.0_backport_devmapper_simplify_udev_error
[23.0 backport] daemon/graphdriver/devicemapper: simplify Udev log, and update link
2023-01-02 18:09:07 +01:00
Albin Kerouanton
6a0a2c4f79
Always use iptables -C to look for rules
iptables -C flag was introduced in v1.4.11, which was released ten
years ago. Thus, there're no more Linux distributions supported by
Docker using this version. As such, this commit removes the old way of
checking if an iptables rule exists (by using substring matching).

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
(cherry picked from commit 799cc143c9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-31 18:17:38 +01:00
Albin Kerouanton
91f2d963c6
Merge iptables.probe() into iptables.detectIptables()
The former was doing some checks and logging warnings, whereas
the latter was doing the same checks but to set some internal variables.
As both are called only once and from the same place, there're now
merged together.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
(cherry picked from commit 205e5278c6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-31 18:17:14 +01:00
Albin Kerouanton
2bf66f725c
Check ipt options before looking for ip6t
iptables package has a function `detectIptables()` called to initialize
some local variables. Since v20.10.0, it first looks for iptables bin,
then ip6tables and finally it checks what iptables flags are available
(including -C). It early exits when ip6tables isn't available, and
doesn't execute the last check.

To remove port mappings (eg. when a container stops/dies), Docker
first checks if those NAT rules exist and then deletes them. However, in
the particular case where there's no ip6tables bin available, iptables
`-C` flag is considered unavailable and thus it looks for NAT rules by
using some substring matching. This substring matching then fails
because `iptables -t nat -S POSTROUTING` dumps rules in a slighly format
than what's expected.

For instance, here's what `iptables -t nat -S POSTROUTING` dumps:

```
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 9999 -j MASQUERADE
```

And here's what Docker looks for:

```
POSTROUTING -p tcp -s 172.18.0.2 -d 172.18.0.2 --dport 9999 -j MASQUERADE
```

Because of that, those rules are considered non-existant by Docker and
thus never deleted. To fix that, this change reorders the code in
`detectIptables()`.

Fixes #42127.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
(cherry picked from commit af7236f85a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-31 18:07:42 +01:00
Chee Hau Lim
1b27ab4c73
libnetwork/iptables: Fix test panic when execute only one test
- use local variables for chains instead of sharing global variables
- make createNewChain a t.Helper

Signed-off-by: Chee Hau Lim <ch33hau@gmail.com>
(cherry picked from commit a2cea992c2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-31 18:07:39 +01:00
Sebastiaan van Stijn
316231cced
daemon/graphdriver/devicemapper: simplify Udev log, and update link
Simplify the error message so that we don't have to distinguish between static-
and non-static builds. Also update the link to the storage-driver section to
use a "/go/" redirect in the docs, as the anchor link was no longer correct.
Using a "/go/" redirect  makes sure the link remains functional if docs is  moving
around.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a5ebd28797)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-31 17:02:30 +01:00
Akihiro Suda
7bf0572f64
Merge pull request #44724 from thaJeztah/23.0_backport_use_the_osusergo_luke
[23.0 backport] daemon/logger/gcplogs: remove ensureHomeIfIAmStatic workaround
2023-01-01 00:12:14 +09:00
Sebastiaan van Stijn
770ace39d8
daemon/logger/gcplogs: remove ensureHomeIfIAmStatic workaround
This function was added in b86e3bee5a to
work around an issue in os/user.Current(), which SEGFAULTS when compiling
statically with cgo enabled (see golang/go#13470).

We hit similar issues in other parts, and contributed a "osusergo" build-
tag in https://go-review.googlesource.com/c/go/+/330753. The "osusergo"
build tag must be set when compiling static binaries with cgo enabled.
If that build-tag is set, the cgo implementation for user.Current() won't
be used, and a pure-go implementation is used instead;
https://github.com/golang/go/blob/go1.19.4/src/os/user/cgo_lookup_unix.go#L5

With the above in place, we no longer need this workaround, and can remove
the ensureHomeIfIAmStatic() function.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 155e39187c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-31 14:50:43 +01:00
Yamazaki Masashi
5e4464798f
api: can marshal and unmarshal when args.fields is empty
Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com>
Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
(cherry picked from commit 5fed968b1d)
Signed-off-by: Bjorn Neergaard <bneergaard@mirantis.com>
2022-12-29 15:40:54 -07:00
Akihiro Suda
2b5ec481e7
Merge pull request #44714 from thaJeztah/23.0_backport_carry_44600_update_delve
[23.0 backport] Update delve version
2022-12-30 07:24:15 +09:00
Bjorn Neergaard
045de829e8
Merge pull request #44708 from zhsj/23.0-btrfs-progs-6.1
[23.0 backport] daemon/graphdriver/btrfs: workaround field rename in btrfs-progs 6.1
2022-12-29 14:47:13 -07:00
Sebastiaan van Stijn
8cb02d8c58
Merge pull request #44703 from thaJeztah/23.0_backport_docker_proxy_version_flag
[23.0 backport] cmd/docker-proxy: add "-v / --version" flag
2022-12-29 21:13:44 +01:00
oscar.chen
515219b716
Update delve version
Signed-off-by: oscar.chen <2972789494@qq.com>
(cherry picked from commit ad8804885c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-29 19:13:08 +01:00
Shengjing Zhu
0393897695 daemon/graphdriver/btrfs: workaround field rename in btrfs-progs 6.1
Closes: #44698

Signed-off-by: Shengjing Zhu <zhsj@debian.org>
(cherry picked from commit ffbbe3d103)
Signed-off-by: Shengjing Zhu <zhsj@debian.org>
2022-12-29 21:16:47 +08:00
Sebastiaan van Stijn
7199300239
cmd/docker-proxy: add "-v / --version" flag
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 12df6024db)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-28 08:24:47 +01:00
Sebastiaan van Stijn
4ddde6dde5
Merge pull request #44702 from thaJeztah/23.0_backport_fix_buildkit_ci
[23.0 backport] gha: update buildkit to fix integration tests
2022-12-27 21:54:53 +01:00
Sebastiaan van Stijn
e9075f70c7
gha: update buildkit to fix integration tests
full diff: 0bfcd83e6d...d77361423c

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c42b304f62)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-27 19:23:28 +01:00
Tianon Gravi
cba986b340
Merge pull request #44679 from thaJeztah/23.0_backport_errdefs_improve_debug_logs
[23.0 backport] errdefs: FromStatusCode() don't log "FIXME" debug message
2022-12-21 09:38:50 -08:00
Sebastiaan van Stijn
b2d2f012b4
errdefs: FromStatusCode() don't log "FIXME" debug message
This utility is used by the client, which cannot do anything about errors
received from the API. In situations where no API connection was possible,
for example, if the client has no permissions to connect to the socket,
the request would have a "-1" status-code;
3e39ec60da/client/request.go (L133-L134)

In this case, a client with "debug" enabled, would print _and_ log a confusing
error message:

    DEBU[0000] FIXME: Got an status-code for which error does not match any expected type!!!  error="Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile.repro&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=repro&target=&ulimits=null&version=1\": dial unix /var/run/docker.sock: connect: permission denied" module=api status_code=-1
    Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile.repro&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=repro&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied

In the above; `DEBU` logs the error (including the "FIXME"), and the second
line is the error message printed.

This was a mistake on my side when I added the `FromStatusCode` utility. I
implemented that to be the counterpart to `FromError`, but in doing so also
copied over the logging (see 1af30c50ca). That
log-message is only intended to be logged on the daemon side, for situations
where we return an error without a proper errdefs (which would result in an
500 "internal server error" to be returned by the API).

This patch removes the debug log, and a minor cleanup to explicitly return
"nil" if we didn't get an error in the first place.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2e67c827bb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-20 16:07:07 +01:00