Commit graph

46671 commits

Author SHA1 Message Date
Sebastiaan van Stijn
2cb982b506
pkg/plugins: run tests with t.Parallel()
Some tests are testing timeouts and take a long time to run. Run the tests
in parallel, so that the test-suite takes shorter to run.

Before:

    ok  github.com/docker/docker/pkg/plugins	34.013s

After:

    ok  github.com/docker/docker/pkg/plugins	17.945s

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-12 22:30:46 +02:00
Sebastiaan van Stijn
b39362295a
pkg/plugins: don't share plugin server between tests
Refactor setupRemotePluginServer() to be a helper, and to spin up a test-
server for each test instead of sharing the same instance between tests.

This allows the tests to be run in parallel without stepping on each-other's
toes (tearing down the server).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-12 22:30:43 +02:00
Brian Goff
1d0bb72c5b
Merge pull request #45947 from crazy-max/ci-buildkit-goversion 2023-07-12 12:22:10 -07:00
Cory Snider
b1ead679f3
Merge pull request #45931 from corhere/install-compose-plugin
Dockerfile/shell: install compose cli plugin
2023-07-12 13:24:12 -04:00
CrazyMax
ee9fe2c838
ci(buildkit): match moby go version for buildkit tests
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-07-12 19:17:25 +02:00
CrazyMax
3379414c76
vendor: github.com/moby/buildkit@v0.11 (0a15675)
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-07-12 19:17:24 +02:00
Cory Snider
5e4878ed2b Dockerfile/shell: install compose cli plugin
It's convenient to have in the dev container when debugging issues which
reproduce consistently when deploying containers through compose.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-12 12:19:32 -04:00
Bjorn Neergaard
06b991f48f
Merge pull request #45944 from neersighted/moby-bin-tags
ci(bin-image): add SHA-based tags
2023-07-12 08:03:48 -06:00
Sebastiaan van Stijn
e57479dc30
Merge pull request #45888 from thaJeztah/cleanup_iptables
libnetwork/iptables: some cleanups and refactoring
2023-07-12 15:14:47 +02:00
Bjorn Neergaard
ecfa4f5866
ci(bin-image): add SHA-based tags
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-07-12 07:04:41 -06:00
Sebastiaan van Stijn
93e65a2a6c
Merge pull request #45918 from akerouanton/hack-make-run-disable-tls
Disable tls when launching dockerd through hack/make.sh
2023-07-12 14:37:56 +02:00
Bjorn Neergaard
daa3166cb6
Merge pull request #45932 from crazy-max/fix-moby-bin2
ci(bin-image): fix meta step
2023-07-12 06:11:56 -06:00
Kevin Alvarez
0a126a85a4
ci(bin-image): fix meta step
We can't upload the same file in a matrix so generate
metadata in prepare job instead. Also fixes wrong bake meta
file in merge job.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-07-11 22:54:27 +02:00
Bjorn Neergaard
ebcb230cff
Merge pull request #45923 from thaJeztah/client_header
client: remove custom "headers" type (use http.Header), and omit "version" header on API >= 1.30
2023-07-11 14:30:20 -06:00
CrazyMax
749d7449f9
ci(bin-image): fix typo
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-07-11 22:01:55 +02:00
CrazyMax
8b7ec57cb4
Merge pull request #45930 from crazy-max/fix-moby-bin
ci(bin-image): don't set tags when pushing by digest
2023-07-11 21:31:32 +02:00
Sebastiaan van Stijn
6fee7c8f8b
Merge pull request #45861 from thaJeztah/fileutils_deprecate_GetTotalUsedFds
pkg/fileutils: GetTotalUsedFds(): slight optimization for macOS
2023-07-11 20:06:24 +02:00
CrazyMax
1686540594
ci(bin-image): don't set tags when pushing by digest
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-07-11 19:57:53 +02:00
Bjorn Neergaard
080844b9b3
Merge pull request #44740 from crazy-max/moby-bin
ci: push bin image to Docker Hub
2023-07-11 11:44:36 -06:00
Bjorn Neergaard
8c301fa933
Merge pull request #45915 from akerouanton/create-overlay-on-non-manager
libnet: Return proper error when overlay network can't be created
2023-07-11 10:13:08 -06:00
Sebastiaan van Stijn
43007108e6
Merge pull request #45917 from josegonzalez/patch-2
Sort unconsumed build arguments before usage
2023-07-11 13:25:58 +02:00
Sebastiaan van Stijn
e3f20f5088
Merge pull request #45924 from thaJeztah/registry_minor_fixes
registry: minor fixes and cleanups in search code
2023-07-11 13:19:31 +02:00
Sebastiaan van Stijn
83477ce8d0
client: remove custom "headers" type, and use "http.Header" instead
Use http.Header, which is more descriptive on intent, and we're already
importing the package in the client. Removing the "header" type also fixes
various locations where the type was shadowed by local variables named
"headers".

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-11 13:14:28 +02:00
Sebastiaan van Stijn
e4ce676d3d
client: ServiceCreate, ServiceUpdate: omit "version" header on API >= 1.30
The "version" header was added in c0afd9c873,
but used the wrong information to get the API version.

This issue was fixed in a9d20916c3, which switched
the API handler code to get the API version from the context. That change is part
of Docker Engine 20.10 (API v1.30 and up)

This patch updates the code to only set the header on APi v1.29 and older, as it's
not used by newer API versions.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-11 13:00:00 +02:00
Albin Kerouanton
d29240d9eb
libnet: Return a 403 when overlay network isn't allowed
With this change, the API will now return a 403 instead of a 500 when
trying to create an overlay network on a non-manager node.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-07-11 12:41:24 +02:00
Albin Kerouanton
21dcbada2d
libnet: Return proper error when overlay network can't be created
The commit befff0e13f inadvertendly
disabled the error returned when trying to create an overlay network on
a node which is not part of a Swarm cluster.

Since commit e3708a89cc the overlay
netdriver returns the error: `no VNI provided`.

This commit reinstate the original error message by checking if the node
is a manager before calling libnetwork's `controller.NewNetwork()`.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-07-11 12:40:55 +02:00
Sebastiaan van Stijn
93853eca94
pkg/fileutils: GetTotalUsedFds(): slight optimization for macOS
This patch contains some optimizations I still had stashed when working
on eaa9494b71.

- Use the bytes package for handling the output of "lsof", instead of
  converting to a string.
- Count the number of newlines in the output, instead of splitting the
  output into a slice of strings. We're only interested in the number
  of lines in the output.
- Use lsof's -F option to only print the file-descriptor for each line,
  as we don't need other information.
- Use the -l, -n, and -P options to omit converting usernames, host names,
  and port numbers.

From the [LSOF(8)][1] man-page:

   -l    This option inhibits the conversion of user ID numbers to
         login names. It is also useful when login name lookup is
         working improperly or slowly.

   -n    This option inhibits the conversion of network numbers to host
         names for network files. Inhibiting conversion can make lsof run faster.
         It is also useful when host name lookup is not working properly.

   -P    This option inhibits the conversion of port numbers to port names for network files.
         Inhibiting the conversion can make lsof run a little faster.
         It is also useful when host name lookup is not working properly.

Output looks something like;

    lsof -lnP -Ff -p 39849
    p39849
    fcwd
    ftxt
    ftxt
    f0
    f1
    f2
    f3
    f4
    f5
    f6
    f7
    f8
    f9
    f10
    f11

Before/After:

    BenchmarkGetTotalUsedFds-10  122  9479384 ns/op   10816 B/op  63 allocs/op
    BenchmarkGetTotalUsedFds-10  154  7814697 ns/op    7257 B/op  60 allocs/op

[1]: https://opensource.apple.com/source/lsof/lsof-49/lsof/lsof.man.auto.html

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-11 10:50:31 +02:00
Sebastiaan van Stijn
c5c977855d
registry: session.searchRepositories(): return typed error, and small cleanup
- return a errdefs.System if we fail to decode the registry's response
- use strconv.Itoa instead of fmt.Sprintf

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-11 09:47:02 +02:00
Sebastiaan van Stijn
68ebfa2f18
registry: session.searchRepositories(): log actual search URL
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-11 09:41:52 +02:00
Sebastiaan van Stijn
27698f1d13
registry: ParseSearchIndexInfo: remove redundant error-handling
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-10 17:48:10 +02:00
Sebastiaan van Stijn
462d6ef826
Merge pull request #45901 from thaJeztah/api_move_system_info
api/types: move system info types to api/types/system
2023-07-10 17:45:13 +02:00
Sebastiaan van Stijn
e477a57cc5
Merge pull request #45920 from thaJeztah/fix_expose_npe
daemon/containerd: fix assignment to entry in nil map during commit
2023-07-10 13:53:16 +02:00
Sebastiaan van Stijn
a0e1155b28
daemon/containerd: fix assignment to entry in nil map during commit
A panic would happen when converting an config that had ports exposed, because
the ExposedPorts map in the OCI-spec was not initialized. This could happen
when committing a container, or when using the classic builder and the
parent image had ports exposed, for example

    FROM busybox AS stage0
    EXPOSE 80

    FROM stage0 AS stage1
    RUN echo hello

Example of the panic:

    2023/07/07 15:13:02 http: panic serving @: assignment to entry in nil map
    goroutine 1944 [running]:
    net/http.(*conn).serve.func1()
    	/usr/local/go/src/net/http/server.go:1854 +0xbf
    panic({0x45f660, 0xb6a8d0})
    	/usr/local/go/src/runtime/panic.go:890 +0x263
    github.com/docker/docker/daemon/containerd.containerConfigToOciImageConfig(...)
    	/go/src/github.com/docker/docker/daemon/containerd/image_import.go:397
    github.com/docker/docker/daemon/containerd.generateCommitImageConfig({0xc001470498, {0x0, 0x0}, {0xc000c437d8, 0x5}, {0x0, 0x0}, {0xc000c43b27, 0x5}, {0x0, ...}, ...}, ...)
    	/go/src/github.com/docker/docker/daemon/containerd/image_commit.go:138 +0x40e
    github.com/docker/docker/daemon/containerd.(*ImageService).CommitImage(0xc0008853e0, {0xb8f660, 0xc000c4f7c0}, {{0x0, 0x0}, {0x0, 0x0}, 0xc00104b900, 0xc00104b180, {0xc0011a7640, ...}, ...})
    	/go/src/github.com/docker/docker/daemon/containerd/image_commit.go:82 +0x73b
    github.com/docker/docker/daemon/containerd.(*ImageService).CommitBuildStep(0xc0008853e0, {0xb8f660, 0xc000c4f7c0}, {{0x0, 0x0}, {0x0, 0x0}, 0xc00104b900, 0xc00104b180, {0xc0011a7640, ...}, ...})
    	/go/src/github.com/docker/docker/daemon/containerd/image_commit.go:308 +0x110
    github.com/docker/docker/builder/dockerfile.(*Builder).commitContainer(0xc0012b8cc0, {0xb8f660, 0xc000c4f7c0}, 0xc0010b2b60, {0xc0011a7640, 0x40}, 0xc00104b180)
    	/go/src/github.com/docker/docker/builder/dockerfile/internals.go:61 +0x168
    github.com/docker/docker/builder/dockerfile.(*Builder).commit(0xc0012b8cc0, {0xb8f660, 0xc000c4f7c0}, 0xc0010b2b60, {0xc0012a7d80?, 0xc001340060?})
    	/go/src/github.com/docker/docker/builder/dockerfile/internals.go:45 +0x1aa
    github.com/docker/docker/builder/dockerfile.dispatchLabel({0xb8f660, 0xc000c4f7c0}, {0xc0010b2b60, 0xc000c6b628, 0xc0012b8cc0, {0xb80f60, 0xc0011a46c0}, 0xc000bc2560}, 0x1e24a85?)
    	/go/src/github.com/docker/docker/builder/dockerfile/dispatchers.go:83 +0x258
    github.com/docker/docker/builder/dockerfile.dispatch({0xb8f660, 0xc000c4f7c0}, {0xc0010b2b60, 0xc000c6b628, 0xc0012b8cc0, {0xb80f60, 0xc0011a46c0}, 0xc000bc2560}, {0xb7be40, 0xc00111cde0})
    	/go/src/github.com/docker/docker/builder/dockerfile/evaluator.go:74 +0x529
    github.com/docker/docker/builder/dockerfile.(*Builder).dispatchDockerfileWithCancellation(0xc0012b8cc0, {0xb8f660, 0xc000c4f7c0}, {0xc000b1d380, 0x1, 0xc0011a4660?}, {0x0, 0x0, 0x0?}, 0x5c, ...)
    	/go/src/github.com/docker/docker/builder/dockerfile/builder.go:296 +0x8f2
    github.com/docker/docker/builder/dockerfile.(*Builder).build(0xc0012b8cc0, {0xb8f660, 0xc000c4f7c0}, {0xb80f60, 0xc0011a46c0}, 0xc0011a49f0)
    	/go/src/github.com/docker/docker/builder/dockerfile/builder.go:211 +0x2e5
    github.com/docker/docker/builder/dockerfile.(*BuildManager).Build(0xc0008868c0, {0xb8f708, 0xc0011a44b0}, {{0xb789c0, 0xc0011a4540}, {{0xb6b940, 0xc000c22a50}, {0xb6c5e0, 0xc000c22a68}, {0xb6c5e0, ...}, ...}, ...})
    	/go/src/github.com/docker/docker/builder/dockerfile/builder.go:98 +0x358
    github.com/docker/docker/api/server/backend/build.(*Backend).Build(0xc0007d0870, {0xb8f708, 0xc0011a44b0}, {{0xb789c0, 0xc0011a4540}, {{0xb6b940, 0xc000c22a50}, {0xb6c5e0, 0xc000c22a68}, {0xb6c5e0, ...}, ...}, ...})
    	/go/src/github.com/docker/docker/api/server/backend/build/backend.go:69 +0x186
    github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild(0xc0008333c0, {0xb8f708, 0xc0011a44b0}, {0xb8e130, 0xc0000ed500}, 0xc0010d4800, 0xc0012df760?)
    	/go/src/github.com/docker/docker/api/server/router/build/build_routes.go:280 +0x7a6
    github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1({0xb8f708, 0xc0011a44b0}, {0xb8e130?, 0xc0000ed500?}, 0x36cf80?, 0xc0010ab550?)
    	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x15b
    github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1({0xb8f708, 0xc0011a4480}, {0xb8e130, 0xc0000ed500}, 0xc000d787e8?, 0xc000d787a0?)
    	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x4d7
    github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0xb8f708, 0xc0011a4480}, {0xb8e130?, 0xc0000ed500?}, 0xc0010d4800, 0xc0010ab500?)
    	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x649
    github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0xb8e130, 0xc0000ed500}, 0xc0010d4700)
    	/go/src/github.com/docker/docker/api/server/server.go:53 +0x1ce
    net/http.HandlerFunc.ServeHTTP(0xc0010d4600?, {0xb8e130?, 0xc0000ed500?}, 0xc000d789e8?)
    	/usr/local/go/src/net/http/server.go:2122 +0x2f
    github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0001a7e00, {0xb8e130, 0xc0000ed500}, 0xc000d37600)
    	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0x1cf
    net/http.serverHandler.ServeHTTP({0xb7ec58?}, {0xb8e130, 0xc0000ed500}, 0xc000d37600)
    	/usr/local/go/src/net/http/server.go:2936 +0x316
    net/http.(*conn).serve(0xc0012661b0, {0xb8f708, 0xc000fd0360})
    	/usr/local/go/src/net/http/server.go:1995 +0x612
    created by net/http.(*Server).Serve
    	/usr/local/go/src/net/http/server.go:3089 +0x5ed

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-10 10:58:06 +02:00
Albin Kerouanton
6b1b71ced4
Disable tls when launching dockerd through hack/make.sh
The daemon sleeps for 15 seconds at start up when the API binds to a TCP
socket with no TLS certificate set. That's what the hack/make/run script
does, but it doesn't explicitly disable tls, thus we're experiencing
this annoying delay every time we use this script.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-07-09 22:26:07 +02:00
Sebastiaan van Stijn
49b55b220e
Merge pull request #42138 from cpuguy83/test_daemon_logging
Improve test daemon logging
2023-07-09 10:02:43 +02:00
Jose Diaz-Gonzalez
079a9d4562 Sort unconsumed build arguments before usage
Golang map iteration order is not guaranteed, so in some cases the built slice has it's output of order as well. This means that testing for exact warning messages in docker build output would result in random test failures, making it more annoying for end-users to test against this functionality.

Signed-off-by: Jose Diaz-Gonzalez <email@josediazgonzalez.com>
2023-07-09 03:23:06 -04:00
CrazyMax
41261ea4ec
ci: push bin image to Docker Hub
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-07-09 02:59:03 +02:00
Bjorn Neergaard
c57097bcd4
Merge pull request #45908 from corhere/libn/drop-swarmkit-cruft
libnetwork: drop cruft formerly needed by Swarmkit
2023-07-07 15:26:00 -06:00
Cory Snider
5b3acc15d1 libnetwork: drop legacy driver cruft
...that Swarmkit no longer needs now that it has been migrated to use
the new-style driver registration APIs.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-07 15:02:58 -04:00
Cory Snider
f9a83daebb libnetwork: delete package bitseq
Package idm was the last remaining user of bitseq.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-07 14:53:44 -04:00
Cory Snider
8b167535db libnetwork: delete package idm
The only remaining user is Swarmkit, which now has its own private copy
of the package tailored to its needs.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-07 14:52:08 -04:00
Cory Snider
7366e44019 vendor: github.com/moby/swarmkit/v2 v2.0.0-20230707182847-6f78b8199b05
36334ed187...6f78b8199b

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-07 14:48:12 -04:00
Brian Goff
7d548c6c00
Merge pull request #45902 from thaJeztah/fix_volume_npe 2023-07-07 07:38:21 -07:00
Sebastiaan van Stijn
a490248f4d
daemon: daemon.prepareMountPoints(): fix panic if mount is not a volume
The daemon.lazyInitializeVolume() function only handles restoring Volumes
if a Driver is specified. The Container's MountPoints field may also
contain other kind of mounts (e.g., bind-mounts). Those were ignored, and
don't return an error; 1d9c8619cd/daemon/volumes.go (L243-L252C2)

However, the prepareMountPoints() assumed each MountPoint was a volume,
and logged an informational message about the volume being restored;
1d9c8619cd/daemon/mounts.go (L18-L25)

This would panic if the MountPoint was not a volume;

    github.com/docker/docker/daemon.(*Daemon).prepareMountPoints(0xc00054b7b8?, 0xc0007c2500)
            /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/mounts.go:24 +0x1c0
    github.com/docker/docker/daemon.(*Daemon).restore.func5(0xc0007c2500, 0x0?)
            /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:552 +0x271
    created by github.com/docker/docker/daemon.(*Daemon).restore
            /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:530 +0x8d8
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x564e9be4c7c0]

This issue was introduced in 647c2a6cdd

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-07 15:31:00 +02:00
Sebastiaan van Stijn
c90229ed9a
api/types: move system info types to api/types/system
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-07 13:01:36 +02:00
Bjorn Neergaard
98d3da79ef
Merge pull request #45881 from corhere/libn/register-configless-drivers
libnetwork: stop passing config to drivers which ignore the config argument
2023-07-06 12:17:17 -06:00
Cory Snider
d519bde5da libnet/drivers: stop passing config to drivers...
...which ignore the config argument. Notably, none of the network
drivers referenced by Swarmkit use config, which is good as Swarmkit
unconditionally passes nil for the config when registering drivers.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-06 12:57:00 -04:00
Cory Snider
1980deffae libn: refactor platform driver registration
Hide knowledge of the network driver initializer functions from
controller.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-06 12:56:09 -04:00
Bjorn Neergaard
b60c02b065
Merge pull request #45887 from thaJeztah/move_mtu
daemon/config: move MTU to BridgeConfig, and warn when using on Windows
2023-07-06 09:41:06 -06:00
Sebastiaan van Stijn
a2e9ae0055
Merge pull request #45895 from thaJeztah/albin_maintainer 2023-07-06 17:14:01 +02:00