Commit graph

46637 commits

Author SHA1 Message Date
Cory Snider
5e4878ed2b Dockerfile/shell: install compose cli plugin
It's convenient to have in the dev container when debugging issues which
reproduce consistently when deploying containers through compose.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-12 12:19:32 -04:00
CrazyMax
8b7ec57cb4
Merge pull request #45930 from crazy-max/fix-moby-bin
ci(bin-image): don't set tags when pushing by digest
2023-07-11 21:31:32 +02:00
Sebastiaan van Stijn
6fee7c8f8b
Merge pull request #45861 from thaJeztah/fileutils_deprecate_GetTotalUsedFds
pkg/fileutils: GetTotalUsedFds(): slight optimization for macOS
2023-07-11 20:06:24 +02:00
CrazyMax
1686540594
ci(bin-image): don't set tags when pushing by digest
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-07-11 19:57:53 +02:00
Bjorn Neergaard
080844b9b3
Merge pull request #44740 from crazy-max/moby-bin
ci: push bin image to Docker Hub
2023-07-11 11:44:36 -06:00
Bjorn Neergaard
8c301fa933
Merge pull request #45915 from akerouanton/create-overlay-on-non-manager
libnet: Return proper error when overlay network can't be created
2023-07-11 10:13:08 -06:00
Sebastiaan van Stijn
43007108e6
Merge pull request #45917 from josegonzalez/patch-2
Sort unconsumed build arguments before usage
2023-07-11 13:25:58 +02:00
Sebastiaan van Stijn
e3f20f5088
Merge pull request #45924 from thaJeztah/registry_minor_fixes
registry: minor fixes and cleanups in search code
2023-07-11 13:19:31 +02:00
Albin Kerouanton
d29240d9eb
libnet: Return a 403 when overlay network isn't allowed
With this change, the API will now return a 403 instead of a 500 when
trying to create an overlay network on a non-manager node.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-07-11 12:41:24 +02:00
Albin Kerouanton
21dcbada2d
libnet: Return proper error when overlay network can't be created
The commit befff0e13f inadvertendly
disabled the error returned when trying to create an overlay network on
a node which is not part of a Swarm cluster.

Since commit e3708a89cc the overlay
netdriver returns the error: `no VNI provided`.

This commit reinstate the original error message by checking if the node
is a manager before calling libnetwork's `controller.NewNetwork()`.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-07-11 12:40:55 +02:00
Sebastiaan van Stijn
93853eca94
pkg/fileutils: GetTotalUsedFds(): slight optimization for macOS
This patch contains some optimizations I still had stashed when working
on eaa9494b71.

- Use the bytes package for handling the output of "lsof", instead of
  converting to a string.
- Count the number of newlines in the output, instead of splitting the
  output into a slice of strings. We're only interested in the number
  of lines in the output.
- Use lsof's -F option to only print the file-descriptor for each line,
  as we don't need other information.
- Use the -l, -n, and -P options to omit converting usernames, host names,
  and port numbers.

From the [LSOF(8)][1] man-page:

   -l    This option inhibits the conversion of user ID numbers to
         login names. It is also useful when login name lookup is
         working improperly or slowly.

   -n    This option inhibits the conversion of network numbers to host
         names for network files. Inhibiting conversion can make lsof run faster.
         It is also useful when host name lookup is not working properly.

   -P    This option inhibits the conversion of port numbers to port names for network files.
         Inhibiting the conversion can make lsof run a little faster.
         It is also useful when host name lookup is not working properly.

Output looks something like;

    lsof -lnP -Ff -p 39849
    p39849
    fcwd
    ftxt
    ftxt
    f0
    f1
    f2
    f3
    f4
    f5
    f6
    f7
    f8
    f9
    f10
    f11

Before/After:

    BenchmarkGetTotalUsedFds-10  122  9479384 ns/op   10816 B/op  63 allocs/op
    BenchmarkGetTotalUsedFds-10  154  7814697 ns/op    7257 B/op  60 allocs/op

[1]: https://opensource.apple.com/source/lsof/lsof-49/lsof/lsof.man.auto.html

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-11 10:50:31 +02:00
Sebastiaan van Stijn
c5c977855d
registry: session.searchRepositories(): return typed error, and small cleanup
- return a errdefs.System if we fail to decode the registry's response
- use strconv.Itoa instead of fmt.Sprintf

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-11 09:47:02 +02:00
Sebastiaan van Stijn
68ebfa2f18
registry: session.searchRepositories(): log actual search URL
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-11 09:41:52 +02:00
Sebastiaan van Stijn
27698f1d13
registry: ParseSearchIndexInfo: remove redundant error-handling
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-10 17:48:10 +02:00
Sebastiaan van Stijn
462d6ef826
Merge pull request #45901 from thaJeztah/api_move_system_info
api/types: move system info types to api/types/system
2023-07-10 17:45:13 +02:00
Sebastiaan van Stijn
e477a57cc5
Merge pull request #45920 from thaJeztah/fix_expose_npe
daemon/containerd: fix assignment to entry in nil map during commit
2023-07-10 13:53:16 +02:00
Sebastiaan van Stijn
a0e1155b28
daemon/containerd: fix assignment to entry in nil map during commit
A panic would happen when converting an config that had ports exposed, because
the ExposedPorts map in the OCI-spec was not initialized. This could happen
when committing a container, or when using the classic builder and the
parent image had ports exposed, for example

    FROM busybox AS stage0
    EXPOSE 80

    FROM stage0 AS stage1
    RUN echo hello

Example of the panic:

    2023/07/07 15:13:02 http: panic serving @: assignment to entry in nil map
    goroutine 1944 [running]:
    net/http.(*conn).serve.func1()
    	/usr/local/go/src/net/http/server.go:1854 +0xbf
    panic({0x45f660, 0xb6a8d0})
    	/usr/local/go/src/runtime/panic.go:890 +0x263
    github.com/docker/docker/daemon/containerd.containerConfigToOciImageConfig(...)
    	/go/src/github.com/docker/docker/daemon/containerd/image_import.go:397
    github.com/docker/docker/daemon/containerd.generateCommitImageConfig({0xc001470498, {0x0, 0x0}, {0xc000c437d8, 0x5}, {0x0, 0x0}, {0xc000c43b27, 0x5}, {0x0, ...}, ...}, ...)
    	/go/src/github.com/docker/docker/daemon/containerd/image_commit.go:138 +0x40e
    github.com/docker/docker/daemon/containerd.(*ImageService).CommitImage(0xc0008853e0, {0xb8f660, 0xc000c4f7c0}, {{0x0, 0x0}, {0x0, 0x0}, 0xc00104b900, 0xc00104b180, {0xc0011a7640, ...}, ...})
    	/go/src/github.com/docker/docker/daemon/containerd/image_commit.go:82 +0x73b
    github.com/docker/docker/daemon/containerd.(*ImageService).CommitBuildStep(0xc0008853e0, {0xb8f660, 0xc000c4f7c0}, {{0x0, 0x0}, {0x0, 0x0}, 0xc00104b900, 0xc00104b180, {0xc0011a7640, ...}, ...})
    	/go/src/github.com/docker/docker/daemon/containerd/image_commit.go:308 +0x110
    github.com/docker/docker/builder/dockerfile.(*Builder).commitContainer(0xc0012b8cc0, {0xb8f660, 0xc000c4f7c0}, 0xc0010b2b60, {0xc0011a7640, 0x40}, 0xc00104b180)
    	/go/src/github.com/docker/docker/builder/dockerfile/internals.go:61 +0x168
    github.com/docker/docker/builder/dockerfile.(*Builder).commit(0xc0012b8cc0, {0xb8f660, 0xc000c4f7c0}, 0xc0010b2b60, {0xc0012a7d80?, 0xc001340060?})
    	/go/src/github.com/docker/docker/builder/dockerfile/internals.go:45 +0x1aa
    github.com/docker/docker/builder/dockerfile.dispatchLabel({0xb8f660, 0xc000c4f7c0}, {0xc0010b2b60, 0xc000c6b628, 0xc0012b8cc0, {0xb80f60, 0xc0011a46c0}, 0xc000bc2560}, 0x1e24a85?)
    	/go/src/github.com/docker/docker/builder/dockerfile/dispatchers.go:83 +0x258
    github.com/docker/docker/builder/dockerfile.dispatch({0xb8f660, 0xc000c4f7c0}, {0xc0010b2b60, 0xc000c6b628, 0xc0012b8cc0, {0xb80f60, 0xc0011a46c0}, 0xc000bc2560}, {0xb7be40, 0xc00111cde0})
    	/go/src/github.com/docker/docker/builder/dockerfile/evaluator.go:74 +0x529
    github.com/docker/docker/builder/dockerfile.(*Builder).dispatchDockerfileWithCancellation(0xc0012b8cc0, {0xb8f660, 0xc000c4f7c0}, {0xc000b1d380, 0x1, 0xc0011a4660?}, {0x0, 0x0, 0x0?}, 0x5c, ...)
    	/go/src/github.com/docker/docker/builder/dockerfile/builder.go:296 +0x8f2
    github.com/docker/docker/builder/dockerfile.(*Builder).build(0xc0012b8cc0, {0xb8f660, 0xc000c4f7c0}, {0xb80f60, 0xc0011a46c0}, 0xc0011a49f0)
    	/go/src/github.com/docker/docker/builder/dockerfile/builder.go:211 +0x2e5
    github.com/docker/docker/builder/dockerfile.(*BuildManager).Build(0xc0008868c0, {0xb8f708, 0xc0011a44b0}, {{0xb789c0, 0xc0011a4540}, {{0xb6b940, 0xc000c22a50}, {0xb6c5e0, 0xc000c22a68}, {0xb6c5e0, ...}, ...}, ...})
    	/go/src/github.com/docker/docker/builder/dockerfile/builder.go:98 +0x358
    github.com/docker/docker/api/server/backend/build.(*Backend).Build(0xc0007d0870, {0xb8f708, 0xc0011a44b0}, {{0xb789c0, 0xc0011a4540}, {{0xb6b940, 0xc000c22a50}, {0xb6c5e0, 0xc000c22a68}, {0xb6c5e0, ...}, ...}, ...})
    	/go/src/github.com/docker/docker/api/server/backend/build/backend.go:69 +0x186
    github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild(0xc0008333c0, {0xb8f708, 0xc0011a44b0}, {0xb8e130, 0xc0000ed500}, 0xc0010d4800, 0xc0012df760?)
    	/go/src/github.com/docker/docker/api/server/router/build/build_routes.go:280 +0x7a6
    github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1({0xb8f708, 0xc0011a44b0}, {0xb8e130?, 0xc0000ed500?}, 0x36cf80?, 0xc0010ab550?)
    	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x15b
    github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1({0xb8f708, 0xc0011a4480}, {0xb8e130, 0xc0000ed500}, 0xc000d787e8?, 0xc000d787a0?)
    	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x4d7
    github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0xb8f708, 0xc0011a4480}, {0xb8e130?, 0xc0000ed500?}, 0xc0010d4800, 0xc0010ab500?)
    	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x649
    github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0xb8e130, 0xc0000ed500}, 0xc0010d4700)
    	/go/src/github.com/docker/docker/api/server/server.go:53 +0x1ce
    net/http.HandlerFunc.ServeHTTP(0xc0010d4600?, {0xb8e130?, 0xc0000ed500?}, 0xc000d789e8?)
    	/usr/local/go/src/net/http/server.go:2122 +0x2f
    github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0001a7e00, {0xb8e130, 0xc0000ed500}, 0xc000d37600)
    	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0x1cf
    net/http.serverHandler.ServeHTTP({0xb7ec58?}, {0xb8e130, 0xc0000ed500}, 0xc000d37600)
    	/usr/local/go/src/net/http/server.go:2936 +0x316
    net/http.(*conn).serve(0xc0012661b0, {0xb8f708, 0xc000fd0360})
    	/usr/local/go/src/net/http/server.go:1995 +0x612
    created by net/http.(*Server).Serve
    	/usr/local/go/src/net/http/server.go:3089 +0x5ed

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-10 10:58:06 +02:00
Sebastiaan van Stijn
49b55b220e
Merge pull request #42138 from cpuguy83/test_daemon_logging
Improve test daemon logging
2023-07-09 10:02:43 +02:00
Jose Diaz-Gonzalez
079a9d4562 Sort unconsumed build arguments before usage
Golang map iteration order is not guaranteed, so in some cases the built slice has it's output of order as well. This means that testing for exact warning messages in docker build output would result in random test failures, making it more annoying for end-users to test against this functionality.

Signed-off-by: Jose Diaz-Gonzalez <email@josediazgonzalez.com>
2023-07-09 03:23:06 -04:00
CrazyMax
41261ea4ec
ci: push bin image to Docker Hub
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-07-09 02:59:03 +02:00
Bjorn Neergaard
c57097bcd4
Merge pull request #45908 from corhere/libn/drop-swarmkit-cruft
libnetwork: drop cruft formerly needed by Swarmkit
2023-07-07 15:26:00 -06:00
Cory Snider
5b3acc15d1 libnetwork: drop legacy driver cruft
...that Swarmkit no longer needs now that it has been migrated to use
the new-style driver registration APIs.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-07 15:02:58 -04:00
Cory Snider
f9a83daebb libnetwork: delete package bitseq
Package idm was the last remaining user of bitseq.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-07 14:53:44 -04:00
Cory Snider
8b167535db libnetwork: delete package idm
The only remaining user is Swarmkit, which now has its own private copy
of the package tailored to its needs.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-07 14:52:08 -04:00
Cory Snider
7366e44019 vendor: github.com/moby/swarmkit/v2 v2.0.0-20230707182847-6f78b8199b05
36334ed187...6f78b8199b

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-07 14:48:12 -04:00
Brian Goff
7d548c6c00
Merge pull request #45902 from thaJeztah/fix_volume_npe 2023-07-07 07:38:21 -07:00
Sebastiaan van Stijn
a490248f4d
daemon: daemon.prepareMountPoints(): fix panic if mount is not a volume
The daemon.lazyInitializeVolume() function only handles restoring Volumes
if a Driver is specified. The Container's MountPoints field may also
contain other kind of mounts (e.g., bind-mounts). Those were ignored, and
don't return an error; 1d9c8619cd/daemon/volumes.go (L243-L252C2)

However, the prepareMountPoints() assumed each MountPoint was a volume,
and logged an informational message about the volume being restored;
1d9c8619cd/daemon/mounts.go (L18-L25)

This would panic if the MountPoint was not a volume;

    github.com/docker/docker/daemon.(*Daemon).prepareMountPoints(0xc00054b7b8?, 0xc0007c2500)
            /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/mounts.go:24 +0x1c0
    github.com/docker/docker/daemon.(*Daemon).restore.func5(0xc0007c2500, 0x0?)
            /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:552 +0x271
    created by github.com/docker/docker/daemon.(*Daemon).restore
            /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:530 +0x8d8
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x564e9be4c7c0]

This issue was introduced in 647c2a6cdd

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-07 15:31:00 +02:00
Sebastiaan van Stijn
c90229ed9a
api/types: move system info types to api/types/system
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-07 13:01:36 +02:00
Bjorn Neergaard
98d3da79ef
Merge pull request #45881 from corhere/libn/register-configless-drivers
libnetwork: stop passing config to drivers which ignore the config argument
2023-07-06 12:17:17 -06:00
Cory Snider
d519bde5da libnet/drivers: stop passing config to drivers...
...which ignore the config argument. Notably, none of the network
drivers referenced by Swarmkit use config, which is good as Swarmkit
unconditionally passes nil for the config when registering drivers.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-06 12:57:00 -04:00
Cory Snider
1980deffae libn: refactor platform driver registration
Hide knowledge of the network driver initializer functions from
controller.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-06 12:56:09 -04:00
Bjorn Neergaard
b60c02b065
Merge pull request #45887 from thaJeztah/move_mtu
daemon/config: move MTU to BridgeConfig, and warn when using on Windows
2023-07-06 09:41:06 -06:00
Sebastiaan van Stijn
a2e9ae0055
Merge pull request #45895 from thaJeztah/albin_maintainer 2023-07-06 17:14:01 +02:00
Sebastiaan van Stijn
065b00b787
Merge pull request #45894 from thaJeztah/laura_maintainer 2023-07-06 17:12:33 +02:00
Sebastiaan van Stijn
bf42ccfbec
Merge pull request #45893 from thaJeztah/kevin_maintainer 2023-07-06 17:08:50 +02:00
Bjorn Neergaard
c9397ec8d4
Merge pull request #45892 from corhere/libn/overlay-cut-vni-list
libnetwork/drivers/overlay: parse VNI list option in O(1) memory
2023-07-06 08:38:19 -06:00
Sebastiaan van Stijn
c87c4b5bbf
Add Albin Kerouanton (@akerouanton) as maintainer
Albin is currently a curator, has been contributing for various years prior
to that, and has taken on the daunting task to work on Moby's networking stack.

Albin would be a great addition to our list of maintainers and to allow him
to perform his work in these areas in a more official capacity.

I nominated Albin as maintainer, and votes passed, so opening a PR to
make it official.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-06 14:47:20 +02:00
Sebastiaan van Stijn
0066138379
Add Kevin Alvarez (@crazy-max) as maintainer
Kevin is a maintainer for BuildKit, Buildx, and Docker's official GitHub
actions (among others), has been our "in-house GitHub actions expert"
for a long time, and has made significant contributions to the integration
with BuildKit, and to improve our build pipeline(s).

Kevin would be a great addition to our list of maintainers and to allow him
to perform his work in these areas in a more official capacity.

I nominated Kevin as maintainer, and votes passed, so opening a PR to
make it official.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-06 14:43:27 +02:00
Sebastiaan van Stijn
d8056ddecd
Add Laura Brehm (@laurazard) as maintainer
Laura has done significant work on the containerd integration, helping
triage and fixing bugs, both in this repository, containerd, and the
docker CLI, and would make a great addition to our list of maintainers.

I nominated Laura as maintainer, and votes passed, so opening a PR to
make it official.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-06 14:42:39 +02:00
Bjorn Neergaard
e4c866f387
Merge pull request #40894 from cpuguy83/health_start_interval
Add health start interval
2023-07-05 19:47:41 -06:00
Brian Goff
2216d3ca8d Add health start interval
This adds an additional interval to be used by healthchecks during the
start period.
Typically when a container is just starting you want to check if it is
ready more quickly than a typical healthcheck might run. Without this
users have to balance between running healthchecks to frequently vs
taking a very long time to mark a container as healthy for the first
time.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-05 23:44:17 +00:00
Brian Goff
914888cf8b Improve test daemon logging
1. On failed start tail the daemon logs
2. Exposes generic tailing functions to make test debugging simpler

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-07-05 22:23:25 +00:00
Cory Snider
51bcf5954f libnet/d/overlay: parse VNI list w/o allocating
Parse in O(1) memory by using strings.Cut to split the string
iteratively.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-05 18:12:06 -04:00
Cory Snider
96ca669fa4 libnet/d/overlay: extract VNI option parsing...
...to a shared utility function.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-05 18:12:04 -04:00
Bjorn Neergaard
e77f9d2327
Merge pull request #45877 from thaJeztah/imports_and_vars
fix some minor linting issues
2023-07-05 15:54:57 -06:00
Bjorn Neergaard
ab9ff592c5
Merge pull request #45882 from corhere/libn/ditch-idm
libnetwork/d/o/ovmanager: switch to using bitmap package directly
2023-07-05 15:49:20 -06:00
Cory Snider
564a13285f libnetwork/bitmap: improve documentation
Clarify that the argument to New is an exclusive upper bound.

Correct the documentation for SetAnyInRange: the end argument is
inclusive rather than exclusive.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-05 16:10:32 -04:00
Cory Snider
0fc6bf9a6e libn/d/o/ovmanager: assign vxlans using bitmap pkg
The idm package wraps bitseq.Handle to provide an offset and
synchronization. bitseq.Handle wraps bitmap.Bitmap to provide
persistence in a datastore. As no datastore is passed and the offset is
zero, the idm.Idm instance is nothing more than a concurrency-safe
wrapper around a bitmap.Bitmap with differently-named methods. Switch
over to using bitmap.Bitmap directly, using the ovmanager driver's mutex
for concurrency control.

Hold the driver mutex for the entire duration that VXLANs are being
assigned to the new network. This makes allocating VXLANs for a network
an atomic operation.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-05 16:10:30 -04:00
Cory Snider
f0127f1617 libn/d/o/ovmanager: inline obtainVxlanID method
Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-05 16:10:12 -04:00
Cory Snider
e97492e579 libn/d/o/ovmanager: drop mutex from network type
In the network.obtainVxlanID() method, the mutex only guards a local
variable and a function argument. Locking is therefore unnecessary.

The network.releaseVxlanID() method is only called in two contexts:
driver.NetworkAllocate(), where the network struct is a local variable
and network.releaseVxlanID() is only called in failure code-paths in
which the network does not escape; and driver.NetworkFree(), while the
driver mutex is held. Locking is therefore unnecessary.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-07-05 16:10:08 -04:00