Commit graph

48535 commits

Author SHA1 Message Date
Sebastiaan van Stijn
508a2d979a
Revert "temporary: Disable deprecate-integration-cli validation"
This reverts commit bdc7d0c2db.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-17 15:59:52 +01:00
Sebastiaan van Stijn
1fd682930a
Merge pull request #46634 from rumpl/c8d-classic-builder-cache
c8d: make the cache in classic builder work
2024-01-17 15:51:16 +01:00
Evan Lezar
f7065ab207
Add testutil.TempDir function
This change adds a TempDir function that ensures the correct permissions for
the fake-root user in rootless mode.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2024-01-17 15:44:47 +01:00
Akihiro Suda
84d2229ffe
dockerd-rootless-setuptool.sh: add nsenter subcommand
Usage: `dockerd-rootless-setuptool.sh nsenter -- ip a`

Expected to be used for debugging.

Ported from nerdctl's `containerd-rootless-setuptool.sh`
https://github.com/containerd/nerdctl/blob/v1.7.2/extras/rootless/containerd-rootless-setuptool.sh#L142-L147

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2024-01-17 22:59:18 +09:00
Akihiro Suda
7d08d84b03
dockerd-rootless.sh: set rootlesskit --state-dir=DIR
Now the state dir is set to `${XDG_RUNTIME_DIR}/dockerd-rootless`.

This is similar to `${XDG_RUNTIME_DIR}/containerd-rootless` used in nerdctl:
https://github.com/containerd/nerdctl/blob/v1.7.2/extras/rootless/containerd-rootless.sh#L35

Prior to this commit, the state dir was unset and a random dir under `/tmp` was used.
(e.g., `/tmp/rootlesskit1869901982`)

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2024-01-17 22:59:18 +09:00
Akihiro Suda
ae9fbcb82c
testutil/daemon: shorten XDG_RUNTIME_DIR
XDG_RUNTIME_DIR will contain sockets so its path mustn't be too long.
Prior to this commit, it was set to very long path like
`/go/src/github.com/docker/docker/bundles/test-integration/TestDiskUsage/de4fb36576d7d/xdgrun`

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2024-01-17 22:59:18 +09:00
Paweł Gronowski
bdc7d0c2db
temporary: Disable deprecate-integration-cli validation
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-17 14:57:58 +01:00
Paweł Gronowski
8390bc5683
c8d/cache: Use ContainerConfig from content store
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-17 14:57:55 +01:00
Paweł Gronowski
f760cb4f97
c8d/builder: Store ContainerConfig
Serialize ContainerConfig to content store and store its digest in
label.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-17 14:57:54 +01:00
Paweł Gronowski
a5a15c7782
c8d/cache: Optimize FROM scratch case
Consider only images that were built `FROM scratch` as valid candidates
for the `FROM scratch` + INSTRUCTION build step.

The images are marked as `FROM scratch` based by the classic builder
with a special label. It must be a new label instead of empty parent
label, because empty label values are not persisted.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-17 14:57:53 +01:00
Djordje Lukic
71ebfc7c63
c8d: make the cache in classic builder work
In order for the cache in the classic builder to work we need to:
- use the came comparison function as the graph drivers implementation
- save the container config when commiting the image
- use all images to search a 'FROM "scratch"' image
- load all images if `cacheFrom` is empty

Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-17 14:57:50 +01:00
Sebastiaan van Stijn
af4f6c124d
Merge pull request #47088 from vvoland/testing-setupTest-parallel
environment: Error when t.Parallel was called before Protect
2024-01-17 14:57:19 +01:00
Sebastiaan van Stijn
60ed73bb91
Merge pull request #47044 from corhere/lazy-health-status
Lazily checkpoint container health status to disk
2024-01-17 14:12:38 +01:00
Sebastiaan van Stijn
d3e08fe3cf
Merge pull request #47087 from neersighted/cdi_feature
cdi: use separate feature-flag
2024-01-17 14:11:41 +01:00
Paweł Gronowski
24da5233dd
integration: Fix Parallel before setupTest
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-17 13:06:05 +01:00
Bjorn Neergaard
d22c775e04
cdi: use separate feature-flag
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-17 11:31:37 +01:00
Paweł Gronowski
fa4deb02af
environment: Error when t.Parallel was called before Protect
Protecting the environment relies on the shared state (containers,
images, etc) which might already be mutated by other tests if the test
opted in into the Parallel execution before Protect was called.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-17 10:54:11 +01:00
Cory Snider
97d32bb7d7 daemon: stop checkpointing health probes to disk
The health status and probe log of containers are not mission-criticial
data which must survive a crash. It is not worth prematrely wearing out
consumer-grade flash storage by overwriting and fsync()ing the container
config on after every probe. Update only the live Container object and
the ViewDB replica on every container health probe instead. It will
eventually get checkpointed along with some other state (or config)
change. Running containers will not be checkpointed on daemon shutdown
when live-restore is enabled, but it does not matter: the health status
and probe log will be zeroed out when the daemon starts back up.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2024-01-16 14:09:40 -05:00
Sebastiaan van Stijn
2c47a6df0d
Merge pull request #46978 from vvoland/c8d-fix-export
c8d/exporter: Use WithSkipMissing
2024-01-16 20:04:17 +01:00
Brian Goff
353bccdf22
Merge pull request #47051 from corhere/reinit-health-on-live-restore 2024-01-16 10:08:20 -08:00
Sebastiaan van Stijn
9045bea756
Merge pull request #47082 from vvoland/c8d-integration-parallel-fix
integration/image: Move Parallel after setupTest
2024-01-16 18:41:30 +01:00
Sebastiaan van Stijn
8afd6daa3a
Merge pull request #47086 from AkihiroSuda/fix-47085
(*Daemon).fillRootlessVersion: fix nil panic
2024-01-16 18:34:30 +01:00
Akihiro Suda
a6d9462cb1
(*Daemon).fillRootlessVersion: fix nil panic
Fix issue 47085

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2024-01-17 00:50:53 +09:00
Akihiro Suda
5c4189e6b2
Merge pull request #47083 from AkihiroSuda/follow-up-47076
dockerd-rootless.sh: fix inconsistent description about "builtin" driver
2024-01-17 00:19:38 +09:00
Akihiro Suda
e954cdc188
dockerd-rootless.sh: fix inconsistent description about "builtin" driver
The "builtin" port driver was marked as "Slow" in the row for the lxc-user-nic
network driver, while it was marked as "Fast" in other rows.

It had to be consistently marked as "Fast" regardless to the network driver.
It is still not as fast as rootful.

Follow-up to PR 47076
Fixes: b5a5ecf4a3

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2024-01-16 23:11:40 +09:00
Paweł Gronowski
4ff8436ebd
Merge pull request #47081 from vvoland/c8d-integration-cli-skipcontainerconfig
c8d/integration-cli: Skip tests checking `ContainerConfig`
2024-01-16 14:50:02 +01:00
Paweł Gronowski
9de132dbf6
integration/image: Move Parallel after setupTest
setupTest should be called before Parallel as it modifies the test
environment which might produce:

```
fatal error: concurrent map writes

goroutine 143 [running]:
github.com/docker/docker/testutil/environment.(*Execution).ProtectContainer(...)
	/go/src/github.com/docker/docker/testutil/environment/protect.go:59
github.com/docker/docker/testutil/environment.ProtectContainers({0x12e8d98, 0xc00040e420}, {0x12f2878?, 0xc0004fc340}, 0xc0001fac00)
	/go/src/github.com/docker/docker/testutil/environment/protect.go:68 +0xb1
github.com/docker/docker/testutil/environment.ProtectAll({0x12e8d98, 0xc00040e210}, {0x12f2878, 0xc0004fc340}, 0xc0001fac00)
	/go/src/github.com/docker/docker/testutil/environment/protect.go:45 +0xf3
github.com/docker/docker/integration/image.setupTest(0xc0004fc340)
	/go/src/github.com/docker/docker/integration/image/main_test.go:46 +0x59
```

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-16 14:45:26 +01:00
Paweł Gronowski
f0efb90f21
c8d/integration-cli: Skip tests checking ContainerConfig
Field is not supported with containerd integration and deprecated in
API 1.44

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-16 14:03:58 +01:00
Paweł Gronowski
2a02403a94
c8d/exporter: Use WithSkipMissing
Save the unmodified manifest list to keep the image ID of the
multi-platform images when not all platforms are present.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-16 13:06:43 +01:00
Sebastiaan van Stijn
a42726f866
Merge pull request #47078 from vvoland/docker-save-fix-layers-2
image/save: Derive Descriptor from actual tar archive
2024-01-16 12:55:55 +01:00
Sebastiaan van Stijn
b12cf05c82
Merge pull request #47076 from AkihiroSuda/rootlesskit-v2
Update RootlessKit to v2.0.0 (support Pasta)
2024-01-16 12:12:25 +01:00
Paweł Gronowski
9f50e06993
image/save: Refactor saveLayer
`diffIDPaths` is not used and can be removed.

`savedConfig` stores if the config was already saved (ID of the image is
the ID of the config).
`savedLayers` stores if the layer (diff ID) was already saved.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-16 11:14:15 +01:00
Akihiro Suda
b5a5ecf4a3
dockerd-rootless.sh: add comments about pasta
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2024-01-16 10:56:30 +09:00
Akihiro Suda
f5d05b43d7
Dockerfile: update RootlessKit to v2.0.0
https://github.com/rootless-containers/rootlesskit/releases/tag/v2.0.0

=== Pasta ===
RootlessKit v2 adds the support for pasta (https://passt.top/passt/).
Pasta is similar to slirp4netns but its port forwarder achieves better
throughput than slirp4netns port driver.

It is still not faster than RootlessKit's `builtin` port driver, but unlike the
`builtin` port driver, pasta can retain source IP address information.

Network driver | Port driver    | Net throughput | Port throughput | Src IP | No SUID | Note
---------------|----------------|----------------|-----------------|--------|---------|--------------------------------------------
slirp4netns    | builtin        | Slow           | Fast          |      |       | Default in typical setup
vpnkit         | builtin        | Slow           | Fast          |      |       | Default when slirp4netns is not installed
slirp4netns    | slirp4netns    | Slow           | Slow            |      |       |
**pasta**      | **implicit**   | Slow           | Fast          |      |       | Experimental
lxc-user-nic   | builtin        | Fast         | Slow            |      |       | Experimental
(bypass4netns) | (bypass4netns) | Fast         | Fast          |      |       | (Not integrated to RootlessKit)

=== Detach-netns ===
Aside from pasta, RootlessKit v2 also brings the support for
"detach-netns" mode, which leaves the runtime in the host network namespace to
eliminate the slirp overhead for pull/push and to allow accessing the "real"
127.0.0.1.

See containerd/nerdctl PR 2723 for how detach-netns is being adopted in
nerdctl v2.

Integrating detach-netns into Docker/Moby will need an extra work and will be
deferred to Docker v26 (or later).

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2024-01-16 10:56:29 +09:00
Akihiro Suda
ba86626642
vendor.mod: github.com/rootless-containers/rootlesskit/v2 v2.0.0
https://github.com/rootless-containers/rootlesskit/releases/tag/v2.0.0

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2024-01-16 10:56:27 +09:00
Cory Snider
312450d079 integration: test container healthcheck is reset
Update the TestDaemonRestartKilContainers integration test to assert
that a container's healthcheck status is always reset to the Starting
state after a daemon restart, even when the container is live-restored.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2024-01-15 15:30:22 -05:00
Paweł Gronowski
c58acf06c2 image/save: Derive Descriptor from actual tar archive
Layer size is the sum of the individual files count, not the tar
archive. Use the total bytes read returned by `io.Copy` to populate the
`Size` field.

Also set the digest to the actual digest of the tar archive.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-01-15 16:35:54 +01:00
Sebastiaan van Stijn
3eba4216e0
Merge pull request #47080 from thaJeztah/bump_mountinfo
vendor: github.com/moby/sys/mountinfo v0.7.1
2024-01-15 13:11:50 +01:00
Sebastiaan van Stijn
6d90f1168b
Merge pull request #47079 from thaJeztah/bump_x_sys
vendor: golang.org/x/sys v0.16.0
2024-01-15 13:11:35 +01:00
Sebastiaan van Stijn
c4d2968183
vendor: github.com/moby/sys/mountinfo v0.7.1
full diff: https://github.com/moby/sys/compare/mountinfo/v0.6.2...mountinfo/v0.7.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-15 12:09:02 +01:00
Sebastiaan van Stijn
075e1141b8
vendor: golang.org/x/sys v0.16.0
full diff: https://github.com/golang/sys/compare/v0.15.0...v0.16.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-15 12:05:29 +01:00
Sebastiaan van Stijn
87041b207f
Merge pull request #46980 from thaJeztah/bump_compare
vendor: github.com/google/go-cmp v0.6.0, github.com/google/uuid v1.5.0
2024-01-15 11:55:41 +01:00
Sebastiaan van Stijn
770271d98f
Merge pull request #47075 from rapsealk/docs/doc-comments-on-volume-mounts-mountpoint-spec
doc: Correct typo in doc comments for volume/mounts/mounts/MountPoint
2024-01-15 10:44:47 +01:00
Sebastiaan van Stijn
a759acd958
vendor: github.com/google/uuid v1.5.0
full diff: https://github.com/google/uuid/compare/v1.3.1...v1.5.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-15 10:40:15 +01:00
Sebastiaan van Stijn
3bcb01a84f
vendor: github.com/google/go-cmp v0.6.0
full diff: https://github.com/google/go-cmp/compare/v0.5.9...v0.6.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-15 10:40:13 +01:00
Jeongseok Kang
36dd2f029f doc: Correct typo in doc comments for volume/mounts/mounts/MountPoint
Signed-off-by: Jeongseok Kang <piono623@naver.com>
2024-01-15 09:25:09 +09:00
Sebastiaan van Stijn
608238ead6
Merge pull request #46979 from thaJeztah/move_mergo
vendor: containerd v1.7.12, and switch to dario.cat/mergo v1.0.0
2024-01-12 21:14:59 +01:00
Sebastiaan van Stijn
3a15c6c9d7
Merge pull request #47070 from thaJeztah/bump_containerd_binary
update containerd binary to v1.7.12
2024-01-12 21:14:07 +01:00
Sebastiaan van Stijn
8f89f10b40
Merge pull request #47068 from vvoland/docker-save-fix-layers
image/save: Fix missing layers in manifest
2024-01-12 19:39:56 +01:00
Sebastiaan van Stijn
6e2e7641f9
Merge pull request #47067 from thaJeztah/bump_buildx_compose
Dockerfile: update buildx to v0.12.1
2024-01-12 18:37:10 +01:00