Commit graph

48301 commits

Author SHA1 Message Date
Djordje Lukic
876d4e5484
c8d: cleanup imports in the image builder file
Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
2023-11-08 12:31:03 +01:00
Rachit Sharma
7995e3288f
Add until filter to docker image ls
Signed-off-by: Rachit Sharma <rachitsharma613@gmail.com>

add handling for multiple filters

Signed-off-by: Rachit Sharma <rachitsharma613@gmail.com>

Update integration/image/list_test.go

Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Rachit Sharma <rachitsharma613@gmail.com>

Add documentation of filter

Signed-off-by: Rachit Sharma <rachitsharma613@gmail.com>

Update integration/image/list_test.go

Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Rachit Sharma <rachitsharma613@gmail.com>

Fix bug with CommitOptions

Signed-off-by: Rachit Sharma <rachitsharma613@gmail.com>

add wrapping of text to 80 chars

Signed-off-by: Rachit Sharma <rachitsharma613@gmail.com>
2023-11-08 12:12:23 +01:00
Sebastiaan van Stijn
0c0943bcdf
Merge pull request #46783 from thaJeztah/fix_test
integration/networking: fix TestBridgeICC
2023-11-08 12:11:40 +01:00
Sebastiaan van Stijn
58785c2932
integration/networking: fix TestBridgeICC
This test broke in 98323ac114.

This commit renamed WithMacAddress into WithContainerWideMacAddress.
This helper sets the MacAddress field in container.Config. However, API
v1.44 now ignores this field if the NetworkMode has no matching entry in
EndpointsConfig.

This fix uses the helper WithMacAddress and specify for which
EndpointConfig the MacAddress is specified.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-11-08 10:23:24 +01:00
Sebastiaan van Stijn
c14694a424
Merge pull request #46779 from dmcgowan/c8d-default-auth-domain
Default the auth config domain to the target image domain
2023-11-07 16:44:31 +01:00
Sebastiaan van Stijn
49cea49cfa
Merge pull request #45905 from akerouanton/endpoint-specific-mac-address
api: Add a field MacAddress to EndpointSettings
2023-11-07 16:37:27 +01:00
Ken Bannister
6979503a85 Include port in URL for locally built Swagger docs
Signed-off-by: Ken Bannister <kb2ma@runbox.com>
2023-11-07 08:07:32 -05:00
Derek McGowan
755f008c1e
Default the auth config domain to the target image domain
When server address is not provided with the auth configuration,
use the domain from the image provided with the auth.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2023-11-06 17:02:18 -08:00
Sebastiaan van Stijn
796da163f9
Merge pull request #46778 from corhere/libc8d/revert-serialize-exec-starts-workaround
Revert "libcontainerd: work around exec start bug in c8d"
2023-11-06 21:12:39 +01:00
Cory Snider
7d9d601e6d project: document supported containerd versions
We only support containerd versions which have fixes for
https://github.com/containerd/containerd/issues/8557.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-11-06 13:43:18 -05:00
Cory Snider
29ac09ee9d Revert "libcontainerd: work around exec start bug in c8d"
The workaround is no longer required. The bug has been fixed in stable
versions of all supported containerd branches.

This reverts commit fb7ec1555c.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-11-06 13:26:44 -05:00
Sebastiaan van Stijn
c1a289e05a
docs/api: update redirect metadata for hugo
docs.docker.com switched from Jekyll to Hugo, which uses "aliases"
instead of "redirect_from".

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-11-06 18:16:47 +01:00
Sebastiaan van Stijn
02011af7b0
Merge pull request #46774 from elezar/bump-cdi-dependency
Update container-device-interface to v0.6.2
2023-11-04 13:27:17 +01:00
Evan Lezar
49e04102c1 Update container-device-interface to v0.6.2
This includes migrating from the github.com/container-orchestrated-devices
repo to tags.cncf.io.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2023-11-04 01:00:19 +01:00
Paweł Gronowski
3cb8e9526e
Merge pull request #46769 from vvoland/c8d-shared-mounter-impl
daemon/snapshotter: Align mounter implementations
2023-11-03 18:45:36 +01:00
Paweł Gronowski
26f63600c1
daemon/snapshotter: Align mounter implementations
Change the non-refcounted implementation to perform the mount using the
same identity and access right. They should be the same regardless if
we're refcounting or not.

This also allows to refactor refCountMounter into a mounter decorator.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-11-03 18:13:14 +01:00
Albin Kerouanton
6fd893bd3a
Merge pull request #46768 from akerouanton/dedicated-default-nw-config-struct
daemon/config: Put params for the default network into a dedicated struct
2023-11-03 17:03:05 +01:00
Albin Kerouanton
d5d41c2849
daemon/config: Put params for the default network into a dedicated struct
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-11-03 14:10:41 +01:00
Sebastiaan van Stijn
ed1a61dcb7
Merge pull request #46663 from akerouanton/ci-otel-windows
ci: Setup otel tracing for windows integration tests
2023-11-03 13:51:39 +01:00
Sebastiaan van Stijn
5b19725de2
Merge pull request #46668 from corhere/libn/svc-record-update-without-store
libnetwork: svc record update without store
2023-11-03 13:47:12 +01:00
Sebastiaan van Stijn
0ac748a340
Merge pull request #46628 from sgehrig/46621-container_wait
#46621 ensure that errors reading the response body are returned to the caller
2023-11-03 13:20:59 +01:00
Sebastiaan van Stijn
26c054edbf
Merge pull request #46531 from akerouanton/networking-suite-bridge-tests
integration: Add a new networking integration test suite
2023-11-03 12:33:44 +01:00
Sebastiaan van Stijn
39393f651e
Merge pull request #46766 from thaJeztah/fix_TestInfoAPI
integration/system: update TestInfoAPI to not use string-matching
2023-11-03 11:43:28 +01:00
Sebastiaan van Stijn
587ad8845a
integration/system: update TestInfoAPI to not use string-matching
This test was rewritten from an integration-cli test in commit
68d9beedbe, and originally implemented in
f4942ed864, which rewrote it from a unit-
test to an integration test.

Originally, it would check for the raw JSON response from the daemon, and
check for individual fields to be present in the output, but after commit
0fd5a65428, `client.Info()` was used, and
now the response is unmarshalled into a `system.Info`.

The remainder of the test remained the same in that rewrite, and as a
result were were now effectively testing if a `system.Info` struct,
when marshalled as JSON would show all the fields (surprise: it does).

TL;DR; the test would even pass with an empty `system.Info{}` struct,
which didn't provide much coverage, as it passed without a daemon:

    func TestInfoAPI(t *testing.T) {
        // always shown fields
        stringsToCheck := []string{
            "ID",
            "Containers",
            "ContainersRunning",
            "ContainersPaused",
            "ContainersStopped",
            "Images",
            "LoggingDriver",
            "OperatingSystem",
            "NCPU",
            "OSType",
            "Architecture",
            "MemTotal",
            "KernelVersion",
            "Driver",
            "ServerVersion",
            "SecurityOptions",
        }

        out := fmt.Sprintf("%+v", system.Info{})
        for _, linePrefix := range stringsToCheck {
            assert.Check(t, is.Contains(out, linePrefix))
        }
    }

This patch makes the test _slightly_ better by checking if the fields
are non-empty. More work is needed on this test though; currently it
uses the (already running) daemon, so it's hard to check for specific
fields to be correct (withouth knowing state of the daemon), but it's
not unlikely that other tests (partially) cover some of that. A TODO
comment was added to look into that (we should probably combine some
tests to prevent overlap, and make it easier to spot "gaps" as well).

While working on this, also moving the `SystemTime` into this test,
because that field is (no longer) dependent on "debug" state

(It is was actually this change that led me down this rabbit-hole)

                         ()_()
                         (-.-)
                        '(")(")'

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-11-03 10:17:15 +01:00
Albin Kerouanton
c1ab6eda4b
integration/networking: Test bridge ICC and INC
Following tests are implemented in this specific commit:

- Inter-container communications for internal and non-internal
  bridge networks, over IPv4 and IPv6.
- Inter-container communications using IPv6 link-local addresses for
  internal and non-internal bridge networks.
- Inter-network communications for internal and non-internal bridge
  networks, over IPv4 and IPv6, are disallowed.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-11-03 09:58:50 +01:00
Albin Kerouanton
409ea700c7
integration: Add a new networking integration test suite
This commit introduces a new integration test suite aimed at testing
networking features like inter-container communication, network
isolation, port mapping, etc... and how they interact with daemon-level
and network-level parameters.

So far, there's pretty much no tests making sure our networks are well
configured: 1. there're a few tests for port mapping, but they don't
cover all use cases ; 2. there're a few tests that check if a specific
iptables rule exist, but that doesn't prevent that specific iptables
rule to be wrong in the first place.

As we're planning to refactor how iptables rules are written, and change
some of them to fix known security issues, we need a way to test all
combinations of parameters. So far, this was done by hand, which is
particularly painful and time consuming. As such, this new test suite is
foundational to upcoming work.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-11-03 09:58:50 +01:00
Sebastiaan van Stijn
050e6066af
Merge pull request #46517 from rumpl/c8d-windows-pull-linux
c8d: test a backend dependent error on pull
2023-11-03 09:28:16 +01:00
Sebastiaan van Stijn
8068038a60
Merge pull request #46758 from rumpl/c8d-image-list-digest
Skip the busybox digest check when containerd is enabled
2023-11-03 09:16:48 +01:00
Sebastiaan van Stijn
90cc178baa
Merge pull request #46759 from rumpl/c8d-index-pull-test
Skip TestPullManifestList when using containerd
2023-11-03 09:14:42 +01:00
Sebastiaan van Stijn
5baf486545
Merge pull request #46764 from cpuguy83/fix_probe_interval
Fix case where health start interval is 0 uses default
2023-11-03 08:32:55 +01:00
Sebastiaan van Stijn
ec32f0db82
Merge pull request #46762 from akerouanton/seccomp-io_uring
seccomp: block io_uring_* syscalls in default profile
2023-11-02 21:10:29 +01:00
Brian Goff
02a932d63f Fix case where health start interval is 0 uses default
When the start interval is 0 we should treat that as unset.
This is especially important for older API versions where we reset the
value to 0.

Instead of using the default probe value we should be using the
configured `interval` value (which may be a default as well) which gives
us back the old behavior before support for start interval was added.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-11-02 20:02:16 +00:00
Brian Goff
2f138d860e
Merge pull request #46685 from rumpl/c8d-tag-does-not-exist
c8d: Return the "tag does not exist error"
2023-11-02 12:33:36 -07:00
Brian Goff
d3f4876f41
Merge pull request #46751 from thaJeztah/fix_TestSaveRepoWithMultipleImages_flaky
integration: make TestSaveRepoWithMultipleImages less flaky
2023-11-02 12:32:07 -07:00
Albin Kerouanton
891241e7e7
seccomp: block io_uring_* syscalls in default profile
This syncs the seccomp profile with changes made to containerd's default
profile in [1].

The original containerd issue and PR mention:

> Security experts generally believe io_uring to be unsafe. In fact
> Google ChromeOS and Android have turned it off, plus all Google
> production servers turn it off. Based on the blog published by Google
> below it seems like a bunch of vulnerabilities related to io_uring can
> be exploited to breakout of the container.
>
> [2]
>
> Other security reaserchers also hold this opinion: see [3] for a
> blackhat presentation on io_uring exploits.

For the record, these syscalls were added to the allowlist in [4].

[1]: a48ddf4a20
[2]: https://security.googleblog.com/2023/06/learnings-from-kctf-vrps-42-linux.html
[3]: https://i.blackhat.com/BH-US-23/Presentations/US-23-Lin-bad_io_uring.pdf
[4]: https://github.com/moby/moby/pull/39415

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2023-11-02 19:05:47 +01:00
Djordje Lukic
3c2b06099d
Skip TestPullManifestList when using containerd
This test is very weird, the Size in the manifests that it creates is
wrong, graph drivers only print a warning in that case but containerd
fails because it verifies more things. The media types are also wrong in
the containerd case, the manifest list forces the media type to be
"schema2.MediaTypeManifest" but in the containerd case the media type is
an OCI one.

Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
2023-11-02 16:09:35 +01:00
Djordje Lukic
6ca1b9e4ce
Skip the busybox digest check when containerd is enabled
We always have a digest with containerd.

Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
2023-11-02 14:07:49 +01:00
Sebastiaan van Stijn
fb3cc5e716
Merge pull request #46755 from corhere/libn/netip-overlaps
libnetwork/ipam: refactor prefix-overlap checks
2023-11-02 13:33:40 +01:00
Sebastiaan van Stijn
217054ddea
Merge pull request #46754 from Frankkkkk/master
builder/dockerfile: errmsg: quote build target
2023-11-02 12:40:22 +01:00
Frank Villaro-Dixon
123ade3763 builder/dockerfile: errmsg: quote build target
The build target is not quoted and it makes it difficult for some
persons to see what the problem is.

By quoting it we emphasize that the target name is variable.

Signed-off-by: Frank Villaro-Dixon <frank.villarodixon@merkle.com>
2023-11-02 09:38:14 +01:00
Cory Snider
7257c77e19 libnetwork/ipam: refactor prefix-overlap checks
I am finally convinced that, given two netip.Prefix values a and b, the
expression

    a.Contains(b.Addr()) || b.Contains(a.Addr())

is functionally equivalent to

    a.Overlaps(b)

The (netip.Prefix).Contains method works by masking the address with the
prefix's mask and testing whether the remaining most-significant bits
are equal to the same bits in the prefix. The (netip.Prefix).Overlaps
method works by masking the longer prefix to the length of the shorter
prefix and testing whether the remaining most-significant bits are
equal. This is equivalent to
shorterPrefix.Contains(longerPrefix.Addr()), therefore applying Contains
symmetrically to two prefixes will always yield the same result as
applying Overlaps to the two prefixes in either order.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-11-01 11:44:24 -04:00
Sebastiaan van Stijn
60e88c873e
Merge pull request #46736 from corhere/robust-tarsum
pkg/tarsum: handle xattrs like archive/tar does
2023-10-31 19:52:47 +01:00
Sebastiaan van Stijn
d9dce8e0d0
Merge pull request #46618 from vvoland/c8d-pull-all-tags-2
c8d/pull: Handle pull all tags (2nd approach)
2023-10-31 16:45:18 +01:00
Sebastiaan van Stijn
4be5b4147d
Merge pull request #46748 from tonistiigi/containerd-platforms-normalize
builder-next: make sure worker platforms normalized for containerd
2023-10-31 12:38:09 +01:00
Sebastiaan van Stijn
4e8ba395f2
integration: TestSaveRepoWithMultipleImages: minor cleanup
- use consts for fixed values
- remove redundant `cmp.Nil(err)`

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-31 10:54:57 +01:00
Sebastiaan van Stijn
0dae0f2b5e
integration: TestSaveRepoWithMultipleImages remove redundant remove
This delete was originally added in b37fdc5dd1
and migrated from `deleteImages(repoName)` in commit 1e55ace875,
however, deleting `foobar-save-multi-images-test` (`foobar-save-multi-images-test:latest`)
always resulted in an error;

    Error response from daemon: No such image: foobar-save-multi-images-test:latest

This patch removes the redundant image delete.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-31 10:52:51 +01:00
Sebastiaan van Stijn
30cd8b8fca
integration: make TestSaveRepoWithMultipleImages less flaky
Shutting down containers on Windows can take a long time (with hyper-v),
causing this test to be flaky; seen failing on windows 2022;

    === FAIL: github.com/docker/docker/integration/image TestSaveRepoWithMultipleImages (23.16s)
        save_test.go:104: timeout waiting for container to exit

Looking at the test, we run a container only to commit it, and the test
does not make changes to the container's filesystem; it only runs a container
with a custom command (`true`).

Instead of running the container, we can _create_ a container and commit it;
this simplifies the tests, and prevents having to wait for the container to
exit (before committing).

To verify:

    make BIND_DIR=. DOCKER_GRAPHDRIVER=vfs TEST_FILTER=TestSaveRepoWithMultipleImages test-integration

    INFO: Testing against a local daemon
    === RUN   TestSaveRepoWithMultipleImages
    --- PASS: TestSaveRepoWithMultipleImages (1.20s)
    PASS

    DONE 1 tests in 2.668s

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-31 10:32:02 +01:00
Sebastiaan van Stijn
1c50526092
daemon: improve daemon start informational log message
When starting a daemon in debug mode (such as used in CI), many log-messages
are printed during startup. As a result, the log message indicating whether
graph-drivers or snapshotters are used may appear far separate from the
informational log about the daemon (and selected storage-driver).

The existing log-driver also unconditionally uses the legacy "graph-driver"
terminology, instead of the more generic "storage-driver".

This patch changes the log message shown during startup to use the generic
"graph-driver" as field, and adds a new field that indicates wheter we're
using snapshotters or graph-drivers.

Given that snapshotters will be the default at some point, an alternative
could be to include the _type_ of driver used, for example;
`io.containerd.snapshotter.v1`, which may continue to be relevant after
snapshotters become the default, and at which point (potentially) the
type of snapshotter becomes more relevant.

Before this change:

    TEST_INTEGRATION_USE_SNAPSHOTTER=1 DOCKER_GRAPHDRIVER=overlayfs dockerd
    ...
    INFO[2023-10-31T09:12:33.586269801Z] Starting daemon with containerd snapshotter integration enabled
    INFO[2023-10-31T09:12:33.586322176Z] Loading containers: start.
    INFO[2023-10-31T09:12:33.640514759Z] Loading containers: done.
    INFO[2023-10-31T09:12:33.646498134Z] Docker daemon                                 commit=dcf7287d647bcb515015e389df46ccf1e09855b7 graphdriver=overlayfs version=dev
    INFO[2023-10-31T09:12:33.646706551Z] Daemon has completed initialization
    INFO[2023-10-31T09:12:33.658840592Z] API listen on /var/run/docker.sock

With this change;

    TEST_INTEGRATION_USE_SNAPSHOTTER=1 DOCKER_GRAPHDRIVER=overlayfs dockerd
    ...
    INFO[2023-10-31T08:41:38.841155928Z] Starting daemon with containerd snapshotter integration enabled
    INFO[2023-10-31T08:41:38.841207512Z] Loading containers: start.
    INFO[2023-10-31T08:41:38.902461053Z] Loading containers: done.
    INFO[2023-10-31T08:41:38.910535137Z] Docker daemon                                 commit=dcf7287d647bcb515015e389df46ccf1e09855b7 containerd-snapshotter=true storage-driver=overlayfs version=dev
    INFO[2023-10-31T08:41:38.910936803Z] Daemon has completed initialization

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-31 10:20:27 +01:00
Tonis Tiigi
a99bb24ea8
builder-next: make sure worker platforms normalized for containerd
These platforms are filled by default from containerd
introspection API and may not be normalized. Initializing
wrong platform in here results in incorrect platform
for BUILDPLATFORM and TARGETPLATFORM build-args for
Dockerfile frontend (and probably other side effects).

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-10-30 23:14:34 -07:00
Sebastiaan van Stijn
e9efc0a361
Merge pull request #46741 from vvoland/c8d-push-notags
c8d/push: Return error when repository has no tags
2023-10-30 20:08:31 +01:00