Commit graph

46152 commits

Author SHA1 Message Date
Paweł Gronowski
e5fbc3f75a
hack/cli.sh: Quiet origin cleanup
Don't show `error: No such remote: 'origin'` error when building for the
first time and the cached git repository doesn't a remote yet.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 5433b88e2d)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-05 17:03:25 +02:00
Paweł Gronowski
1a078977e1
Dockerfile/shell: Install buildx cli plugin
Installs the buildx cli plugin in the container shell by default.
Previously user had to manually download the buildx binary to use
buildkit.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 49f76a34b5)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-05 16:47:18 +02:00
Paweł Gronowski
c4198e6053
Dockerfile: Use separate cli for shell and integration-cli
Use separate cli for integration-cli to allow use newer CLI for
interactive dev shell usage.

Both versions can be overriden with DOCKERCLI_VERSION or
DOCKERCLI_INTEGRATION_VERSION. Binary is downloaded from
download.docker.com if it's available, otherwise it's built from the
source.

For backwards compatibility DOCKER_CLI_PATH overrides BOTH clis.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 17c99f7164)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-05 16:47:16 +02:00
Bjorn Neergaard
8e70a1b23e
Merge pull request #45692 from vvoland/builder-use-moby-exporter-by-default-24
[backport 24.0] builder-next: Set moby exporter as default
2023-06-02 09:22:02 -06:00
Bjorn Neergaard
c671434cd2
Merge pull request #45688 from vvoland/c8d-load-unpack-attestation-24
[24.0 backport] c8d/load: Don't unpack pseudo images
2023-06-02 08:28:20 -06:00
Paweł Gronowski
647ba03224
builder-next: Set moby exporter as default
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit d63569c73d)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-02 16:06:11 +02:00
Sebastiaan van Stijn
2f65bb7bb5
Merge pull request #45685 from thaJeztah/24.0_backport_dockerfile_more_link
[24.0 backport] Dockerfile: use COPY --link for source code as well
2023-06-02 15:12:05 +02:00
Paweł Gronowski
961fe27408
c8d/handlers: Handle error in walkPresentChildren
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 4295806736)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-02 15:09:28 +02:00
Paweł Gronowski
087cf6f238
c8d/load: Don't unpack pseudo images
Don't unpack image manifests which are not a real images that can't be
unpacked.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 4d3238dc0b)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-02 15:09:25 +02:00
Paweł Gronowski
0b9d68f59d
c8d/load: Use walkImageManifests
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit b08bff8ba3)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-02 15:09:24 +02:00
Paweł Gronowski
cbf0779bfc
c8d/list: Use walkImageManifests
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 5210f48bfc)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-02 15:09:22 +02:00
Paweł Gronowski
0139309fef
c8d: Add walkImageManifests and ImageManifest wrapper
The default implementation of the containerd.Image interface provided by
the containerd operates on the parent index/manifest list of the image
and the platform matcher.

This isn't convenient when a specific manifest is already known and it's
redundant to search the whole index for a manifest that matches the
given platform matcher. It can also result in a different manifest
picked up than expected when multiple manifests with the same platform
are present.

This introduces a walkImageManifests which walks the provided image and
calls a handler with a ImageManifest, which is a simple wrapper that
implements containerd.Image interfaces and performs all containerd.Image
operations against a platform specific manifest instead of the root
manifest list/index.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit fabc1d5bef)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-06-02 15:09:20 +02:00
Sebastiaan van Stijn
18278d3dc1
Merge pull request #45684 from thaJeztah/24.0_backport_c8d_useragent_more_details
[24.0 backport] containerd: add c8d version and storage-driver to User-Agent
2023-06-02 14:35:21 +02:00
Sebastiaan van Stijn
e1c7956764
Dockerfile: use COPY --link for source code as well
I missed the most important COPY in 637ca59375

Copying the source code into the dev-container does not depend on the parent
layers, so can use the --link option as well.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ff2342154b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-02 13:22:05 +02:00
Sebastiaan van Stijn
4b3329d3dd
Merge pull request #45678 from thaJeztah/24.0_backport_fix_snapshotter_change
[24.0 backport] Make sure the image is unpacked for the current snapshotter
2023-06-02 13:17:54 +02:00
Sebastiaan van Stijn
32d442aee1
Merge pull request #45672 from thaJeztah/24.0_backport_bump_zfs
[24.0 backport] vendor: github.com/mistifyio/go-zfs/v3 v3.0.1
2023-06-02 11:48:36 +02:00
Sebastiaan van Stijn
75afe3201b
containerd: add c8d version and storage-driver to User-Agent
With this patch, the user-agent has information about the containerd-client
version and the storage-driver that's used when using the containerd-integration;

    time="2023-06-01T11:27:07.959822887Z" level=info msg="listening on [::]:5000" go.version=go1.19.9 instance.id=53590f34-096a-4fd1-9c58-d3b8eb7e5092 service=registry version=2.8.2
    ...
    172.18.0.1 - - [01/Jun/2023:11:30:12 +0000] "HEAD /v2/multifoo/blobs/sha256:c7ec7661263e5e597156f2281d97b160b91af56fa1fd2cc045061c7adac4babd HTTP/1.1" 404 157 "" "docker/dev go/go1.20.4 git-commit/8d67d0c1a8 kernel/5.15.49-linuxkit-pr os/linux arch/arm64 containerd-client/1.6.21+unknown storage-driver/overlayfs UpstreamClient(Docker-Client/24.0.2 \\(linux\\))"

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d099e47e00)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-02 11:41:10 +02:00
Sebastiaan van Stijn
8018ee4689
dockerversion: DockerUserAgent(): allow custom versions to be passed
Allow additional metadata to be passed as part of the generated User-Agent.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a6da1480b5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-02 11:41:09 +02:00
Sebastiaan van Stijn
ed376a603f
dockerversion: remove insertUpstreamUserAgent()
It was not really "inserting" anything, just formatting and appending.
Simplify this by changing this in to a `getUpstreamUserAgent()` function
which returns the upstream User-Agent (if any) into a `UpstreamClient()`.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9a1f2e6d7c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-02 11:41:09 +02:00
Sebastiaan van Stijn
1d45ea52f4
dockerversion: simplify escapeStr()
Use a const for the characters to escape, instead of implementing
this as a generic escaping function.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ff40d2d787)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-02 11:41:09 +02:00
Sebastiaan van Stijn
a27b0381a6
dockerversion: add a basic unit-test
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit eb9a5392bc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-02 11:41:09 +02:00
Djordje Lukic
1fc19772e0
Make sure the image is unpacked for the current snapshotter
Switching snapshotter implementations would result in an error when
preparing a snapshot, check that the image is indeed unpacked for the
current snapshot before trying to prepare a snapshot.

Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
(cherry picked from commit ed32f5e241)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-01 22:50:53 +02:00
Brian Goff
9bdb6adf92
Merge pull request #45673 from thaJeztah/24.0_backport_consistent_alias
[24.0 backport] fixing consistent aliases for OCI spec imports
2023-06-01 12:34:00 -07:00
Sebastiaan van Stijn
7dbab75fec
Merge pull request #45674 from thaJeztah/24.0_backport_dockerfile_copy_link
[24.0 backport] Dockerfile: use COPY --link to copy artifacts from build-stages
2023-06-01 18:37:14 +02:00
Sebastiaan van Stijn
e7b1501832
Merge pull request #45671 from thaJeztah/24.0_backport_c8d_useragent
[24.0 backport] containerd: set user-agent when pushing/pulling images
2023-06-01 18:36:46 +02:00
Sebastiaan van Stijn
4217d9ea0a
Dockerfile: use COPY --link to copy artifacts from build-stages
Build-cache for the build-stages themselves are already invalidated if the
base-images they're using is updated, and the COPY operations don't depend
on previous steps (as there's no overlap between artifacts copied).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 637ca59375)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-01 17:29:46 +02:00
Jeyanthinath Muthuram
4c6b8e737f
added alias validation
Signed-off-by: Jeyanthinath Muthuram <jeyanthinath10@gmail.com>
(cherry picked from commit 71d7908656)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-01 17:23:07 +02:00
Jeyanthinath Muthuram
e370f224ae
fixing consistent aliases for OCI spec imports
Signed-off-by: Jeyanthinath Muthuram <jeyanthinath10@gmail.com>
(cherry picked from commit 307b09e7eb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-01 17:22:49 +02:00
Sebastiaan van Stijn
ac1a867282
vendor: github.com/mistifyio/go-zfs/v3 v3.0.1
Switching to the v3 version, which was renamed to be compatible with
go modules.

Full diff: https://github.com/mistifyio/go-zfs/compare/f784269be439...v3.0.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 86108812b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-01 17:14:33 +02:00
Sebastiaan van Stijn
2949fee1d3
containerd: set user-agent when pushing/pulling images
Before this, the client would report itself as containerd, and the containerd
version from the containerd go module:

    time="2023-06-01T09:43:21.907359755Z" level=info msg="listening on [::]:5000" go.version=go1.19.9 instance.id=67b89d83-eac0-4f85-b36b-b1b18e80bde1 service=registry version=2.8.2
    ...
    172.18.0.1 - - [01/Jun/2023:09:43:33 +0000] "HEAD /v2/multifoo/blobs/sha256:cb269d7c0c1ca22fb5a70342c3ed2196c57a825f94b3f0e5ce3aa8c55baee829 HTTP/1.1" 404 157 "" "containerd/1.6.21+unknown"

With this patch, the user-agent has the docker daemon information;

    time="2023-06-01T11:27:07.959822887Z" level=info msg="listening on [::]:5000" go.version=go1.19.9 instance.id=53590f34-096a-4fd1-9c58-d3b8eb7e5092 service=registry version=2.8.2
    ...
    172.18.0.1 - - [01/Jun/2023:11:27:20 +0000] "HEAD /v2/multifoo/blobs/sha256:c7ec7661263e5e597156f2281d97b160b91af56fa1fd2cc045061c7adac4babd HTTP/1.1" 404 157 "" "docker/dev go/go1.20.4 git-commit/8d67d0c1a8 kernel/5.15.49-linuxkit-pr os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.2 \\(linux\\))"

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 66137ae429)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-06-01 17:10:08 +02:00
Bjorn Neergaard
7861aa7e80
Merge pull request #45659 from corhere/backport-24.0/libn/setup-resolver-with-verbose-iptables
[24.0 backport] libnetwork: fix resolver restore w/ chatty 'iptables -C'
2023-05-30 15:31:12 -06:00
Bjorn Neergaard
ebe29481ec
Merge pull request #45658 from corhere/backport-24.0/libn/fix-embedded-resolver-live-reload
[24.0 backport] libnetwork: fix sandbox restore
2023-05-30 15:30:39 -06:00
Cory Snider
f9c68e5fbc libn: fix resolver restore w/ chatty 'iptables -C'
Resolver.setupIPTable() checks whether it needs to flush or create the
user chains used for NATing container DNS requests by testing for the
existence of the rules which jump to said user chains. Unfortunately it
does so using the IPTable.RawCombinedOutputNative() method, which
returns a non-nil error if the iptables command returns any output even
if the command exits with a zero status code. While that is fine with
iptables-legacy as it prints no output if the rule exists, iptables-nft
v1.8.7 prints some information about the rule. Consequently,
Resolver.setupIPTable() would incorrectly think that the rule does not
exist during container restore and attempt to create it. This happened
work work by coincidence before 8f5a9a741b
because the failure to create the already-existing table would be
ignored and the new NAT rules would be inserted before the stale rules
left in the table from when the container was last started/restored. Now
that failing to create the table is treated as a fatal error, the
incompatibility with iptables-nft is no longer hidden.

Switch to using IPTable.ExistsNative() to test for the existence of the
jump rules as it correctly only checks the iptables command's exit
status without regard for whether it outputs anything.

Signed-off-by: Cory Snider <csnider@mirantis.com>
(cherry picked from commit 1178319313)
Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-05-30 15:49:17 -04:00
Cory Snider
3452a76589 libnetwork: fix sandbox restore
The method to restore a network namespace takes a collection of
interfaces to restore with the options to apply. The interface names are
structured data, tuples of (SrcName, DstPrefix) but for whatever reason
are being passed into Restore() serialized to strings. A refactor,
f0be4d126d, accidentally broke the
serialization by dropping the delimiter. Rather than fix the
serialization and leave the time-bomb for someone else to trip over,
pass the interface names as structured data.

Signed-off-by: Cory Snider <csnider@mirantis.com>
(cherry picked from commit 50eb2d2782)
Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-05-30 15:47:20 -04:00
Cory Snider
fec801a103 libnetwork: log why osl sandbox restore failed
Signed-off-by: Cory Snider <csnider@mirantis.com>
(cherry picked from commit 18bf3aa442)
Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-05-30 15:47:20 -04:00
Akihiro Suda
143a25144a
Merge pull request #45643 from thaJeztah/24.0_backport_exec_npe
[24.0 backport] Fix npe in exec resize when exec errored
2023-05-29 09:54:07 +09:00
Akihiro Suda
f5899cc1f6
Merge pull request #45633 from thaJeztah/24.0_backport_deprecate_builder_streaming
[24.0 backport] builder/remotecontext: deprecate CachableSource, NewCachableSource
2023-05-29 09:53:47 +09:00
Brian Goff
d9e39914a7
Fix npe in exec resize when exec errored
In cases where an exec start failed the exec process will be nil even
though the channel to signal that the exec started was closed.

Ideally ExecConfig would get a nice refactor to handle this case better
(ie. it's not started so don't close that channel).
This is a minimal fix to prevent NPE. Luckilly this would only get
called by a client and only the http request goroutine gets the panic
(http lib recovers the panic).

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 487ea81316)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-28 19:06:35 +02:00
Sebastiaan van Stijn
0a59892a88
Merge pull request #45637 from corhere/backport-24.0/libn/fix-encrypted-overlay-nonstandard-port
[24.0 backport] libnetwork/d/overlay: support encryption on any port
2023-05-27 00:45:32 +02:00
Cory Snider
042f0799db libn/d/overlay: support encryption on any port
While the VXLAN interface and the iptables rules to mark outgoing VXLAN
packets for encryption are configured to use the Swarm data path port,
the XFRM policies for actually applying the encryption are hardcoded to
match packets with destination port 4789/udp. Consequently, encrypted
overlay networks do not pass traffic when the Swarm is configured with
any other data path port: encryption is not applied to the outgoing
VXLAN packets and the destination host drops the received cleartext
packets. Use the configured data path port instead of hardcoding port
4789 in the XFRM policies.

Signed-off-by: Cory Snider <csnider@mirantis.com>
(cherry picked from commit 9a692a3802)
Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-05-26 16:41:42 -04:00
Sebastiaan van Stijn
ec8ec9056c
builder/remotecontext: deprecate CachableSource, NewCachableSource
This type (as well as TarsumBackup), was used for the experimental --stream
support for the classic builder. This feature was removed in commit
6ca3ec88ae, which also removed uses of
the CachableSource type.

As far as I could find, there's no external consumers of these types,
but let's deprecated it, to give potential users a heads-up that it
will be removed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 37d4b0bee9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-26 15:13:27 +02:00
Sebastiaan van Stijn
659604f9ee
Merge pull request #45625 from thaJeztah/24.0_backport_serialize_exec_starts_workaround
[24.0 backport] libcontainerd: work around exec start bug in c8d
2023-05-25 23:22:22 +02:00
Sebastiaan van Stijn
6660133ffb
Merge pull request #45582 from thaJeztah/24.0_backport_vendor_buildkit_0.11.7_dev
[24.0 backport] vendor: github.com/moby/buildkit v0.11.7-0.20230525183624-798ad6b0ce9f
2023-05-25 23:03:09 +02:00
Sebastiaan van Stijn
67b3563d09
Merge pull request #45623 from vvoland/c8d-inspect-created-time-24
[24.0 backport] c8d/inspect: Fill `Created` time if available
2023-05-25 22:22:34 +02:00
Cory Snider
7a4ea19803
libcontainerd: work around exec start bug in c8d
It turns out that the unnecessary serialization removed in
b75246202a happened to work around a bug
in containerd. When many exec processes are started concurrently in the
same containerd task, it takes seconds to minutes for them all to start.
Add the workaround back in, only deliberately this time.

Signed-off-by: Cory Snider <csnider@mirantis.com>
(cherry picked from commit fb7ec1555c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-25 22:18:14 +02:00
Sebastiaan van Stijn
ae6e9333c0
vendor: github.com/moby/buildkit v0.11.7-0.20230525183624-798ad6b0ce9f
full diff: https://github.com/moby/buildkit/compare/v0.11.6...798ad6b0ce9f2fe86dfb2b0277e6770d0b545871

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 79ca6630d4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-25 21:45:42 +02:00
Paweł Gronowski
0d9acd24fe c8d/inspect: Fill Created time if available
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit b9b8b6597a)
2023-05-25 21:28:22 +02:00
Bjorn Neergaard
37bc639704
Merge pull request #45620 from thaJeztah/24.0_backport_update_go_runc_v1.1.0
[24.0 backport] vendor: github.com/containerd/go-runc v1.1.0
2023-05-25 13:14:38 -06:00
Sebastiaan van Stijn
04eccf8165
vendor: github.com/containerd/go-runc v1.1.0
full diff: https://github.com/containerd/go-runc/compare/v1.0.0...v1.1.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3512b04093)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-25 19:01:26 +02:00
Sebastiaan van Stijn
24722779ff
Merge pull request #45616 from thaJeztah/24.0_backport_lock_in_snapshotter_setting
[24.0 backport] daemon: lock in snapshotter setting at daemon init
2023-05-25 15:49:53 +02:00