Commit graph

46512 commits

Author SHA1 Message Date
Paweł Gronowski
c35376c455
volume/local: Don't unmount, restore mounted status
On startup all local volumes were unmounted as a cleanup mechanism for
the non-clean exit of the last engine process.

This caused live-restored volumes that used special volume opt mount
flags to be broken. While the refcount was restored, the _data directory
was just unmounted, so all new containers mounting this volume would
just have the access to the empty _data directory instead of the real
volume.

With this patch, the mountpoint isn't unmounted. Instead, if the volume
is already mounted, just mark it as mounted, so the next time Mount is
called only the ref count is incremented, but no second attempt to mount
it is performed.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 2689484402)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-29 19:59:05 +02:00
Paweł Gronowski
5d4cc0b5b5
integration/liveRestore: Check volume content
Make sure that the content in the live-restored volume mounted in a new
container is the same as the content in the old container.
This checks if volume's _data directory doesn't get unmounted on
startup.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit aef703fa1b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-29 19:58:57 +02:00
Sebastiaan van Stijn
c78abd96ce
Merge pull request #46330 from thaJeztah/24.0_backport_api_docs_update_urls
[24.0 backport] docs: update links to Go documentation
2023-08-25 17:44:23 +02:00
Sebastiaan van Stijn
6282d95b9e
Merge pull request #46331 from thaJeztah/24.0_backport_update_golangci_lint
[24.0 backport] update golangci-lint to v1.54.2
2023-08-25 17:27:59 +02:00
Sebastiaan van Stijn
1d983e2e8a
update golangci-lint to v1.54.2
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit cd49f9affd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-25 16:24:36 +02:00
Sebastiaan van Stijn
d2e9a19358
CONTRIBUTING.md: update links to golang docs and blog
- docs moved to https://go.dev/doc/
- blog moved to https://go.dev/blog/

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b18e170631)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-25 16:22:35 +02:00
Sebastiaan van Stijn
73f6053bb3
api: swagger: update link to Go documentation
Go documentation moved to the `go.dev` domain;

    curl -sI https://golang.org/doc/install/source#environment | grep 'location'
    location: https://go.dev/doc/install/source

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 136e86bb5c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-25 16:22:35 +02:00
Sebastiaan van Stijn
de13951b9d
docs/api: update links to Go documentation
Go documentation moved to the `go.dev` domain;

    curl -sI https://golang.org/doc/install/source#environment | grep 'location'
    location: https://go.dev/doc/install/source

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4862d39144)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-25 16:22:32 +02:00
Sebastiaan van Stijn
7741a89966
Merge pull request #46325 from thaJeztah/24.0_backport_hack_less_redirects
[24.0 backport] Dockerfile: Windows: update Golang download domains to cut down redirects
2023-08-25 14:48:40 +02:00
Sebastiaan van Stijn
377af4c9b4
Dockerfile: Windows: update Golang download domains to cut down redirects
The `golang.org` domain moved to `go.dev`, and the download-URLs we were
using resulted in 2 redirects;

    curl -sI https://golang.org/dl/go1.20.windows-amd64.zip | grep 'location'
    location: https://go.dev/dl/go1.20.windows-amd64.zip

    curl -sI https://go.dev/dl/go1.20.windows-amd64.zip | grep 'location'
    location: https://dl.google.com/go/go1.20.windows-amd64.zip

    curl -sI https://dl.google.com/go/go1.20.windows-amd64.zip
    HTTP/2 200
    # ...

This patch cuts it down to one redirects. I decided not to use the "final"
(`dl.google.com`) URL, because that URL is not documented in the Golang docs,
and visiting the domain itself (https://dl.google.com/) redirects to a marketing
page for "Google Chrome".

Trying the `/go/` path (https://dl.google.com/go/) also does not show a landing
page that lists downloads, so I'm considering those URLs to be "unstable".

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f6a5318f94)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-25 13:32:24 +02:00
Sebastiaan van Stijn
088cec8f0f
hack: update link to GOPATH documentation
This documentation moved to a different page, and the Go documentation
moved to the https://go.dev/ domain.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2aabd64477)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-25 13:32:10 +02:00
Sebastiaan van Stijn
8ff9ef2a7a
Merge pull request #46310 from vvoland/c8d-legacybuilder-fix-layermismatch-24
[24.0 backport] c8d/legacybuilder: Fix `mismatched image rootfs` errors
2023-08-24 15:03:28 +02:00
Paweł Gronowski
ed2f5d1d85
c8d/builder: Don't drop fields from created image
Previous image created a new partially filled image.
This caused child images to lose their parent's layers.

Instead of creating a new object and trying to replace its fields, just
clone the original passed image and change its ID to the manifest
digest.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 01214bafd2)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-24 12:47:56 +02:00
Djordje Lukic
aade22d31e
Merge pull request #46302 from thaJeztah/24.0_backport_c8d-legacybuilder-fix-from-scratch
[24.0 backport] c8d: Fix building Dockerfiles that have `FROM scratch`
2023-08-23 20:58:41 +02:00
Paweł Gronowski
1d10e8633d
daemon: Handle NotFound when deleting container lease
If the lease doesn't exit (for example when creating the container
failed), just ignore the not found error.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit bedcc94de4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-23 18:59:50 +02:00
Paweł Gronowski
63422515ba
c8d/run: Allow running container without image
This allows the legacy builder to apply changes to the `FROM scratch`
layer.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit dfaff9598c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-23 18:59:43 +02:00
Paweł Gronowski
49671250f6
c8d/commit: Don't produce an empty layer
If the diff is empty and don't produce an empty layer.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit eb56493f4e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-23 18:59:27 +02:00
Sebastiaan van Stijn
e0661af25c
Merge pull request #46300 from thaJeztah/24.0_backport_fix-platform-check
[24.0 backport] Don't return an error if the lease is not found
2023-08-23 17:41:14 +02:00
Djordje Lukic
b83f5a89f4
Don't return an error if the lease is not found
If the image for the wanted platform doesn't exist then the lease
doesn't exist either. Returning this error hides the real error, so
let's not return it.

Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
(cherry picked from commit b8ff8ea58e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-23 15:55:48 +02:00
Bjorn Neergaard
a7cc790265
Merge pull request #46289 from thaJeztah/24.0_backport_ci-bin-image-repo-origin
[24.0 backport] ci(bin-image): GHA and metadata improvements
2023-08-23 06:23:46 -06:00
Sebastiaan van Stijn
20130006b7
Merge pull request #46286 from thaJeztah/24.0_backport_gha_report_timeout
[24.0 backport] gha: set 10-minute timeout on "report" actions
2023-08-22 00:02:56 +02:00
Bjorn Neergaard
e6c959b172
Merge pull request #46290 from thaJeztah/24.0_backport_v1_deprecation
[24.0 backport] distribution: update warning for deprecated image formats
2023-08-21 14:12:39 -06:00
Bjorn Neergaard
4ac2355d62
hack: use long SHA for DOCKER_GITCOMMIT
This better aligns to GHA/CI settings, and is in general a better
practice in the year 2023.

We also drop the 'unsupported' fallback for `git rev-parse` in the
Makefile; we have a better fallback behavior for an empty
DOCKER_GITCOMMIT in `hack/make.sh`.

Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
(cherry picked from commit d125823d3f)
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-08-21 14:02:56 -06:00
Bjorn Neergaard
ac2a80fcc3
ci(bin-image): clean up metadata
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
(cherry picked from commit 2010f4338e)
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-08-21 14:02:55 -06:00
Bjorn Neergaard
5eef5a7f59
ci(bin-image): clean up env var handling
There are still messy special cases (e.g. DOCKER_GITCOMMIT vs VERSION),
but this makes things a little easier to follow, as we keep
GHA-specifics in the GHA files.

Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
(cherry picked from commit ad91fc1b00)
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-08-21 14:02:54 -06:00
CrazyMax
aaf84dd4cf
remove Dockerfile.e2e
Dockerfile.e2e is not used anymore. Integration tests run
through the main Dockerfile.

Also removes the daemon OS/Arch detection script that is not
necessary anymore. It was used to select the Dockerfile based
on the arch like Dockerfile.arm64 but we don't have those
anymore. Was also used to check referenced frozen images
in the Dockerfile.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 5efe72415d)
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-08-21 14:02:49 -06:00
Sebastiaan van Stijn
a99e62fa3d
distribution: show image schema deprecation on all registries
When we added this deprecation warning, some registries had not yet
moved away from the deprecated specification, so we made the warning
conditional for pulling from Docker Hub.

That condition was added in 647dfe99a5,
which is over 4 Years ago, which should be time enough for images
and registries to have moved to current specifications.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8c4af5dacb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-21 21:06:29 +02:00
Sebastiaan van Stijn
e239799583
distribution: update warning for deprecated image formats
- Use the same warning for both "v1 in manifest-index" and bare "v1" images.
- Update URL to use a "/go/" redirect, which allows the docs team to more
  easily redirect the URL to relevant docs (if things move).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 982bc0e228)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-21 21:05:16 +02:00
CrazyMax
bb22b8a418
ci(bin-image): check repo origin
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 219d4d9db9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-21 20:52:01 +02:00
Sebastiaan van Stijn
fb6784bdf0
gha: set 10-minute timeout on "report" actions
I had a CI run fail to "Upload reports":

    Exponential backoff for retry #1. Waiting for 4565 milliseconds before continuing the upload at offset 0
    Finished backoff for retry #1, continuing with upload
    Total file count: 211 ---- Processed file #160 (75.8%)
    ...
    Total file count: 211 ---- Processed file #164 (77.7%)
    Total file count: 211 ---- Processed file #164 (77.7%)
    Total file count: 211 ---- Processed file #164 (77.7%)
    A 503 status code has been received, will attempt to retry the upload
    ##### Begin Diagnostic HTTP information #####
    Status Code: 503
    Status Message: Service Unavailable
    Header Information: {
      "content-length": "592",
      "content-type": "application/json; charset=utf-8",
      "date": "Mon, 21 Aug 2023 14:08:10 GMT",
      "server": "Kestrel",
      "cache-control": "no-store,no-cache",
      "pragma": "no-cache",
      "strict-transport-security": "max-age=2592000",
      "x-tfs-processid": "b2fc902c-011a-48be-858d-c62e9c397cb6",
      "activityid": "49a48b53-0411-4ff3-86a7-4528e3f71ba2",
      "x-tfs-session": "49a48b53-0411-4ff3-86a7-4528e3f71ba2",
      "x-vss-e2eid": "49a48b53-0411-4ff3-86a7-4528e3f71ba2",
      "x-vss-senderdeploymentid": "63be6134-28d1-8c82-e969-91f4e88fcdec",
      "x-frame-options": "SAMEORIGIN"
    }
    ###### End Diagnostic HTTP information ######
    Retry limit has been reached for chunk at offset 0 to https://pipelinesghubeus5.actions.githubusercontent.com/Y2huPMnV2RyiTvKoReSyXTCrcRyxUdSDRZYoZr0ONBvpl5e9Nu/_apis/resources/Containers/8331549?itemPath=integration-reports%2Fubuntu-22.04-systemd%2Fbundles%2Ftest-integration%2FTestInfoRegistryMirrors%2Fd20ac12e48cea%2Fdocker.log
    Warning: Aborting upload for /tmp/reports/ubuntu-22.04-systemd/bundles/test-integration/TestInfoRegistryMirrors/d20ac12e48cea/docker.log due to failure
    Error: aborting artifact upload
    Total file count: 211 ---- Processed file #165 (78.1%)
    A 503 status code has been received, will attempt to retry the upload
    Exponential backoff for retry #1. Waiting for 5799 milliseconds before continuing the upload at offset 0

As a result, the "Download reports" continued retrying:

    ...
    Total file count: 1004 ---- Processed file #436 (43.4%)
    Total file count: 1004 ---- Processed file #436 (43.4%)
    Total file count: 1004 ---- Processed file #436 (43.4%)
    An error occurred while attempting to download a file
    Error: Request timeout: /Y2huPMnV2RyiTvKoReSyXTCrcRyxUdSDRZYoZr0ONBvpl5e9Nu/_apis/resources/Containers/8331549?itemPath=integration-reports%2Fubuntu-20.04%2Fbundles%2Ftest-integration%2FTestCreateWithDuplicateNetworkNames%2Fd47798cc212d1%2Fdocker.log
        at ClientRequest.<anonymous> (/home/runner/work/_actions/actions/download-artifact/v3/dist/index.js:3681:26)
        at Object.onceWrapper (node:events:627:28)
        at ClientRequest.emit (node:events:513:28)
        at TLSSocket.emitRequestTimeout (node:_http_client:839:9)
        at Object.onceWrapper (node:events:627:28)
        at TLSSocket.emit (node:events:525:35)
        at TLSSocket.Socket._onTimeout (node:net:550:8)
        at listOnTimeout (node:internal/timers:559:17)
        at processTimers (node:internal/timers:502:7)
    Exponential backoff for retry #1. Waiting for 5305 milliseconds before continuing the download
    Total file count: 1004 ---- Processed file #436 (43.4%)

And, it looks like GitHub doesn't allow cancelling the job, possibly
because it is defined with `if: always()`?

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d6f340e784)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-21 20:21:13 +02:00
Paweł Gronowski
277429622f
Merge pull request #46266 from vvoland/c8d-more-mount-refcount-24
[24.0 backport] c8d integration: Use refcount mounter for diff and export
2023-08-21 12:50:22 +02:00
Sebastiaan van Stijn
8d43772b09
Merge pull request #46244 from vvoland/c8d-inspect-handle-missing-config-24
[24.0 backport] c8d/inspect: Ignore manifest with missing config
2023-08-18 15:36:26 +02:00
Paweł Gronowski
74bf46aea6
c8d/diff: Reuse mount, mount parent as read-only
The container rw layer may already be mounted, so it's not safe to use
it in another overlay mount. Use the ref counted mounter (which will
reuse the existing mount if it exists) to avoid that.

Also, mount the parent mounts (layers of the base image) in a read-only
mode.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 6da42ca830)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-18 15:32:24 +02:00
Paweł Gronowski
b76a0c7d00
c8d/export: Use ref counted mounter
To prevent mounting the container rootfs in a rw mode if it's already
mounted.  This can't use `mount.WithReadonlyTempMount` because the
archive code does a chroot with a pivot_root, which creates a new
directory in the rootfs.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 051d51b222)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-18 15:32:20 +02:00
Paweł Gronowski
54953f2f5a
integration: Add test for not breaking overlayfs
Check that operations that could potentially perform overlayfs mounts
that could cause undefined behaviors.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 303e2b124e)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-18 15:32:04 +02:00
Paweł Gronowski
e4ecfef405
Merge pull request #46263 from vvoland/volume-decrement-refcount-typo-24
[24.0 backport] volume/local: Fix debug log typo
2023-08-18 14:58:18 +02:00
Paweł Gronowski
3897724f4a
volume/local: Fix debug log typo
Active count is incremented, but message claimed the opposite.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 7f965d55c7)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-18 13:12:57 +02:00
Sebastiaan van Stijn
821e6d8b42
Merge pull request #46260 from vvoland/ci-mobybin-sha-24
[24.0 backport] Fix DOCKER_GITCOMMIT in moby-bin
2023-08-18 12:52:11 +02:00
Bjorn Neergaard
0c131f58ba
ci(bin-image): populate DOCKER_GITCOMMIT, take 2
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
(cherry picked from commit 73ffb48bfb)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-18 09:45:37 +02:00
Bjorn Neergaard
448ae33f87
ci(bin-image): populate DOCKER_GITCOMMIT
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
(cherry picked from commit 9aed6308d4)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-18 09:45:16 +02:00
Paweł Gronowski
3ce0dc7e35
bakefile: Remove default value of DOCKER_GITCOMMIT
"HEAD" will still be used as a version if no DOCKER_COMMIT is provided
(for example when not running via `make`), but it won't prevent it being
set to the GITHUB_SHA variable when it's present.

This should fix `Git commit` reported by `docker version` for the
binaries generated by `moby-bin`.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit d7a9f15775)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-18 09:45:09 +02:00
Paweł Gronowski
600aa7b7a5
c8d/inspect: Ignore manifest with missing config
Fix a failure to inspect image if any of its present manifest references
an image config which isn't present locally.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit a64adda4e7)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-08-17 09:49:20 +02:00
Bjorn Neergaard
f1cc5760d9
Merge pull request #46214 from thaJeztah/24.0_backport_allow-all-ipv6-icc
[24.0 backport] libnet/d/bridge: Allow IPv6 ICC from any IP address
2023-08-15 16:07:36 -06:00
Bjorn Neergaard
215c6e7aec
Merge pull request #46215 from thaJeztah/24.0_backport_windows_fix_service_register
[24.0 backport] windows: fix --register-service when executed from within binary directory
2023-08-15 16:06:58 -06:00
Bjorn Neergaard
635524ea26
Merge pull request #46218 from thaJeztah/24.0_backport_bump_containerd_1.6.22
[24.0 backport] vendor: github.com/containerd/containerd v1.6.22
2023-08-15 16:06:38 -06:00
Bjorn Neergaard
6f8cf86c30
Merge pull request #46219 from thaJeztah/24.0_backport_with-go-mod_ROOTDIR
[24.0 backport] hack: use Git-free ROOTDIR
2023-08-15 16:06:24 -06:00
Bjorn Neergaard
ad7a03eb33
Merge pull request #46221 from thaJeztah/24.0_backport_capabilites
[24.0 backport] Do not drop effective&permitted set
2023-08-15 16:05:39 -06:00
Bjorn Neergaard
ff50c6e166
Merge pull request #46228 from thaJeztah/24.0_backport_runc_binary_1.1.9
[24.0 backport] update runc binary to v1.1.9
2023-08-15 16:05:06 -06:00
Sebastiaan van Stijn
bb06416756
Merge pull request #46231 from thaJeztah/24.0_backport_remove-ibm-jenkins-jobs
[24.0 backport] remove s390x and ppc64ls pipelines
2023-08-15 20:29:23 +02:00
Sam Thibault
5d2c383d72
remove s390x and ppc64ls pipelines
Signed-off-by: Sam Thibault <sam.thibault@docker.com>
(cherry picked from commit 59aa3dce8a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-14 18:46:28 +02:00