Commit graph

94 commits

Author SHA1 Message Date
Paweł Gronowski
7532420f3b
container/SetupWorkingDirectory: Don't mutate config
Don't mutate the container's `Config.WorkingDir` permanently with a
cleaned path when creating a working directory.

Move the `filepath.Clean` to the `translateWorkingDir` instead.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-04-19 12:42:20 +02:00
Paweł Gronowski
e68f71259a
integration/build: Use fsutil.NewFS
StaticDirSource definition changed and can no longer be initialized from
the composite literal.

a80b48544c

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-02-27 11:26:10 +01:00
Paweł Gronowski
dd6992617e
integration/build: Use new buildkit progressui
Introduced in: 37131781d7

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2024-02-27 11:26:09 +01:00
Sebastiaan van Stijn
64a6cc3afd
integration/build: improve some asserts, and add asserts for unhandled errs
- add some asserts for unhandled errors
- use consts for fixed values, and slightly re-format Dockerfile contentt
- inline one-line Dockerfiles
- fix some vars to be properly camel-cased
- improve assert for error-types;

Before:

    === RUN   TestBuildPlatformInvalid
        build_test.go:685: assertion failed: expression is false: errdefs.IsInvalidParameter(err)
    --- FAIL: TestBuildPlatformInvalid (0.01s)
    FAIL

After:

    === RUN   TestBuildPlatformInvalid
        build_test.go:689: assertion failed: error is Error response from daemon: "foobar": unknown operating system or architecture: invalid argument (errdefs.errSystem), not errdefs.IsInvalidParameter
    --- FAIL: TestBuildPlatformInvalid (0.01s)
    FAIL

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-21 13:45:06 +01:00
Sebastiaan van Stijn
66cf6e3a7a
rename "image" vars to prevent conflicts with imports
We have many "image" packages, so these vars easily conflict/shadow
imports. Let's rename them (and in some cases use a const) to
prevent that.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-19 12:49:53 +01:00
Paweł Gronowski
d5eba1bfe5
integration/build: Use setupTest
These tests build new images, setupTest sets up the test cleanup
function that clears the test environment from created images,
containers, etc.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-12-05 12:49:30 +01:00
Sebastiaan van Stijn
713c7d49a1
integration(-cli): remove skips for old daemon versions (<20.10)
This removes various skips that accounted for running the integration tests
against older versions of the daemon before 20.10 (API version v1.41). Those
versions are EOL, and we don't run tests against them.

This reverts most of e440831802, and similar
PRs.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-12-05 01:03:50 +01:00
Sebastiaan van Stijn
3bfb6a9420
frozen images: update to debian:bookworm-slim
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-11-30 12:01:06 +01:00
Sebastiaan van Stijn
ebef4efb88
api/types: move ContainerLogsOptions to api/types/container
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-12 11:30:12 +02:00
Sebastiaan van Stijn
9670d9364d
api/types: move ContainerListOptions to api/types/container
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-12 11:29:24 +02:00
Paweł Gronowski
0cd02ab1b4
integration/c8d: Skip TestBuildSquashParent
`docker build --squash` is an experimental feature which is not
implemented for containerd image store.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-10-11 15:45:43 +02:00
Bjorn Neergaard
c217e3c87a
vendor: github.com/moby/buildkit v0.12.2
The following changes were required:
* integration/build: progressui's signature changed in 6b8fbed01e
* builder-next: flightcontrol.Group has become a generic type in 8ffc03b8f0
* builder-next/executor: add github.com/moby/buildkit/executor/resources types, necessitated by 6e87e4b455
* builder-next: stub util/network/Namespace.Sample(), necessitated by 963f16179f

Co-authored-by: CrazyMax <crazy-max@users.noreply.github.com>
Co-authored-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-09-21 14:18:45 -06:00
Brian Goff
9b7784781d Add test for buildkit history trace propagation
This test ensures that we are able to propagate traces into buildkit's
history API.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-09-07 18:38:22 +00:00
Brian Goff
e8dc902781 Wire up tests to support otel tracing
Integration tests will now configure clients to propagate traces as well
as create spans for all tests.

Some extra changes were needed (or desired for trace propagation) in the
test helpers to pass through tracing spans via context.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-09-07 18:38:22 +00:00
Sebastiaan van Stijn
07e6b0ac70
integration: format code with gofumpt
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-24 17:55:14 +02:00
Paweł Gronowski
f9e2eed55d
integration: Don't env cleanup before parallel subtests
Calling function returned from setupTest (which calls testEnv.Clean) in
a defer block inside a test that spawns parallel subtests caused the
cleanup function to be called before any of the subtest did anything.

Change the defer expressions to use `t.Cleanup` instead to call it only
after all subtests have also finished.
This only changes tests which have parallel subtests.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-07-13 13:41:00 +02:00
Cory Snider
0c68b655f6 Fix loop-closure bugs in tests
...which were flagged by golangci-lint v1.51.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-02-15 12:44:52 -05:00
Tonis Tiigi
666334bd48
vendor: update buildkit to v0.11.2
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-02-01 23:29:10 -08:00
Brian Goff
402d106142
Merge pull request #43009 from thaJeztah/skip_TestBuildWCOWSandboxSize
testing: skip TestBuildWCOWSandboxSize for now
2021-11-10 11:59:09 -08:00
Sebastiaan van Stijn
2a6a4587fa
testing: skip TestBuildWCOWSandboxSize for now
This test is failing frequently once nodes have less disk space
available. Skipping the test for now, but we can continue looking
for a good solution.

Tracked through https://github.com/moby/moby/issues/42743

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-11-10 15:38:22 +01:00
Sebastiaan van Stijn
eea2758761
TestBuildUserNamespaceValidateCapabilitiesAreV2: cleanup daemon storage
This should help with Jenkins failing to clean up the Workspace:

- make sure "cleanup" is also called in the defer for all daemons. keeping
  the daemon's storage around prevented Jenkins from cleaning up.
- close client connections and some readers (just to be sure)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-11-10 12:25:16 +01:00
Eng Zer Jun
c55a4ac779
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-27 14:56:57 +08:00
Sebastiaan van Stijn
f68260ba22
Dockerfile: frozen images: update to bullseye, remove buildpack-dep
Update the frozen images to also be based on Debian bullseye. Using the "slim"
variant (which looks to have all we're currently using),  and remove the
buildpack-dep frozen image.

The buildpack-dep image is quite large, and it looks like we only use it to
compile some C binaries, which should work fine on a regular debian image;

    docker build -t debian:bullseye-slim-gcc -<<EOF
    FROM debian:bullseye-slim
    RUN apt-get update && apt-get install -y gcc libc6-dev --no-install-recommends
    EOF

    docker image ls

    REPOSITORY        TAG                 IMAGE ID       CREATED          SIZE
    debian            bullseye-slim-gcc   1851750242af   About a minute ago   255MB
    buildpack-deps    bullseye            fe8fece98de2   2 days ago           834MB

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-20 18:23:06 +02:00
Roman Volosatovs
47ad2f3dd6
API,daemon: support type URL parameter to /system/df
Let clients choose object types to compute disk usage of.

Signed-off-by: Roman Volosatovs <roman.volosatovs@docker.com>
Co-authored-by: Sebastiaan van Stijn <github@gone.nl>
2021-07-27 12:17:45 +02:00
Sebastiaan van Stijn
7c91fd4240
unused: ignore false positives
integration/build/build_session_test.go:92:6: func `testBuildWithSession` is unused (unused)
    func testBuildWithSession(t *testing.T, client dclient.APIClient, daemonHost string, dir, dockerfile string) (outStr string) {
         ^
    integration/container/checkpoint_test.go:23:6: func `containerExec` is unused (unused)
    func containerExec(t *testing.T, client client.APIClient, cID string, cmd []string) {
         ^
    integration/network/service_test.go:295:6: func `swarmIngressReady` is unused (unused)
    func swarmIngressReady(client client.NetworkAPIClient) func(log poll.LogT) poll.Result {
         ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-10 13:03:32 +02:00
Sebastiaan van Stijn
fa480403c7
TestBuildUserNamespaceValidateCapabilitiesAreV2: verify build completed
Check if the `docker build` completed successfully before continuing.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-15 16:08:40 +01:00
Sebastiaan van Stijn
26965fbfa0
TestBuildUserNamespaceValidateCapabilitiesAreV2: use correct image name
This currently doesn't make a difference, because load.FrozenImagesLinux()
currently loads all frozen images, not just the specified one, but in case
that is fixed/implemented at some point.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-15 14:02:41 +01:00
Sebastiaan van Stijn
f266f13965
Merge pull request #41636 from TBBle/37352-test-and-fix
Set 127GB default sandbox size for WCOW, and ensure storage-opts is honoured on all paths under WCOW and LCOW
2021-01-25 14:34:34 +01:00
Sebastiaan van Stijn
41e5d459f1
Merge pull request #41865 from thaJeztah/load_frozen_images
integration: fix TestBuildUserNamespaceValidateCapabilitiesAreV2 not using frozen image
2021-01-13 18:50:22 +01:00
Sebastiaan van Stijn
46dfc31342
integration: fix TestBuildUserNamespaceValidateCapabilitiesAreV2 not using frozen image
Commit f2f5106c92 added this test to verify loading
of images that were built with user-namespaces enabled.

However, because this test spins up a new daemon, not the daemon that's set up by
the test-suite's `TestMain()` (which loads the frozen images).

As a result, the `debian:bullseye` image was pulled from Docker Hub when running
the test;

    Calling POST /v1.41/images/load?quiet=1
    Applying tar in /go/src/github.com/docker/docker/bundles/test-integration/TestBuildUserNamespaceValidateCapabilitiesAreV2/d4d366b15997b/root/165536.165536/overlay2/3f7f9375197667acaf7bc810b34689c21f8fed9c52c6765c032497092ca023d6/diff" storage-driver=overlay
    Applied tar sha256:845f0e5159140e9dbcad00c0326c2a506fbe375aa1c229c43f082867d283149c to 3f7f9375197667acaf7bc810b34689c21f8fed9c52c6765c032497092ca023d6, size: 5922359
    Calling POST /v1.41/build?buildargs=null&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&t=capabilities%3A1.0&target=&ulimits=null&version=
    Trying to pull debian from https://registry-1.docker.io v2
    Fetching manifest from remote" digest="sha256:f169dbadc9021fc0b08e371d50a772809286a167f62a8b6ae86e4745878d283d" error="<nil>" remote="docker.io/library/debian:bullseye
    Pulling ref from V2 registry: debian:bullseye
    ...

This patch updates `TestBuildUserNamespaceValidateCapabilitiesAreV2` to load the
frozen image. `StartWithBusybox` is also changed to `Start`, because the test
is not using the busybox image, so there's no need to load it.

In a followup, we should probably add some utilities to make this easier to set up
(and to allow passing the list frozen images that we want to load, without having
to "hard-code" the image name to load).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-01-08 13:25:21 +01:00
Eric Mountain
95eb490780
Use v2 capabilities in layer archives
When building images in a user-namespaced container, v3 capabilities are
stored including the root UID of the creator of the user-namespace.

This UID does not make sense outside the build environment however. If
the image is run in a non-user-namespaced runtime, or if a user-namespaced
runtime uses a different UID, the capabilities requested by the effective
bit will not be honoured by `execve(2)` due to this mismatch.

Instead, we convert v3 capabilities to v2, dropping the root UID on the
fly.

Signed-off-by: Eric Mountain <eric.mountain@datadoghq.com>
2020-12-23 14:16:41 +01:00
Eric Mountain
f2f5106c92
Add a test for #41723
Capabilities are serialised in VFS_CAP_REVISION_3 when an image is
built in a user-namespaced daemon, instead of VFS_CAP_REVISION_2.

This adds a test for this, though it's currently wired to fail if
the capabilities are serialised in VFS_CAP_REVISION_2 instead in this
situation, since this is unexpected.

Signed-off-by: Eric Mountain <eric.mountain@datadoghq.com>
2020-12-18 07:51:44 +01:00
Paul "TBBle" Hampson
695b151a18 Work around small disk on Windows-RS5 CI nodes
The free disk space on the Windows RS5 CI nodes appears to be just the
right size that the TestBuildWCOWSandboxSize test can generate 21GB of
layers, and then a 21GB sandbox inside a container, and then runs out of
space while committing the layer.

Helpfully, this failure is distinguishable in the logs from a failure
when the sandbox is too small, so we can do that.

TODO: Revert this if-and-when the Windows-RS5 CI nodes have more free
space.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-10 19:51:46 +11:00
Paul "TBBle" Hampson
142b2b785b Add TestBuildWCOWSandboxSize integration test
This test validates that `RUN` and `COPY` both target a read-write
sandbox on Windows that is configured according to the daemon's
`storage-opts` setting.

Sadly, this is a slow test, so we need to bump the timeout to 60 minutes
from the default of 10 minutes.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-11-10 19:51:46 +11:00
Tibor Vass
29b149ebb1
Merge pull request #41430 from TBBle/40444-update-gowinio-for-8gB-file-fix
Revendor Microsoft/go-winio for 8gB file fix when importing or committing image layers
2020-09-24 11:16:10 -07:00
Olli Janatuinen
66a37b460c Windows: Enable more integration tests
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2020-09-20 22:09:27 +03:00
Paul "TBBle" Hampson
2dd9faaa16 Enable TestBuildWithHugeFile on Windows
This reproduces #40444, based on a suggestion from GitHub user @marosset

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-09-19 23:13:44 +10:00
Kir Kolyshkin
9b77370213 int/build/TestBuildWithEmptyLayers: prevent panic
The test case panics like that:

> build_test.go:381: assertion failed: 3 (int) != 1 (int)
> panic: runtime error: index out of range [2] with length 1 [recovered]
> panic: runtime error: index out of range [2] with length 1

The fix is trivial.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2020-06-04 11:22:04 -07:00
Cristina Yenyxe Gonzalez Garcia
ebd025b63a Using test names instead of hardcoded ones in integration/build directory
Signed-off-by: Cristina Yenyxe Gonzalez Garcia <cristina.yenyxe@gmail.com>
2020-05-18 16:36:23 +01:00
Sebastiaan van Stijn
562880b276
Fix more goimports
```
daemon/logger/splunk/splunk_test.go:33: File is not `goimports`-ed (goimports)
        envKey:      "a",
        envRegexKey: "^foo",
        labelsKey:   "b",
        tagKey:      "c",
integration/build/build_test.go:41: File is not `goimports`-ed (goimports)
            rm:      false,
            forceRm: false,
integration/image/remove_unix_test.go:49: File is not `goimports`-ed (goimports)
        Root: d.Root,
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-11 18:56:25 +01:00
Sebastiaan van Stijn
9f0b3f5609
bump gotest.tools v3.0.1 for compatibility with Go 1.14
full diff: https://github.com/gotestyourself/gotest.tools/compare/v2.3.0...v3.0.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-11 00:06:42 +01:00
Tibor Vass
6ca3ec88ae builder: remove legacy build's session handling
This feature was used by docker build --stream and it was kept experimental.

Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.

Signed-off-by: Tibor Vass <tibor@docker.com>
2019-10-02 20:29:15 +00:00
Sebastiaan van Stijn
554d9cec25
testutil: update WithExperimental signature to be a daemon.Option
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-30 17:38:24 +02:00
Sebastiaan van Stijn
32f1c65162
TestBuildSquashParent: fix non-standard comparisson
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-09-12 00:53:59 +02:00
Sam Whited
b37c214e3c testutil: make testing packages public
This was done with something along the lines of:

```
mv internal/test testutil
pushd testutil/; grep -IRl "package test" | xargs -I '{}' sed -i -e 's|package test|package testutil|g' {}; popd
mv internal/testutil/*.go testutil/ && rm -rf internal/
grep -IRl "github.com\/docker\/docker\/internal\/test" | xargs -I '{}' sed -i -e 's|github.com/docker/docker/internal/test|github.com/docker/docker/test|g' {}
goimports .
```

I also modified the basic plugin path in testutil/fixtures/plugin.

Signed-off-by: Sam Whited <sam@samwhited.com>
2019-09-11 07:47:23 -05:00
Vikram bir Singh
b146b2382c Fixups for 39695 - edit comments, redundant lines
1. Modify comments added in 5858a99267
Windows Volume GUID path format is: \\?\Volume{<GUID Value>}\<path>
Rewrote the example given in comments to conform to the format..

2. Remove two redundant asserts[assert.NilError]. They are redundant
because the last statement will not change the value of err.

Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>
2019-08-15 21:13:46 +00:00
Sebastiaan van Stijn
5858a99267
Builder: fix "COPY --from" to non-existing directory on Windows
This fixes a regression introduced in 6d87f19142,
causing `COPY --from` to fail if the target directory does not exist:

```
FROM mcr.microsoft.com/windows/servercore:ltsc2019 as s1
RUN echo "Hello World" > /hello

FROM mcr.microsoft.com/windows/servercore:ltsc2019
COPY --from=s1 /hello /hello/another/world
```

Would produce an error:

```
Step 4/4 : COPY --from=s1 /hello /hello/another/world
failed to copy files: mkdir \\?: The filename, directory name, or volume label syntax is incorrect.
```

The cause for this was that Go's `os.MkdirAll()` does not support/detect volume GUID paths
(`\\?\Volume{dae8d3ac-b9a1-11e9-88eb-e8554b2ba1db}\hello\another}`), and as a result
attempted to create the volume as a directory (`\\?`), causing it to fail.

This patch replaces `os.MkdirAll()` with our own `system.MkdirAll()` function, which
is capable of detecting GUID volumes.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-08 16:45:10 +02:00
Sebastiaan van Stijn
becd29c665
integration: run build session tests on non-experimental
The session endpoint is no longer experimental since
01c9e7082e, so we don't
need to start an experimental daemon.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-17 23:56:04 +02:00
Sebastiaan van Stijn
9d1b4f5fc3
Add regression tests for invalid platform status codes
Before we handled containerd errors, using an invalid platform produced a 500 status:

```bash
curl -v \
  -X POST \
  --unix-socket /var/run/docker.sock \
  "http://localhost:2375/v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest" \
  -H "Content-Type: application/json"
```

```
* Connected to localhost (docker.sock) port 80 (#0)
> POST /v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest HTTP/1.1
> Host: localhost:2375
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.40
< Content-Length: 85
< Content-Type: application/json
< Date: Mon, 15 Jul 2019 15:25:44 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/19.03.0-rc2 (linux)
<
{"message":"\"foobar\": unknown operating system or architecture: invalid argument"}
```

That problem is now fixed, and the API correctly returns a 4xx status:

```bash
curl -v \
  -X POST \
  --unix-socket /var/run/docker.sock \
  "http://localhost:2375/v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest" \
  -H "Content-Type: application/json"
```

```
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> POST /v1.40/images/create?fromImage=hello-world&platform=foobar&tag=latest HTTP/1.1
> Host: localhost:2375
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 400 Bad Request
< Api-Version: 1.41
< Content-Type: application/json
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Mon, 15 Jul 2019 15:13:42 GMT
< Content-Length: 85
<
{"message":"\"foobar\": unknown operating system or architecture: invalid argument"}
* Curl_http_done: called premature == 0
```

This patch adds tests to validate the behaviour

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-07-15 20:37:00 +02:00
Sebastiaan van Stijn
9f9b4290b9
integration: change container.Run signature to fix linting
Line 59: warning: context.Context should be the first parameter of a function (golint)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-06-07 13:05:26 +02:00