Commit graph

43587 commits

Author SHA1 Message Date
Kazuyoshi Kato
7a10f5a558 daemon/logger: refactor followLogs to write more unit tests
followLogs() is getting really long (170+ lines) and complex.
The function has multiple inner functions that mutate its variables.

To refactor the function, this change introduces follow{} struct.
The inner functions are now defined as ordinal methods, which are
accessible from tests.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-12-27 09:11:12 -08:00
Conor Evans
5cbc08ce57
The flag ForceStopAsyncSend was added to fluent logger lib in v1.9.0
* When async is enabled, this option defines the interval (ms) at which the connection
to the fluentd-address is re-established. This option is useful if the address
may resolve to one or more IP addresses, e.g. a Consul service address.

While the change in #42979 resolves the issue where a Docker container can be stuck
if the fluentd-address is unavailable, this functionality adds an additional benefit
in that a new and healthy fluentd-address can be resolved, allowing logs to flow once again.

This adds a `fluentd-async-reconnect-interval` log-opt for the fluentd logging driver.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Conor Evans <coevans@tcd.ie>

Co-authored-by: Sebastiaan van Stijn <github@gone.nl>
Co-authored-by: Conor Evans <coevans@tcd.ie>
2021-12-24 22:04:08 +01:00
Michael Kuehn
60f868c263
Ensure system clock was set
Single-Board Computer and embedded systems might have a clock that is extremely out of sync with reality.
Adding this target ensures docker is only started after a somewhat realistic clock was set.
More information about the time-set.target can be found here: https://www.freedesktop.org/software/systemd/man/systemd.special.html#time-sync.target

Signed-off-by: Michael Kuehn <micha@kuehn.io>
2021-12-24 16:08:53 +01:00
Sebastiaan van Stijn
10aecb0e65
Merge pull request #43104 from kzys/no-ioutil
Remove io/ioutil again
2021-12-23 21:08:52 +01:00
Kazuyoshi Kato
8b4c445f54 test: use os.CreateTemp instead of ioutil.TempFile
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-12-23 09:09:47 -08:00
Kazuyoshi Kato
9a0b1f4f6b hack: use golangci-lint to find io/ioutil
The package has been deprecated since Go 1.16.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-12-23 08:58:42 -08:00
Conor Evans
3500d7e472
vendor: github.com/fluent/fluent-logger-golang v1.9.0
Updates the fluent logger library to v1.9.0. The update includes the following commit:

* [Add periodic reconnection functionality](fluent/fluent-logger-golang@1c05506)

See https://github.com/fluent/fluent-logger-golang/compare/v1.8.0..v1.9.0

Signed-off-by: Conor Evans <coevans@tcd.ie>
2021-12-23 16:54:12 +01:00
Sebastiaan van Stijn
64441553e7
Merge pull request #43043 from kzys/decode-3
daemon/logger: read the length header correctly
2021-12-23 10:18:15 +01:00
Justin Cormack
dbef655af1
Merge pull request #43097 from thaJeztah/fix_integration_cli_tests
integration-cli: delete existing docker0 bridge to prevent it interfering
2021-12-22 15:40:00 +00:00
Tianon Gravi
885f3c65d5
Merge pull request #43057 from thaJeztah/update_crypto_ssh
go.mod: golang.org/x/crypto 5770296d904e90f15f38f77dfc2e43fdf5efc083
2021-12-21 15:19:54 -08:00
Sebastiaan van Stijn
f415028976
integration-cli: delete existing docker0 bridge to prevent it interfering
A number of tests in the TestDockerDaemonSuite create a custom bridge as part
of the test. In some cases, an existing `docker0` bridge could interfere with
those tests. For example, the `TestDaemonICCLinkExpose` and `TestDaemonICCPing`
verify that no "icc" communication is possible, and for this create a new
bridge with a custom IP-range.

However, depending on which tests ran before the test, a default `docker0` bridge
may exist (e.g., if the`TestDefaultGatewayIPv4Implicit`) with the same IP-range,
in which iptables rules may have been set up that allow communication, and thus
make the "icc" tests fail.

This patch removes the `docker0` interface at the start of tests that create
their own bridge to prevent it from interfering.

Note that alternatively, we could update those tests to use an IP-range that's
less likely to overlap, but this may be more brittle (but could still be done
in addition to this change as a follow-up).

To verify these changes;

    make DOCKER_GRAPHDRIVER=vfs TEST_SKIP_INTEGRATION=1 TESTFLAGS='-test.run TestDockerDaemonSuite/TestDaemon(DefaultGatewayIPv4|ICC)' test-integration-cli

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-12-21 16:12:24 +01:00
Aaron Lehmann
dd66dcad9c fileutils: Avoid compiling a regexp for simple patterns
If we detect that a pattern is either an exact match, prefix match, or
suffix match, use an optimized code path instead of compiling a regexp.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-12-20 16:32:13 -08:00
Sören Tempel
85eaf23bf4 seccomp: add support for "swapcontext" syscall in default policy
This system call is only available on the 32- and 64-bit PowerPC, it is
used by modern programming language implementations (such as gcc-go) to
implement coroutine features through userspace context switches.

Other container environment, such as Systemd nspawn already whitelist
this system call in their seccomp profile [1] [2]. As such, it would be
nice to also whitelist it in moby.

This issue was encountered on Alpine Linux GitLab CI system, which uses
moby, when attempting to execute gcc-go compiled software on ppc64le.

[1]: https://github.com/systemd/systemd/pull/9487
[2]: https://github.com/systemd/systemd/issues/9485

Signed-off-by: Sören Tempel <soeren+git@soeren-tempel.net>
2021-12-18 14:06:07 +01:00
Sebastiaan van Stijn
088afc99e4
Merge pull request #43087 from tonistiigi/fix-darwin-build-system
system: unbreak build for darwin
2021-12-16 20:06:57 +01:00
Tonis Tiigi
26dafe43a4 system: unbreak build for darwin
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-12-15 22:15:07 -08:00
Brian Goff
07cf48ab63
Merge pull request #43078 from thaJeztah/bump_go_1.17.5
Update Go to 1.17.5
2021-12-15 15:48:38 -08:00
Kazuyoshi Kato
f2e458ebc5 daemon/logger: test followLogs' handleDecodeErr case
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-12-15 15:13:02 -08:00
Kazuyoshi Kato
48d387a757 daemon/logger: read the length header correctly
Before this change, if Decode() couldn't read a log record fully,
the subsequent invocation of Decode() would read the record's non-header part
as a header and cause a huge heap allocation.

This change prevents such a case by having the intermediate buffer in
the decoder struct.

Fixes #42125.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-12-15 15:13:02 -08:00
Sebastiaan van Stijn
d620cb6afc
Update Go to 1.17.5
go1.17.5 (released 2021-12-09) includes security fixes to the syscall and net/http
packages. See the Go 1.17.5 milestone on the issue tracker for details:

https://github.com/golang/go/issues?q=milestone%3AGo1.17.5+label%3ACherryPickApproved

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-12-12 01:13:39 +01:00
Tianon Gravi
8955d8da89
Merge pull request #42981 from frobnicaty/patch-1
Fix grammar for "does not exist"
2021-12-09 13:36:53 -08:00
Brian Goff
d456264580
Merge pull request #43062 from AkihiroSuda/runc-1.0.3
Bump up runc v1.0.3
2021-12-08 10:09:46 -08:00
Tianon Gravi
87521affb0
Merge pull request #43064 from thaJeztah/bump_go_1.17.4
Update Go to 1.17.4
2021-12-07 17:17:58 -08:00
Sebastiaan van Stijn
6bb3891c60
Update Go to 1.17.4
go1.17.4 (released 2021-12-02) includes fixes to the compiler, linker, runtime,
and the go/types, net/http, and time packages. See the Go 1.17.4 milestone on
the issue tracker for details:

https://github.com/golang/go/issues?q=milestone%3AGo1.17.4+label%3ACherryPickApproved

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-12-06 10:13:39 +01:00
Mathieu PATUREL
7c17ad8735 docker-rootless-setuptools.sh: use context after install
Signed-off-by: Mathieu PATUREL <mathieu.paturel@gmail.com>
2021-12-06 17:28:34 +11:00
Akihiro Suda
934f955e3d
vendor: github.com/opencontainers/runc v1.0.3
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-12-06 15:11:57 +09:00
Akihiro Suda
53397ac539
update runc binary to v1.0.3
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-12-06 15:10:27 +09:00
frobnicaty
d78b883576 Fix grammar for "does not exist"
as opposed to "does not exists"

Signed-off-by: frobnicaty <92033765+frobnicaty@users.noreply.github.com>
2021-12-03 15:50:13 +00:00
Sebastiaan van Stijn
dd9782fe94
go.mod: golang.org/x/crypto 5770296d904e90f15f38f77dfc2e43fdf5efc083
full diff: 0c34fe9e7d...5770296d90

includes a fix in golang.org/x/crypto/ssh for CVE-2021-43565

- golang/go#49932
- 5770296d90

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-12-03 09:19:28 +01:00
Sebastiaan van Stijn
f6848ae321
Merge pull request #42979 from akerouanton/bump-fluent-logger
vendor: github.com/fluent/fluent-logger-golang v1.8.0
2021-12-02 20:51:04 +01:00
Tianon Gravi
0f92cf2044
Merge pull request #43036 from thaJeztah/bump_hcsshim
vendor: github.com/Microsoft/hcsshim v0.8.23
2021-12-02 11:49:03 -08:00
Sebastiaan van Stijn
787b8fe14f
Merge pull request #42838 from sanjams2/42731-development
Add an option to specify log format for awslogs driver
2021-12-02 20:48:06 +01:00
Brian Goff
b46ab1f579
Merge pull request #43035 from thaJeztah/bump_ttrpc
vendor: github.com/containerd/ttrpc v1.1.0
2021-12-02 11:34:11 -08:00
Tianon Gravi
cf811b1122
Merge pull request #42574 from charlesxsh/fix-deadlock-1
fix potential goroutine leak by making channel non-blocking
2021-12-01 17:35:30 -08:00
Albin Kerouanton
bd61629b6b
fluentd: Turn ForceStopAsyncSend true when async connect is used
The flag ForceStopAsyncSend was added to fluent logger lib in v1.5.0 (at
this time named AsyncStop) to tell fluentd to abort sending logs
asynchronously as soon as possible, when its Close() method is called.
However this flag was broken because of the way the lib was handling it
(basically, the lib could be stucked in retry-connect loop without
checking this flag).

Since fluent logger lib v1.7.0, calling Close() (when ForceStopAsyncSend
is true) will really stop all ongoing send/connect procedure,
wherever it's stucked.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2021-12-02 01:15:28 +01:00
Albin Kerouanton
e24d61b7ef
vendor: github.com/fluent/fluent-logger-golang v1.8.0
Updates the fluent logger library to v1.8.0. Following PRs/commits were
merged since last bump:

* [Add callback for error handling when using
  async](https://github.com/fluent/fluent-logger-golang/pull/97)
* [Fix panic when accessing unexported struct
  field](https://github.com/fluent/fluent-logger-golang/pull/99)
* [Properly stop logger during (re)connect
  failure](https://github.com/fluent/fluent-logger-golang/pull/82)
* [Support a TLS-enabled connection](e5d6aa13b7)

See https://github.com/fluent/fluent-logger-golang/compare/v1.6.1..v1.8.0

Signed-off-by: Albin Kerouanton <albinker@gmail.com>
2021-12-02 01:11:33 +01:00
Sebastiaan van Stijn
17b9e4ec92
Merge pull request #43047 from aaronlehmann/pattern-matcher-parent-results-fix
Fix missing parent info case in MatchesUsingParentResults
2021-12-01 21:15:19 +01:00
Sebastiaan van Stijn
9d9b8e0cf3
daemon.WithDevices(): use containerd's HostDevices()
Trying to reduce the use of libcontainer/devices, as it's considered
to be an "internal" package by runc.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-12-01 15:42:18 +01:00
Aaron Lehmann
55da5245de Fix missing parent info case in MatchesUsingParentResults
Unfortunately, this check was missing in the original version. It could
cause a positive match to be overwritten by checking parent dirs.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-11-26 10:22:19 -08:00
Sebastiaan van Stijn
93d560d5b3
Merge pull request #43037 from aaronlehmann/pattern-matcher-parent-results
pkg/fileutils: Track incremental pattern match results against each pattern
2021-11-24 18:38:51 +01:00
Aaron Lehmann
0f1b68df16 Adjust deprecation comments
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-11-24 07:53:10 -08:00
Samuel Karp
d44ccafefa
Merge pull request #43045 from samuelkarp/distribution-validate-blob-type 2021-11-23 18:54:11 -08:00
Samuel Karp
7c216bcabe
distribution: validate blob type
Signed-off-by: Samuel Karp <skarp@amazon.com>
2021-11-23 17:19:01 -08:00
Sebastiaan van Stijn
00d8a3bb51
Merge pull request #43042 from jelipo/fix_image_spec
fix image spec JSON example
2021-11-23 11:00:11 +01:00
haining.cao
715e78631d fix image spec JSON example
Signed-off-by: haining.cao <haining.cao@daocloud.io>
2021-11-22 23:37:23 +08:00
Aaron Lehmann
4555d3aa54 pkg/fileutils: Track incremental pattern match results against each pattern
The existing code does not correctly handle the case where a file
matches one of the patterns, but should not match overall because of an
exclude pattern that applied to a parent directory (see
https://github.com/docker/buildx/issues/850).

Fix this by independently tracking the results of matching against each
pattern. A file should be considered to match any pattern that matched a
parent dir.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-11-19 15:58:13 -08:00
Sebastiaan van Stijn
a715bfb857
vendor: github.com/Microsoft/hcsshim v0.8.23
full diff: https://github.com/Microsoft/hcsshim/compare/v0.8.22...v0.8.23

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-11-19 13:50:13 +01:00
Sebastiaan van Stijn
cbb4aed0b4
vendor: github.com/containerd/ttrpc v1.1.0
full diff: https://github.com/containerd/ttrpc/compare/v1.0.2...v1.1.0

- client: Handle sending/receiving in separate goroutines
- Return Unimplemented when services or methods are not implemented
- go.mod: sirupsen/logrus v1.7.0
- go.mod: update dependencies
  - go.mod: github.com/gogo/protobuf v1.3.2
  - go.mod: google.golang.org/grpc v1.27.1
  - go.mod: google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63
  - go.mod: github.com/prometheus/procfs v0.6.0
- replace pkg/errors
- Rename branch from master to main
- Use GitHub Actions for CI
- Make "go test" and "go build" work on macOS
- Add protoc-gen-go-ttrpc

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-11-19 13:48:43 +01:00
Sebastiaan van Stijn
ea0f3dc8f4
Merge pull request #43031 from thaJeztah/fix_TestConntrackFlowsLeak_v1
TestConntrackFlowsLeak: use busybox "nc"
2021-11-19 13:39:01 +01:00
Sebastiaan van Stijn
f06314fa84
Merge pull request #42980 from thaJeztah/bump_sys
vendor: github.com/moby/sys/mount v0.3.0, mountinfo v0.5.0, signal v0.6.0, symlink v0.2.0
2021-11-19 13:23:51 +01:00
Sebastiaan van Stijn
ce25968008
vendor: github.com/moby/sys/mount v0.3.0, mountinfo v0.5.0, signal v0.6.0, symlink v0.2.0
full diff: https://github.com/moby/sys/compare/signal/v0.5.0...signal/v0.6.0

Modules:

- github.com/moby/sys/mount v0.3.0
- github.com/moby/sys/mountinfo v0.5.0
- github.com/moby/sys/signal v0.6.0
- github.com/moby/sys/symlink v0.2.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-11-19 09:56:37 +01:00