Better selection of DNS server

Commit e353e7e3f0 updated selection of the
`resolv.conf` file to use in situations where systemd-resolvd is used as
a resolver.

If a host uses `systemd-resolvd`, the system's `/etc/resolv.conf` file is
updated to set `127.0.0.53` as DNS, which is the local IP address for
systemd-resolvd. The DNS servers that are configured by the user will now
be stored in `/run/systemd/resolve/resolv.conf`, and systemd-resolvd acts
as a forwarding DNS for those.

Originally, Docker copied the DNS servers as configured in `/etc/resolv.conf`
as default DNS servers in containers, which failed to work if systemd-resolvd
is used (as `127.0.0.53` is not available inside the container's networking
namespace). To resolve this, e353e7e3f0 instead
detected if systemd-resolvd is in use, and in that case copied the "upstream"
DNS servers from the `/run/systemd/resolve/resolv.conf` configuration.

While this worked for most situations, it had some downsides, among which:

- we're skipping systemd-resolvd altogether, which means that we cannot take
  advantage of addition functionality provided by it (such as per-interface
  DNS servers)
- when updating DNS servers in the system's configuration, those changes were
  not reflected in the container configuration, which could be problematic in
  "developer" scenarios, when switching between networks.

This patch changes the way we select which resolv.conf to use as template
for the container's resolv.conf;

- in situations where a custom network is attached to the container, and the
  embedded DNS is available, we use `/etc/resolv.conf` unconditionally. If
  systemd-resolvd is used, the embedded DNS forwards external DNS lookups to
  systemd-resolvd, which in turn is responsible for forwarding requests to
  the external DNS servers configured by the user.
- if the container is running in "host mode" networking, we also use the
  DNS server that's configured in `/etc/resolv.conf`. In this situation, no
  embedded DNS server is available, but the container runs in the host's
  networking namespace, and can use the same DNS servers as the host (which
  could be systemd-resolvd or DNSMasq
- if the container uses the default (bridge) network, no embedded DNS is
  available, and the container has its own networking namespace. In this
  situation we check if systemd-resolvd is used, in which case we skip
  systemd-resolvd, and configure the upstream DNS servers as DNS for the
  container. This situation is the same as is used currently, which means
  that dynamically switching DNS servers won't be supported for these
  containers.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

daemon/container_operations_unix.go

@@ -391,13 +391,65 @@ func serviceDiscoveryOnDefaultNetwork() bool {
 func (daemon *Daemon) setupPathsAndSandboxOptions(container *container.Container, sboxOptions *[]libnetwork.SandboxOption) error {
 	var err error
 
-	if container.HostConfig.NetworkMode.IsHost() {
-		// Point to the host files, so that will be copied into the container running in host mode
-		*sboxOptions = append(*sboxOptions, libnetwork.OptionOriginHostsPath("/etc/hosts"))
-	}
+	// Set the correct paths for /etc/hosts and /etc/resolv.conf, based on the
+	// networking-mode of the container. Note that containers with "container"
+	// networking are already handled in "initializeNetworking()" before we reach
+	// this function, so do not have to be accounted for here.
+	switch {
+	case container.HostConfig.NetworkMode.IsHost():
+		// In host-mode networking, the container does not have its own networking
+		// namespace, so both `/etc/hosts` and `/etc/resolv.conf` should be the same
+		// as on the host itself. The container gets a copy of these files, but they
+		// may be symlinked, so resolve the original path first.
+		etcHosts, err := filepath.EvalSymlinks("/etc/hosts")
+		if err != nil {
+			return err
+		}
+		resolvConf, err := filepath.EvalSymlinks("/etc/resolv.conf")
+		if err != nil {
+			return err
+		}
 
-	// Copy the host's resolv.conf for the container (/etc/resolv.conf or /run/systemd/resolve/resolv.conf)
-	*sboxOptions = append(*sboxOptions, libnetwork.OptionOriginResolvConfPath(daemon.configStore.GetResolvConf()))
+		*sboxOptions = append(
+			*sboxOptions,
+			libnetwork.OptionOriginHostsPath(etcHosts),
+			libnetwork.OptionOriginResolvConfPath(resolvConf),
+		)
+	case container.HostConfig.NetworkMode.IsUserDefined():
+		// The container uses a user-defined network. We use the embedded DNS
+		// server for container name resolution and to act as a DNS forwarder
+		// for external DNS resolution.
+		// We parse the DNS server(s) that are defined in /etc/resolv.conf on
+		// the host, which may be a local DNS server (for example, if DNSMasq or
+		// systemd-resolvd are in use). The embedded DNS server forwards DNS
+		// resolution to the DNS server configured on the host, which in itself
+		// may act as a forwarder for external DNS servers.
+		// If systemd-resolvd is used, the "upstream" DNS servers can be found in
+		// /run/systemd/resolve/resolv.conf. We do not query those DNS servers
+		// directly, as they can be dynamically reconfigured.
+		resolvConf, err := filepath.EvalSymlinks("/etc/resolv.conf")
+		if err != nil {
+			return err
+		}
+		*sboxOptions = append(*sboxOptions, libnetwork.OptionOriginResolvConfPath(resolvConf))
+	default:
+		// For other situations, such as the default bridge network, container
+		// discovery / name resolution is handled through /etc/hosts, and no
+		// embedded DNS server is available. Without the embedded DNS, we
+		// cannot use local DNS servers on the host (for example, if DNSMasq or
+		// systemd-resolvd is used). If systemd-resolvd is used, we try to
+		// determine the external DNS servers that are used on the host.
+		// This situation is not ideal, because DNS servers configured in the
+		// container are not updated after the container is created, but the
+		// DNS servers on the host can be dynamically updated.
+		//
+		// Copy the host's resolv.conf for the container (/run/systemd/resolve/resolv.conf or /etc/resolv.conf)
+		resolvConf, err := filepath.EvalSymlinks(daemon.configStore.GetResolvConf())
+		if err != nil {
+			return err
+		}
+		*sboxOptions = append(*sboxOptions, libnetwork.OptionOriginResolvConfPath(resolvConf))
+	}
 
 	container.HostsPath, err = container.GetRootResourcePath("hosts")
 	if err != nil {