moby/daemon/logger/journald/journald.go

125 lines
3.1 KiB
Go
Raw Normal View History

// +build linux
// Package journald provides the log driver for forwarding server logs
// to endpoints that receive the systemd format.
package journald // import "github.com/docker/docker/daemon/logger/journald"
import (
"fmt"
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
"sync"
"unicode"
"github.com/coreos/go-systemd/journal"
"github.com/docker/docker/daemon/logger"
"github.com/docker/docker/daemon/logger/loggerutils"
"github.com/sirupsen/logrus"
)
const name = "journald"
type journald struct {
mu sync.Mutex
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
vars map[string]string // additional variables and values to send to the journal along with the log message
readers map[*logger.LogWatcher]struct{}
closed bool
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
}
func init() {
if err := logger.RegisterLogDriver(name, New); err != nil {
logrus.Fatal(err)
}
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
if err := logger.RegisterLogOptValidator(name, validateLogOpt); err != nil {
logrus.Fatal(err)
}
}
// sanitizeKeyMode returns the sanitized string so that it could be used in journald.
// In journald log, there are special requirements for fields.
// Fields must be composed of uppercase letters, numbers, and underscores, but must
// not start with an underscore.
func sanitizeKeyMod(s string) string {
n := ""
for _, v := range s {
if 'a' <= v && v <= 'z' {
v = unicode.ToUpper(v)
} else if ('Z' < v || v < 'A') && ('9' < v || v < '0') {
v = '_'
}
// If (n == "" && v == '_'), then we will skip as this is the beginning with '_'
if !(n == "" && v == '_') {
n += string(v)
}
}
return n
}
// New creates a journald logger using the configuration passed in on
// the context.
func New(info logger.Info) (logger.Logger, error) {
if !journal.Enabled() {
return nil, fmt.Errorf("journald is not enabled on this host")
}
// parse log tag
tag, err := loggerutils.ParseLogTag(info, loggerutils.DefaultTemplate)
if err != nil {
return nil, err
}
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
vars := map[string]string{
"CONTAINER_ID": info.ContainerID[:12],
"CONTAINER_ID_FULL": info.ContainerID,
"CONTAINER_NAME": info.Name(),
"CONTAINER_TAG": tag,
"IMAGE_NAME": info.ImageName(),
"SYSLOG_IDENTIFIER": tag,
}
extraAttrs, err := info.ExtraAttributes(sanitizeKeyMod)
if err != nil {
return nil, err
}
for k, v := range extraAttrs {
vars[k] = v
}
return &journald{vars: vars, readers: make(map[*logger.LogWatcher]struct{})}, nil
}
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
// We don't actually accept any options, but we have to supply a callback for
// the factory to pass the (probably empty) configuration map to.
func validateLogOpt(cfg map[string]string) error {
for key := range cfg {
switch key {
case "labels":
case "env":
case "env-regex":
case "tag":
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
default:
return fmt.Errorf("unknown log opt '%s' for journald log driver", key)
}
}
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
return nil
}
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
func (s *journald) Log(msg *logger.Message) error {
Improve logging of long log lines This change updates how we handle long lines of output from the container. The previous logic used a bufio reader to read entire lines of output from the container through an intermediate BytesPipe, and that allowed the container to cause dockerd to consume an unconstrained amount of memory as it attempted to collect a whole line of output, by outputting data without newlines. To avoid that, we replace the bufio reader with our own buffering scheme that handles log lines up to 16k in length, breaking up anything longer than that into multiple chunks. If we can dispense with noting this detail properly at the end of output, we can switch from using ReadBytes() to using ReadLine() instead. We add a field ("Partial") to the log message structure to flag when we pass data to the log driver that did not end with a newline. The Line member of Message structures that we pass to log drivers is now a slice into data which can be overwritten between calls to the log driver's Log() method, so drivers which batch up Messages before processing them need to take additional care: we add a function (logger.CopyMessage()) that can be used to create a deep copy of a Message structure, and modify the awslogs driver to use it. We update the jsonfile log driver to append a "\n" to the data that it logs to disk only when the Partial flag is false (it previously did so unconditionally), to make its "logs" output correctly reproduce the data as we received it. Likewise, we modify the journald log driver to add a data field with value CONTAINER_PARTIAL_MESSAGE=true to entries when the Partial flag is true, and update its "logs" reader to refrain from appending a "\n" to the data that it retrieves if it does not see this field/value pair (it also previously did this unconditionally). Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-05-24 18:12:47 +00:00
vars := map[string]string{}
for k, v := range s.vars {
vars[k] = v
}
if msg.PLogMetaData != nil && !msg.PLogMetaData.Last {
Improve logging of long log lines This change updates how we handle long lines of output from the container. The previous logic used a bufio reader to read entire lines of output from the container through an intermediate BytesPipe, and that allowed the container to cause dockerd to consume an unconstrained amount of memory as it attempted to collect a whole line of output, by outputting data without newlines. To avoid that, we replace the bufio reader with our own buffering scheme that handles log lines up to 16k in length, breaking up anything longer than that into multiple chunks. If we can dispense with noting this detail properly at the end of output, we can switch from using ReadBytes() to using ReadLine() instead. We add a field ("Partial") to the log message structure to flag when we pass data to the log driver that did not end with a newline. The Line member of Message structures that we pass to log drivers is now a slice into data which can be overwritten between calls to the log driver's Log() method, so drivers which batch up Messages before processing them need to take additional care: we add a function (logger.CopyMessage()) that can be used to create a deep copy of a Message structure, and modify the awslogs driver to use it. We update the jsonfile log driver to append a "\n" to the data that it logs to disk only when the Partial flag is false (it previously did so unconditionally), to make its "logs" output correctly reproduce the data as we received it. Likewise, we modify the journald log driver to add a data field with value CONTAINER_PARTIAL_MESSAGE=true to entries when the Partial flag is true, and update its "logs" reader to refrain from appending a "\n" to the data that it retrieves if it does not see this field/value pair (it also previously did this unconditionally). Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-05-24 18:12:47 +00:00
vars["CONTAINER_PARTIAL_MESSAGE"] = "true"
}
line := string(msg.Line)
source := msg.Source
logger.PutMessage(msg)
if source == "stderr" {
return journal.Send(line, journal.PriErr, vars)
Add log reading to the journald log driver If a logdriver doesn't register a callback function to validate log options, it won't be usable. Fix the journald driver by adding a dummy validator. Teach the client and the daemon's "logs" logic that the server can also supply "logs" data via the "journald" driver. Update documentation and tests that depend on error messages. Add support for reading log data from the systemd journal to the journald log driver. The internal logic uses a goroutine to scan the journal for matching entries after any specified cutoff time, formats the messages from those entries as JSONLog messages, and stuffs the results down a pipe whose reading end we hand back to the caller. If we are missing any of the 'linux', 'cgo', or 'journald' build tags, however, we don't implement a reader, so the 'logs' endpoint will still return an error. Make the necessary changes to the build setup to ensure that support for reading container logs from the systemd journal is built. Rename the Jmap member of the journald logdriver's struct to "vars" to make it non-public, and to make it easier to tell that it's just there to hold additional variable values that we want journald to record along with log data that we're sending to it. In the client, don't assume that we know which logdrivers the server implements, and remove the check that looks at the server. It's redundant because the server already knows, and the check also makes using older clients with newer servers (which may have new logdrivers in them) unnecessarily hard. When we try to "logs" and have to report that the container's logdriver doesn't support reading, send the error message through the might-be-a-multiplexer so that clients which are expecting multiplexed data will be able to properly display the error, instead of tripping over the data and printing a less helpful "Unrecognized input header" error. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-07-23 15:02:56 +00:00
}
return journal.Send(line, journal.PriInfo, vars)
}
func (s *journald) Name() string {
return name
}