Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial zapdriver implementation #1

Merged
merged 2 commits into from
May 25, 2018
Merged

Add initial zapdriver implementation #1

merged 2 commits into from
May 25, 2018

Conversation

JeanMertz
Copy link
Contributor

@JeanMertz JeanMertz commented May 25, 2018

This moves some of blendle/go-logger#5 into a separate package.

I noticed that that Stackdriver implementation was actually incorrect for the use-case we have (which is Kubernetes-based logging, which does not use the Stackdriver error logging format, but instead uses the general LogEntry format).

This adds an easy-to-use general-purpose Stackdriver wrapper on top of Zap for our use-case.

README copy/pasted below:


⚡ Zapdriver

Blazing fast, Zap-based Stackdriver logging.

Usage

This package provides three building blocks to support the full array of
structured logging capabilities of Stackdriver:

The above components can be used separately, but to start, you can create a new
Zap logger with all of the above included:

logger, err := zapdriver.NewProduction() // with sampling
logger, err := zapdriver.NewDevelopment() // with `development` set to `true`

The above functions give back a pointer to a zap.Logger object, so you can use
Zap like you've always done, except that it now logs in the proper
Stackdriver format.

You can also create a configuration struct, and build your logger from there:

config := zapdriver.NewProductionConfig()
config := zapdriver.NewDevelopmentConfig()

Or, get the Zapdriver encoder, and build your own configuration struct from
that:

encoder := zapdriver.NewProductionEncoderConfig()
encoder := zapdriver.NewDevelopmentEncoderConfig()

Read on to learn more about the available Stackdriver-specific log fields, and
how to use the above-mentioned components.

Special purpose logging fields

You can use the following fields to add extra information to your log entries.
These fields are parsed by Stackdriver to make it easier to query your logs or
to use the log details in the Stackdriver monitoring interface.

HTTP

You can log HTTP request/response cycles using the following field:

HTTP(req *HTTPPayload) zap.Field

You can either manually build the request payload:

req := &HTTPPayload{
  RequestMethod: "GET",
  RequestURL: "/",
  Status: 200,
}

Or, you can auto generate the struct, based on the available request and
response objects:

NewHTTP(req *http.Request, res *http.Response) *HTTPPayload

You are free to pass in nil for either the request or response object, if one
of them is unavailable to you at the point of logging. Any field depending on
one or the other will be omitted if nil is passed in.

Note that there are some fields that are not populated by either the request or
response object, and need to be set manually:

  • ServerIP string
  • Latency string
  • CacheLookup bool
  • CacheHit bool
  • CacheValidatedWithOriginServer bool
  • CacheFillBytes string

If you have no need for those fields, the quickest way to get started is like
so:

logger.Info("Request Received.", zapdriver.HTTP(zapdriver.NewHTTP(req, res)))

Label

You can add a "label" to your payload as follows:

Label(key, value string) zap.Field

Note that underwater, this sets the key to labels.<key>. You need to be using
the zapdriver.Core core for this to be converted to the proper format for
Stackdriver to recognize the labels.

See "Custom Stackdriver Zap core" for more details.

If you have a reason not to use the provided Core, you can still wrap labels in
the right labels namespace by using the available function:

Labels(fields ...zap.Field) zap.Field

Like so:

logger.Info(
  "Did something.",
  zapdriver.Labels(
    zapdriver.Label("hello", "world"),
    zapdriver.Label("hi", "universe"),
  ),
)

Again, wrapping the Label calls in Labels is not required if you use the
supplied Zap Core.

SourceLocation

You can add a source code location to your log lines to be picked up by
Stackdriver.

Note that you can set this manually, or use zapdriver.Core to automatically
add this. If you set it manually, and use zapdriver.Core, the manual call
stack will be preserved over the automated one.

SourceLocation(pc uintptr, file string, line int, ok bool) zap.Field

Note that the function signature equals that of the return values of
runtime.Caller(). This allows you to catch the stack frame at one location,
while logging it at a different location, like so:

pc, file, line, ok := runtime.Caller(0)

// do other stuff...

logger.Error("Something happened!", zapdriver.SourceLocation(pc, file, line, ok))

If you use zapdriver.Core, the above use-case is the only use-case where you
would want to manually set the source location. In all other situations, you can
simply omit this field, and it will be added automatically, using the stack
frame at the location where the log line is triggered.

If you don't use zapdriver.Core, and still want to add the source location at
the frame of the triggered log line, you'd do it like this:

logger.Error("Something happened!", zapdriver.SourceLocation(runtime.Caller(0)))

Operation

The Operation log field allows you to group log lines into a single
"operation" performed by the application:

Operation(id, producer string, first, last bool) zap.Field

For a pair of logs that belong to the same operation, you should use the same
id between them. The producer is an arbitrary identifier that should be
globally unique amongst all the logs of all your applications (meaning it should
probably be the unique name of the current application). You should set first
to true for the first log in the operation, and last to true for the final log
of the operation.

logger.Info("Started.", zapdriver.Operation("3g4d3g", "my-app", true, false))
logger.Debug("Progressing.", zapdriver.Operation("3g4d3g", "my-app", false, false))
logger.Info("Done.", zapdriver.Operation("3g4d3g", "my-app", false, true))

Pre-configured Stackdriver-optimized encoder

The Stackdriver encoder maps all Zap log levels to the appropriate
Stackdriver-supported levels:

DEBUG (100) Debug or trace information.

INFO (200) Routine information, such as ongoing status or performance.

WARNING (400) Warning events might cause problems.

ERROR (500) Error events are likely to cause problems.

CRITICAL (600) Critical events cause more severe problems or outages.

ALERT (700) A person must take an action immediately.

EMERGENCY (800) One or more systems are unusable.

It also sets some of the default keys to use the right names, such as
timestamp, severity, and message.

You can use this encoder if you want to build your Zap logger configuration
manually:

zapdriver.NewProductionEncoderConfig()

For parity-sake, there's also zapdriver.NewDevelopmentEncoderConfig(), but it
returns the exact same encoder right now.

Custom Stackdriver Zap core

A custom Zap core is included in this package to support some special use-cases.

First of all, if you use zapdriver.NewProduction() (or NewDevelopment) , you
already have this core enabled, so everything just works ™.

There are two use-cases which require this core:

  1. If you use zapdriver.Label("hello", "world"), it will initially end up in
    your log with the key labels.hello and value world. Now if you have two
    labels, you could also have labels.hi with value universe. This works as-
    is, but for this to be correctly parsed by Stackdriver as true "labels", you
    need to use the Zapdriver core, so that both of these fields get rewritten,
    to use the namespace labels, and use the keys hello and hi within that
    namespace. This is done automatically.

  2. If you don't want to use zapdriver.SourceLocation() on every log call, you
    can use this core for the source location to be automatically added to
    each log entry.

When building a logger, you can inject the Zapdriver core as follows:

zap.WrapCore(func(core zapcore.Core) zapcore.Core {
  return &zapdriver.Core{core}
})

@JeanMertz JeanMertz merged commit ea401a1 into master May 25, 2018
@JeanMertz JeanMertz deleted the version-1 branch May 25, 2018 22:48
mhindery pushed a commit to mhindery/zapdriver that referenced this pull request Sep 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant