Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs Bridge API prototype #4725

Closed
wants to merge 107 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
107 commits
Select commit Hold shift + click to select a range
a6f19fe
Add empty design doc
pellared Nov 17, 2023
c8fcf28
Init Logs Bridge API design
pellared Nov 17, 2023
e48ae14
Document the design
pellared Nov 17, 2023
31c755f
Merge branch 'main' into logs-design
pellared Nov 20, 2023
cd39d63
Add Rationale
pellared Nov 20, 2023
4578694
Move design doc closer to the code
pellared Nov 20, 2023
fbd656f
Cleanup
pellared Nov 20, 2023
9d9f10b
Cleanup
pellared Nov 20, 2023
55bad1e
Fix typo
pellared Nov 20, 2023
249f4ed
Apply suggestions from code review
pellared Nov 20, 2023
33a8aca
Clairfy RecordOption
pellared Nov 20, 2023
2e3587c
document fields that optimize attrs
pellared Nov 21, 2023
f19f0c8
Add missing parenthesis
pellared Nov 21, 2023
5456b9d
Record and NewRecord is needed for SDK
pellared Nov 21, 2023
4365860
Add usage examples
pellared Nov 21, 2023
892ee8f
Document Rejected Alternative: Reuse slog
pellared Nov 21, 2023
f288cf2
Logger.Emit with record as argument
pellared Nov 22, 2023
c0a795b
Document Logger.Emit extensibility
pellared Nov 22, 2023
2704436
Apply suggestions from code review
pellared Nov 24, 2023
a718f69
Write prototype implementation and benchmarks
pellared Nov 24, 2023
700c91f
Add missing vanity import
pellared Nov 24, 2023
2e8403e
gofumpt
pellared Nov 24, 2023
441cde8
Add TODO comments
pellared Nov 24, 2023
de09ce6
Add copywrite header and fix for Go 1.20
pellared Nov 24, 2023
dad0503
refactor
pellared Nov 24, 2023
c43bd51
Update design
pellared Nov 24, 2023
19438ca
Update design
pellared Nov 24, 2023
d4896f6
Rejected Alternative: Passing struct as parameter to LoggerProvider.L…
pellared Nov 24, 2023
cf10dd0
Merge branch 'main' into logs-design
pellared Nov 28, 2023
ddcdbda
Add go.opentelemetry.io/otel/log/benchmark module
pellared Nov 28, 2023
ca5bfb7
Add slog.Handler naive implementation
pellared Nov 28, 2023
21ca157
go mod tidy
pellared Nov 28, 2023
0d9c335
Add logr.LogSink naive implementation
pellared Nov 28, 2023
1c3776b
Update design
pellared Nov 28, 2023
5bb9801
gofumpt
pellared Nov 28, 2023
7084df2
Refactor slogHandler
pellared Nov 28, 2023
3b8a68c
Export AttributesInlineCount
pellared Nov 28, 2023
44e01f8
Merge branch 'main' into logs-design
pellared Nov 30, 2023
e7b8c7a
Rename Record.Attributes to Record.WalkAttributes
pellared Nov 30, 2023
d19ebad
Merge branch 'main' into logs-design
pellared Dec 4, 2023
ba2903a
Unexport AttributesInlineCount
pellared Dec 4, 2023
6443d19
Fix typo
pellared Dec 4, 2023
b170245
Merge branch 'main' into logs-design
pellared Dec 5, 2023
21bea0f
Ignore request cancelation
pellared Dec 5, 2023
c8b3f05
Encapsulate Record fields
pellared Dec 5, 2023
a8ba6be
Fix tests
pellared Dec 5, 2023
60a4e45
Merge branch 'main' into logs-design
pellared Dec 7, 2023
b60433a
Merge branch 'main' into logs-design
pellared Dec 11, 2023
689faff
fix typo
pellared Dec 11, 2023
7f299ff
Merge branch 'main' into logs-design
pellared Dec 15, 2023
9f9acaf
Rejected Alternative: Logger.WithAttributes
pellared Dec 15, 2023
9deedde
Fix MD009/no-trailing-spaces
pellared Dec 15, 2023
ecb503a
Merge branch 'main' into logs-design
pellared Dec 18, 2023
445bae1
Make parallel benchmarks
pellared Dec 18, 2023
cd5a898
Merge branch 'main' into logs-design
pellared Dec 19, 2023
cd8217a
Merge branch 'main' into logs-design
pellared Dec 19, 2023
977f3d7
Record attributes as field and use sync.Pool for reducing allocations
pellared Dec 22, 2023
2d161ca
Merge branch 'main' into logs-design
pellared Dec 22, 2023
11e232c
Merge branch 'main' into logs-design
pellared Dec 27, 2023
033e2e9
go mod tidy
pellared Dec 27, 2023
8c8f7ab
Trace contextg correlation design
pellared Dec 27, 2023
a79b08f
writerLogger and slogHandler handle trace context
pellared Dec 27, 2023
4be490b
go mod tidy
pellared Dec 27, 2023
3552368
logrSink handle trace context
pellared Dec 27, 2023
3fbebee
Refine API implementation section
pellared Dec 27, 2023
8e25e81
Fix grammar
pellared Dec 27, 2023
321a049
Move log/benchmark to log/internal
pellared Dec 27, 2023
2d62729
Merge branch 'main' into logs-design
pellared Jan 10, 2024
a0bc4b8
Update log package
pellared Jan 10, 2024
f6a2308
go mod tidy
pellared Jan 10, 2024
b780472
Remove DESIGN.md
pellared Jan 10, 2024
8eb2fab
Merge branch 'main' into logs-design
pellared Jan 16, 2024
d832817
Add Severity[Level]1 consts
pellared Jan 17, 2024
f78cbec
Record attributes based on slog.Record (#6)
pellared Jan 18, 2024
1a8b1c5
Merge branch 'main' into logs-design
pellared Jan 18, 2024
6730f06
go mod tidy
pellared Jan 18, 2024
d114464
The caller must not subsequently mutate the record passed to Emit
pellared Jan 18, 2024
f08ebbf
Refine Emit docs
pellared Jan 18, 2024
d665181
Merge branch 'main' into logs-design
pellared Jan 18, 2024
a116609
go mod tidy
pellared Jan 18, 2024
5d8c58f
Handle structured body and attributes (#7)
pellared Jan 24, 2024
3131463
Merge branch 'main' into logs-design
pellared Jan 25, 2024
ecc26bc
Record with pointer receivers only (#8)
pellared Jan 26, 2024
2778c55
As[Type] Value methods
pellared Jan 26, 2024
68d99a8
Use stringer for enum types
pellared Jan 26, 2024
e5dc7cf
Revert Makefile changes
pellared Jan 26, 2024
4b96c4d
Merge branch 'main' into logs-design
pellared Jan 29, 2024
4084475
Merge branch 'main' into logs-design
pellared Jan 30, 2024
d4d4274
Fix Severity.String
pellared Jan 30, 2024
91358b4
TestKind
pellared Jan 30, 2024
b90eca9
Refactor Value.String
pellared Jan 30, 2024
a02b8ea
Remove KeyValue.Invalid
pellared Feb 1, 2024
a8a1b7e
Update AddAttributes
pellared Feb 1, 2024
c4a2209
Remove header
pellared Feb 1, 2024
45cfdcb
Update comment
pellared Feb 1, 2024
b55e22e
Refine comment
pellared Feb 1, 2024
aa3b222
Merge branch 'main' into logs-design
pellared Feb 1, 2024
4dccab0
go mod tidy
pellared Feb 1, 2024
4e2e5e3
Merge branch 'logs-design' of https://github.com/pellared/opentelemet…
pellared Feb 1, 2024
a8f265c
Refactor Severity to not use iota
pellared Feb 1, 2024
f8c46f1
Merge branch 'main' into logs-design
pellared Feb 6, 2024
b996e00
Remove Record.Clone as it is not necessary
pellared Feb 7, 2024
ed67f01
Refine KeyValue docs
pellared Feb 7, 2024
5851ead
Remove unused attrsSlice
pellared Feb 7, 2024
273a964
Merge branch 'main' into logs-design
pellared Feb 7, 2024
e89444d
go mod tidy
pellared Feb 7, 2024
147762f
value does not panic and change List to Slice
pellared Feb 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,15 @@ updates:
schedule:
interval: weekly
day: sunday
- package-ecosystem: gomod
directory: /log
labels:
- dependencies
- go
- Skip Changelog
schedule:
interval: weekly
day: sunday
- package-ecosystem: gomod
directory: /metric
labels:
Expand Down
148 changes: 148 additions & 0 deletions docs/design/log-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
# Logs Bridge API

Author: Robert Pająk

Tracking issue at [#4696](https://github.com/open-telemetry/opentelemetry-go/issues/4696).

## Abstract

<!-- A short summary of the proposal. -->

We propose adding a `go.opentelemetry.io/otel/log` Go module which will provide
[Logs Bridge API](https://opentelemetry.io/docs/specs/otel/logs/bridge-api/).

## Background

<!-- An introduction of the necessary background and the problem being solved by the proposed change. -->

They key challenge is to create a well-performant API compliant with the specification.
Performance is seen as one of the most imporatant charactristics of logging libraries in Go.

## Design

This proposed design aims to:

- be specification compliant,
- have similar API to Trace and Metrics API,
- take advantage of both OpenTelemetry and `slog` experience to achieve acceptable performance.

### LoggerProvider

The [`LoggerProvider` abstraction](https://opentelemetry.io/docs/specs/otel/logs/bridge-api/#loggerprovider)
is defined as an interface.

```go
type LoggerProvider interface{
Logger(name string, options ...LoggerOption) Logger
}
```

### Logger

The [`Logger` abstraction](https://opentelemetry.io/docs/specs/otel/logs/bridge-api/#logger)
is defined as an interface.

```go
type Logger interface{
Emit(ctx context.Context, options ...RecordOption)
pellared marked this conversation as resolved.
Show resolved Hide resolved
}
```

The `Logger` has `Emit(context.Context, options ...RecordOption` method.

### Record

The [`LogRecord` abstraction](https://opentelemetry.io/docs/specs/otel/logs/bridge-api/#logger)
is defined as a struct.

```go
type Record struct {
Timestamp time.Time
ObservedTimestamp time.Time
Severity Severity
SeverityText string
Body string

// Allocation optimization: an inline array sized to hold
// the majority of log calls (based on examination of open-source
// code). It holds the start of the list of Attrs.
front [nAttrsInline]attribute.KeyValue

// The number of Attrs in front.
nFront int

// The list of Attrs except for those in front.
// Invariants:
// - len(back) > 0 iff nFront == len(front)
// - Unused array elements are zero. Used to detect mistakes.
back []attribute.KeyValue
}

const nAttrsInline = 5

type Severity int

const (
SeverityUndefined Severity = iota
SeverityTrace
SeverityTrace2
SeverityTrace3
SeverityTrace4
SeverityDebug
SeverityDebug2
SeverityDebug3
SeverityDebug4
SeverityInfo
SeverityInfo2
SeverityInfo3
SeverityInfo4
SeverityWarn
SeverityWarn2
SeverityWarn3
SeverityWarn4
SeverityError
SeverityError2
SeverityError3
SeverityError4
SeverityFatal
SeverityFatal2
SeverityFatal3
SeverityFatal4
)
```

`Record` has `Attributes` and `AddAttributes` methods,
like [`slog.Record.Attrs`](https://pkg.go.dev/log/slog#Record.Attrs)
and [`slog.Record.AddAttrs`](https://pkg.go.dev/log/slog#Record.AddAttrs),
in order to achieve high-performance when accessing and setting attributes efficiently.

The `NewRecord(...RecordOption) Record` is a factory function used to create records using provided options.

`Record` has a `Clone` method to allow copying records so that the SDK can offer concurrency safety.

## Compatibility

The backwards compatibility is achieved using the `embedded` design pattern
that is already used in Trace API and Metrics API.

## Benchmarking

The benchmarks takes inspiration from [`slog`](https://pkg.go.dev/log/slog),
because for the Go team it was also critical to create API that would be fast
and interoperable with existing logging packages. [^1] [^2]

## Rationale
pellared marked this conversation as resolved.
Show resolved Hide resolved

<!-- A discussion of alternate approaches and the trade offs, advantages, and disadvantages of the specified approach. -->

## Implementation

<!-- A description of the steps in the implementation, who will do them, and when. -->

## Open issues (if applicable)

<!-- A discussion of issues relating to this proposal for which the author does not
know the solution. This section may be omitted if there are none. -->

[^1]: Jonathan Amsterdam, [The Go Blog: Structured Logging with slog](https://go.dev/blog/slog)
[^2]: Jonathan Amsterdam, [GopherCon Europe 2023: A Fast Structured Logging Package](https://www.youtube.com/watch?v=tC4Jt3i62ns)
30 changes: 30 additions & 0 deletions log/bench_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

// Copyright 2022 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package log_test

import "testing"

// These benchmarks are based on slog/internal/benchmarks.
// They have the following desirable properties:
//
// - They test a complete log record, from the user's call to its return.
//
// - The benchmarked code is run concurrently in multiple goroutines, to
// better simulate a real server (the most common environment for structured
// logs).
//
// - Some handlers are optimistic versions of real handlers, doing real-world
// tasks as fast as possible (and sometimes faster, in that an
// implementation may not be concurrency-safe). This gives us an upper bound
// on handler performance, so we can evaluate the (handler-independent) core
// activity of the package in an end-to-end context without concern that a
// slow handler implementation is skewing the results.
func BenchmarkEndToEnd(b *testing.B) {
// TODO: Replicate https://github.com/golang/go/blob/master/src/log/slog/internal/benchmarks/benchmarks_test.go
// Run benchmarks against a "noop.Logger" and "fastTextLogger" (based on fastTextHandler)
}
5 changes: 5 additions & 0 deletions log/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

// Package log provides the OpenTelemetry Logs Data Model and Bridge API.
package log // import "go.opentelemetry.io/otel/log"
8 changes: 8 additions & 0 deletions log/example_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package log_test

func Example() {
// Output:
}
3 changes: 3 additions & 0 deletions log/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module go.opentelemetry.io/otel/log

go 1.20
1 change: 1 addition & 0 deletions versions.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,4 @@ module-sets:
- go.opentelemetry.io/otel/schema
excluded-modules:
- go.opentelemetry.io/otel/internal/tools
- go.opentelemetry.io/otel/log