Adds profiles as another signal #1

petethepig · 2023-06-14T22:17:40Z

This PR is my attempt at bringing profiling signal to the collector. It depends on proto defenitions from open-telemetry/opentelemetry-proto repo.

Most of this code is boilerplate, there's a handful of interesting files:

profiles.proto in opentelemetry-proto repo (link)
pdata/internal/cmd/pdatagen/internal/pprofile_package.go connects everything to pdata and needs to be updated manually when proto changes (link)
pdata/pprofile/json.go implements json unmarshaling and needs to be updated manually when proto changes (link)

Workflow I use to test this right now:

Run make genproto to generate go bindings
- (I replace proto version from git with a local one using rsync -av ../opentelemetry-proto/ ${OPENTELEMETRY_PROTO_SRC_DIR})
Run make genpdata to generate more pdata bindings
Run make otelcorecol to verify that collector compiles
Run ./bin/otelcorecol_darwin_arm64 --config=file:tmp/config.yaml (config is here)
Run curl -i -X POST http://localhost:4318/v1/profiles -H 'Content-Type: application/json' -d @profile.json (profile.json is here)
^ This prints the profile in text form

To run benchmarks:

Clone this repo, make sure you're on the right branch (profiling). Run the following command:

make profile-benchmark

If you want to make changes to the proto:

make sure you have open-telemetry/opentelemetry-proto-profile repo checkout out in an adjacent directory (../opentelemetry-proto/)
The most recent version of that repo does not have alternative represenations, so you will need to checkout a specific commit: git checkout 622c1658673283102a9429109185615bfcfaa78e
make changes to the profiling proto in ../opentelemetry-proto/
run make genproto to generate protobuf bindings
run make genpdata to generate pdata bindings

You can now run benchmarks again:

make profile-benchmark

Benchmarks live in /pdata/pprofile directory. The files that you might want to look at are:

/pdata/pprofile/conversion.go
/pdata/pprofile/benchmark_test.go

This is second version of the Profiling Data Model OTEP. After [we've gotten feedback from the greater OTel community](#237) we went back to the drawing board and came up with a new version of the data model. The main difference between the two versions is that the new version is more similar to the original pprof format, which makes it easier to understand and implement. It also has better performance characteristics. We've also incorporated a lot of the feedback we've gotten on the first PR into this OTEP. Some minor details about the data model are still being discussed and will be flushed out in the future OTEPs. We intend to finalize these details after doing experiments with early versions of working client + collector + backend implementations and getting feedback from the community. The goal of this OTEP is to provide a solid foundation for these experiments. So far we've done a number of things to validate it: * we've written a new profiles proto described in this OTEP * we've documented decisions made along the way in a [decision log](https://github.com/open-telemetry/opentelemetry-proto-profile/blob/main/opentelemetry/proto/profiles/v1/decision-log.md) * we've done benchmarking to refine the data representation (see Benchmarking section in a [collector PR](petethepig/opentelemetry-collector#1)) * diff between original pprof and the new proto: [link](open-telemetry/opentelemetry-proto-profile@2cf711b...petethepig:opentelemetry-proto:pprof-experiments#diff-9cb689ea05ecfd2edffc39869eca3282a3f2f45a8e1aa21624b452fa5362d1d2) We're seeking feedback and hoping to get this approved. --- For (a lot) more details, see: * [OTel Profiling SIG Meeting Notes](https://docs.google.com/document/d/19UqPPPlGE83N37MhS93uRlxsP1_wGxQ33Qv6CDHaEp0/edit) --------- Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Felix Geisendörfer <felix@felixge.de> Co-authored-by: Reiley Yang <reyang@microsoft.com>

To resolve the govulncheck reports: ``` Vulnerability #1: GO-2023-1987 Large RSA keys can cause high CPU usage in crypto/tls More info: https://pkg.go.dev/vuln/GO-2023-1987 Standard library Found in: crypto/tls@go1.19.11 Fixed in: crypto/tls@go1.21rc4 Example traces found: Error: #1: service/internal/proctelemetry/config.go:299:27: proctelemetry.initOTLPgRPCExporter calls otlpmetricgrpc.New, which eventually calls tls.Conn.Handshake Error: open-telemetry#2: service/internal/proctelemetry/config.go:156:39: proctelemetry.InitPrometheusServer calls http.Server.ListenAndServe, which eventually calls tls.Conn.HandshakeContext Error: open-telemetry#3: service/service.go:251:36: service.buildResource calls uuid.NewRandom, which eventually calls tls.Conn.Read Error: open-telemetry#4: service/config.go:35:13: service.Config.Validate calls fmt.Printf, which eventually calls tls.Conn.Write Error: #5: service/telemetry/telemetry.go:32:28: telemetry.Telemetry.Shutdown calls trace.TracerProvider.Shutdown, which eventually calls tls.Dialer.DialContext ``` https://github.com/open-telemetry/opentelemetry-collector/actions/runs/5753675727/job/15597394973?pr=8144

This is second version of the Profiling Data Model OTEP. After [we've gotten feedback from the greater OTel community](open-telemetry/oteps#237) we went back to the drawing board and came up with a new version of the data model. The main difference between the two versions is that the new version is more similar to the original pprof format, which makes it easier to understand and implement. It also has better performance characteristics. We've also incorporated a lot of the feedback we've gotten on the first PR into this OTEP. Some minor details about the data model are still being discussed and will be flushed out in the future OTEPs. We intend to finalize these details after doing experiments with early versions of working client + collector + backend implementations and getting feedback from the community. The goal of this OTEP is to provide a solid foundation for these experiments. So far we've done a number of things to validate it: * we've written a new profiles proto described in this OTEP * we've documented decisions made along the way in a [decision log](https://github.com/open-telemetry/opentelemetry-proto-profile/blob/main/opentelemetry/proto/profiles/v1/decision-log.md) * we've done benchmarking to refine the data representation (see Benchmarking section in a [collector PR](petethepig/opentelemetry-collector#1)) * diff between original pprof and the new proto: [link](open-telemetry/opentelemetry-proto-profile@2cf711b...petethepig:opentelemetry-proto:pprof-experiments#diff-9cb689ea05ecfd2edffc39869eca3282a3f2f45a8e1aa21624b452fa5362d1d2) We're seeking feedback and hoping to get this approved.

This is second version of the Profiling Data Model OTEP. After [we've gotten feedback from the greater OTel community](open-telemetry#237) we went back to the drawing board and came up with a new version of the data model. The main difference between the two versions is that the new version is more similar to the original pprof format, which makes it easier to understand and implement. It also has better performance characteristics. We've also incorporated a lot of the feedback we've gotten on the first PR into this OTEP. Some minor details about the data model are still being discussed and will be flushed out in the future OTEPs. We intend to finalize these details after doing experiments with early versions of working client + collector + backend implementations and getting feedback from the community. The goal of this OTEP is to provide a solid foundation for these experiments. So far we've done a number of things to validate it: * we've written a new profiles proto described in this OTEP * we've documented decisions made along the way in a [decision log](https://github.com/open-telemetry/opentelemetry-proto-profile/blob/main/opentelemetry/proto/profiles/v1/decision-log.md) * we've done benchmarking to refine the data representation (see Benchmarking section in a [collector PR](petethepig/opentelemetry-collector#1)) * diff between original pprof and the new proto: [link](open-telemetry/opentelemetry-proto-profile@2cf711b...petethepig:opentelemetry-proto:pprof-experiments#diff-9cb689ea05ecfd2edffc39869eca3282a3f2f45a8e1aa21624b452fa5362d1d2) We're seeking feedback and hoping to get this approved. --- For (a lot) more details, see: * [OTel Profiling SIG Meeting Notes](https://docs.google.com/document/d/19UqPPPlGE83N37MhS93uRlxsP1_wGxQ33Qv6CDHaEp0/edit) --------- Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Felix Geisendörfer <felix@felixge.de> Co-authored-by: Reiley Yang <reyang@microsoft.com>

This is second version of the Profiling Data Model OTEP. After [we've gotten feedback from the greater OTel community](open-telemetry/oteps#237) we went back to the drawing board and came up with a new version of the data model. The main difference between the two versions is that the new version is more similar to the original pprof format, which makes it easier to understand and implement. It also has better performance characteristics. We've also incorporated a lot of the feedback we've gotten on the first PR into this OTEP. Some minor details about the data model are still being discussed and will be flushed out in the future OTEPs. We intend to finalize these details after doing experiments with early versions of working client + collector + backend implementations and getting feedback from the community. The goal of this OTEP is to provide a solid foundation for these experiments. So far we've done a number of things to validate it: * we've written a new profiles proto described in this OTEP * we've documented decisions made along the way in a [decision log](https://github.com/open-telemetry/opentelemetry-proto-profile/blob/main/opentelemetry/proto/profiles/v1/decision-log.md) * we've done benchmarking to refine the data representation (see Benchmarking section in a [collector PR](petethepig/opentelemetry-collector#1)) * diff between original pprof and the new proto: [link](open-telemetry/opentelemetry-proto-profile@2cf711b...petethepig:opentelemetry-proto:pprof-experiments#diff-9cb689ea05ecfd2edffc39869eca3282a3f2f45a8e1aa21624b452fa5362d1d2) We're seeking feedback and hoping to get this approved. --- For (a lot) more details, see: * [OTel Profiling SIG Meeting Notes](https://docs.google.com/document/d/19UqPPPlGE83N37MhS93uRlxsP1_wGxQ33Qv6CDHaEp0/edit) --------- Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Felix Geisendörfer <felix@felixge.de> Co-authored-by: Reiley Yang <reyang@microsoft.com>

This is second version of the Profiling Data Model OTEP. After [we've gotten feedback from the greater OTel community](open-telemetry#237) we went back to the drawing board and came up with a new version of the data model. The main difference between the two versions is that the new version is more similar to the original pprof format, which makes it easier to understand and implement. It also has better performance characteristics. We've also incorporated a lot of the feedback we've gotten on the first PR into this OTEP. Some minor details about the data model are still being discussed and will be flushed out in the future OTEPs. We intend to finalize these details after doing experiments with early versions of working client + collector + backend implementations and getting feedback from the community. The goal of this OTEP is to provide a solid foundation for these experiments. So far we've done a number of things to validate it: * we've written a new profiles proto described in this OTEP * we've documented decisions made along the way in a [decision log](https://github.com/open-telemetry/opentelemetry-proto-profile/blob/main/opentelemetry/proto/profiles/v1/decision-log.md) * we've done benchmarking to refine the data representation (see Benchmarking section in a [collector PR](petethepig/opentelemetry-collector#1)) * diff between original pprof and the new proto: [link](open-telemetry/opentelemetry-proto-profile@2cf711b...petethepig:opentelemetry-proto:pprof-experiments#diff-9cb689ea05ecfd2edffc39869eca3282a3f2f45a8e1aa21624b452fa5362d1d2) We're seeking feedback and hoping to get this approved. --- For (a lot) more details, see: * [OTel Profiling SIG Meeting Notes](https://docs.google.com/document/d/19UqPPPlGE83N37MhS93uRlxsP1_wGxQ33Qv6CDHaEp0/edit) --------- Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Felix Geisendörfer <felix@felixge.de> Co-authored-by: Reiley Yang <reyang@microsoft.com>

This is second version of the Profiling Data Model OTEP. After [we've gotten feedback from the greater OTel community](open-telemetry/oteps#237) we went back to the drawing board and came up with a new version of the data model. The main difference between the two versions is that the new version is more similar to the original pprof format, which makes it easier to understand and implement. It also has better performance characteristics. We've also incorporated a lot of the feedback we've gotten on the first PR into this OTEP. Some minor details about the data model are still being discussed and will be flushed out in the future OTEPs. We intend to finalize these details after doing experiments with early versions of working client + collector + backend implementations and getting feedback from the community. The goal of this OTEP is to provide a solid foundation for these experiments. So far we've done a number of things to validate it: * we've written a new profiles proto described in this OTEP * we've documented decisions made along the way in a [decision log](https://github.com/open-telemetry/opentelemetry-proto-profile/blob/main/opentelemetry/proto/profiles/v1/decision-log.md) * we've done benchmarking to refine the data representation (see Benchmarking section in a [collector PR](petethepig/opentelemetry-collector#1)) * diff between original pprof and the new proto: [link](open-telemetry/opentelemetry-proto-profile@2cf711b...petethepig:opentelemetry-proto:pprof-experiments#diff-9cb689ea05ecfd2edffc39869eca3282a3f2f45a8e1aa21624b452fa5362d1d2) We're seeking feedback and hoping to get this approved. --- For (a lot) more details, see: * [OTel Profiling SIG Meeting Notes](https://docs.google.com/document/d/19UqPPPlGE83N37MhS93uRlxsP1_wGxQ33Qv6CDHaEp0/edit) --------- Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Felix Geisendörfer <felix@felixge.de> Co-authored-by: Reiley Yang <reyang@microsoft.com>

petethepig added 12 commits June 13, 2023 22:16

chore: adds basic profiling support

7c2fc48

uses an empty profile proto

da10c9b

fixes

5a41d99

wip

7a00f04

wip

628905a

wip

8fec712

wip

9e56312

wip

82cbb23

wip

5b0febd

wip

f62e78b

wip

d180b5a

wip

897f657

petethepig mentioned this pull request Jun 27, 2023

Add profiles proto open-telemetry/opentelemetry-proto#488

Closed

This was referenced Jul 6, 2023

Adds Profiling Signal open-telemetry/opentelemetry-proto-profile#2

Closed

Adds Profiling Signal open-telemetry/opentelemetry-proto-profile#3

Merged

petethepig added 4 commits July 25, 2023 21:12

improvements

5bf5ffc

adds benchmarking code

fb7fc48

otlp fix

5eb4b0f

conversion + benchmark code

2e4260d

This was referenced Jul 27, 2023

feat(otlp): experimental otlp support grafana/pyroscope-go#36

Draft

feat(otlp): experimental otlp support grafana/pyroscope#2177

Merged

petethepig added 5 commits July 26, 2023 21:16

adds a dockerized example

9c7cf43

jfr and ruby benchmarks

233adf3

more benchmarks

cf5ddf9

more benchmarking, adds some examples

b0977ad

improvements

4d5897a

petethepig mentioned this pull request Sep 7, 2023

Introduces Profiling Data Model open-telemetry/oteps#237

Closed

petethepig added 3 commits October 4, 2023 22:17

pprofextended snapshot

40bc740

improvements

ed1960f

separate array for locations

cfdffe1

petethepig added 2 commits October 18, 2023 17:40

more bechmarking

cd9041f

panic fix

8ba53ee

petethepig merged commit 978fe04 into main Oct 23, 2023

petethepig mentioned this pull request Nov 2, 2023

Introduces Profiling Data Model v2 open-telemetry/oteps#239

Merged

Lincyaw mentioned this pull request Apr 17, 2024

Is there any otel collector version that supports this new feature? open-telemetry/opentelemetry-ebpf-profiler#12

Closed

florianl mentioned this pull request Aug 12, 2024

Benchmarking changes of the wire protocol open-telemetry/opentelemetry-ebpf-profiler#110

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds profiles as another signal #1

Adds profiles as another signal #1

petethepig commented Jun 14, 2023 •

edited

Loading

Adds profiles as another signal #1

Adds profiles as another signal #1

Conversation

petethepig commented Jun 14, 2023 • edited Loading

To run benchmarks:

petethepig commented Jun 14, 2023 •

edited

Loading