Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics support #94

Merged
merged 8 commits into from
Jul 3, 2018
Merged

Metrics support #94

merged 8 commits into from
Jul 3, 2018

Conversation

axw
Copy link
Member

@axw axw commented May 22, 2018

We introduce support for publishing application metrics to the APM server. The server-side endpoint is still under development, so this is disabled by default. You can enable metrics sending by setting the collection interval, e.g. ELASTIC_APM_METRICS_INTERVAL=30s

The tracer has a builtin metrics gatherer which reports some basic process statistics (memory usage, garbage collection stats), and tracer statistics (number of transactions and errors sent, dropped, and failed sending.) We'll extend this over time, e.g. process CPU usage. There will be some overlap with metricbeat.

Supersedes #31.

Closes #67.

@axw axw force-pushed the metrics branch 3 times, most recently from 2cbe000 to dbb7206 Compare May 22, 2018 06:28
@codecov-io
Copy link

codecov-io commented May 22, 2018

Codecov Report

Merging #94 into master will increase coverage by 0.56%.
The diff coverage is 81.02%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #94      +/-   ##
==========================================
+ Coverage   75.37%   75.94%   +0.56%     
==========================================
  Files          57       61       +4     
  Lines        3099     3558     +459     
==========================================
+ Hits         2336     2702     +366     
- Misses        581      660      +79     
- Partials      182      196      +14
Impacted Files Coverage Δ
tracer_stats.go 100% <ø> (ø)
model/maps.go 0% <0%> (ø) ⬆️
transport/debug.go 0% <0%> (ø) ⬆️
env.go 75.55% <100%> (-0.54%) ⬇️
builtin_metrics.go 100% <100%> (ø)
transport/http.go 83.47% <12.5%> (-5.03%) ⬇️
transport/discard.go 33.33% <25%> (-16.67%) ⬇️
model/marshal.go 65.4% <45%> (-2.69%) ⬇️
tracer.go 84.34% <80.72%> (-2.05%) ⬇️
sender.go 87.7% <82.14%> (-1.66%) ⬇️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d60d5e...625c57f. Read the comment docs.

sender.go Outdated
System: s.tracer.system,
Metrics: s.metrics.metrics,
}
if err := s.tracer.Transport.SendMetrics(ctx, &payload); err != nil {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can race with the other Send methods in the main tracer goroutine. We should probably make it so SendMetrics is called from the tracer goroutine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by splitting the method into two: gatherMetrics and sendMetrics; gatherMetrics kicks off goroutines to gather metrics and sends on a channel when they're all done, sendMetrics is called afterwards to send from the main tracer goroutine.

@axw axw force-pushed the metrics branch 2 times, most recently from 90fa286 to dcca228 Compare June 12, 2018 05:48
@axw axw mentioned this pull request Jun 12, 2018
@axw
Copy link
Member Author

axw commented Jun 13, 2018

Testing with elastic/apm-server#1000, slightly modified to accept "unit" in the samples:

image

@axw axw force-pushed the metrics branch 2 times, most recently from e8de2a8 to 18d2c74 Compare June 22, 2018 13:16
@axw axw force-pushed the metrics branch 6 times, most recently from 457702f to f60dcf1 Compare June 25, 2018 07:51
@axw axw force-pushed the metrics branch 3 times, most recently from 810bc55 to acad4ca Compare July 3, 2018 03:18
axw added 6 commits July 3, 2018 12:15
MetricsGatherer is an interface for
gathering metrics. You add one to a
Tracer using AddMetricsGatherer, and
it is periodically called to gather
all the metrics. Once they are
gathered, they are sent in a payload
to the Elastic APM server.

Mostly for testing purposes, you
can force metrics to be gathered and
sent at any given time by calling
Tracer.SendMetrics. It has the same
sort of behaviour as Tracer.Flush,
blocking until the metrics are sent
or the abort channel is signaled.

Each tracer is created with a builtin
metrics gatherer, which gathers basic
process metrics like memstats, number
of goroutines, and the like.
Package apmprometheus provides a method for
wrapping a prometheus.Gatherer to adapt it
to the elasticapm.MetricsGatherer interface.
Untyped, Gauge and Counter metric types are
supported; Histogram and and Summary metrics
are silently ignored for now.
Add API for adding summary metrics.
Also add go.mem.gc builtin summary
metric, which exposes GC pause
quantiles, count, and sum (total).
@axw axw merged commit 7c25a1a into elastic:master Jul 3, 2018
@axw axw deleted the metrics branch July 3, 2018 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants