Service metrics txn events txn metrics #9819

carsonip · 2022-12-15T15:27:51Z

Motivation/summary

Add event.success_count to transaction events, transaction metrics and service metrics. It was previously available for service metrics at transaction.success_count by #9791

Checklist

Update CHANGELOG.asciidoc
Update package changelog.yml (only if changes to apmpackage have been made)
Documentation has been updated

How to test these changes

Check that event.success_count is correct for transaction event, transaction metrics and service metrics
Check that they work with avg aggregation
Ensure event.success_count.sum is rounded under sampling setup (fixed in Round servicetxmetrics event.success_count.sum #10165)

Related issues

Closes #5243

mergify · 2022-12-15T15:28:24Z

This pull request is now in conflicts. Could you fix it @carsonip? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b service-metrics-txn-events-txn-metrics upstream/service-metrics-txn-events-txn-metrics
git merge upstream/main
git push upstream service-metrics-txn-events-txn-metrics

mergify · 2022-12-15T15:28:25Z

This pull request does not have a backport label. Could you fix it @carsonip? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-7.x is the label to automatically backport to the 7.x branch.
backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

apmmachine · 2022-12-16T10:51:39Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-27T11:47:28.058+0000
Duration: 20 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	154
Skipped	0
Total	154

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate and publish the docker images.
/test windows : Build & tests on Windows.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

apmmachine · 2022-12-16T10:51:42Z

📚 Go benchmark report

Diff with the main branch

name                                          old time/op    new time/op    delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
ContextReset/X-Forwarded-For_ipv6-12             919ns ± 4%     689ns ±26%  -25.07%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ReadEvents/nop_codec/100_events-12               154µs ± 9%     136µs ± 5%  -11.69%  (p=0.016 n=5+4)
IsTraceSampled/unknown-12                        369ns ± 3%     379ns ± 1%   +2.57%  (p=0.032 n=5+5)

name                                          old alloc/op   new alloc/op   delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ReadEvents/json_codec/1_events-12               9.04kB ± 0%    9.03kB ± 0%   -0.21%  (p=0.000 n=4+5)
ReadEvents/json_codec/10_events-12              83.4kB ± 0%    83.3kB ± 0%   -0.18%  (p=0.008 n=5+5)
ReadEvents/json_codec/100_events-12              830kB ± 0%     829kB ± 0%   -0.18%  (p=0.008 n=5+5)
ReadEvents/json_codec/199_events-12             1.10MB ± 0%    1.10MB ± 0%   -0.31%  (p=0.008 n=5+5)
ReadEvents/json_codec_big_tx/1_events-12        11.1kB ± 0%    11.0kB ± 0%   -0.14%  (p=0.016 n=5+4)
ReadEvents/json_codec_big_tx/10_events-12        104kB ± 0%     103kB ± 0%   -0.17%  (p=0.008 n=5+5)
ReadEvents/json_codec_big_tx/100_events-12      1.03MB ± 0%    1.03MB ± 0%   -0.20%  (p=0.008 n=5+5)
ReadEvents/json_codec_big_tx/199_events-12      1.43MB ± 0%    1.42MB ± 0%   -0.25%  (p=0.016 n=5+4)
ReadEvents/json_codec_big_tx/399_events-12      2.22MB ± 0%    2.22MB ± 0%   -0.22%  (p=0.008 n=5+5)
ReadEvents/nop_codec/10_events-12               25.7kB ± 0%    25.7kB ± 0%   -0.14%  (p=0.048 n=5+5)

name                                          old allocs/op  new allocs/op  delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

carsonip · 2022-12-16T13:11:05Z

Keeping this in draft during review since it will have to go with elastic/apm-data#2

axw

Looks great! Just one minor comment.

apmpackage/apm/data_stream/traces/elasticsearch/ingest_pipeline/default.yml

mergify · 2022-12-19T05:41:10Z

This pull request is now in conflicts. Could you fix it @carsonip? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b service-metrics-txn-events-txn-metrics upstream/service-metrics-txn-events-txn-metrics
git merge upstream/main
git push upstream service-metrics-txn-events-txn-metrics

simitt

Code mostly looks good, left a couple minor comments.

Please reflect that this PR depends on elastic/apm-data#2 in the description.

apmpackage/apm/data_stream/internal_metrics/manifest.yml

x-pack/apm-server/aggregation/txmetrics/aggregator.go

simitt · 2022-12-20T08:45:50Z

changelogs/head.asciidoc

@@ -11,6 +11,7 @@ https://github.com/elastic/apm-server/compare/8.5\...main[View commits]
 - `observer.id` and `observer.ephemeral_id` are no longer added to APM documents {pull}9412[9412]
 - `timeseries.instance` has been removed from transaction metrics docs; it was never used {pull}9565[9565]
 - `transaction.failure_count` has been removed. `transaction.success_count` type has changed to `aggregated_metric_double` {pull}9791[9791]
+- `transaction.success_count` has been moved to `event.success_count` {pull}9819[9819]


The changelog is missing that event.success_count has been introduced also for transaction events and transaction metrics. This changelog should contain all changes made to APM Server.

Updated changelog, but split the part about transaction events and transaction metrics outside "breaking changes".

apmpackage/apm/data_stream/traces/elasticsearch/ingest_pipeline/default.yml

mergify · 2022-12-23T09:12:34Z

This pull request is now in conflicts. Could you fix it @carsonip? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b service-metrics-txn-events-txn-metrics upstream/service-metrics-txn-events-txn-metrics
git merge upstream/main
git push upstream service-metrics-txn-events-txn-metrics

simitt · 2022-12-23T13:51:19Z

@carsonip the PR looks pretty solid - are you planning on marking it ready for review soon?

carsonip · 2022-12-23T15:45:13Z

@carsonip the PR looks pretty solid - are you planning on marking it ready for review soon?

Was trying to squash locally before making it ready for review but got into some local permission errors which I thought we fixed yesterday. Anyway, can do a squash merge after approval.

x-pack/apm-server/aggregation/txmetrics/aggregator.go

Co-authored-by: Marc Lopez Rubio <marc5.12@outlook.com>

simitt · 2022-12-27T12:51:52Z

Merged, to unblock @marclop who is working on a PR touching the same code base.

carsonip · 2022-12-27T14:00:13Z

Thanks @simitt

lahsivjar · 2023-03-06T07:34:17Z

Tested with 8.7.0-BC4. The code used for testing generates data as following:

2 transactions for each unique name and transaction type.
A total of 100 unique transactions (200 total transactions based on point 1).
Equal number of unique transactions for success and failure.

Code used for generating traces for testing

package main

import (
	"fmt"
	"time"

	"go.elastic.co/apm/v2"
)

func main() {
	tracer := apm.DefaultTracer()
	tracer.SetSpanCompressionEnabled(false)             // Disable compression.
	tracer.SetExitSpanMinDuration(2 * time.Millisecond) // set duration to 2ms.
	for i := 0; i < 100; i++ {
		for _, duration := range []time.Duration{100, 1000} {
			tx := tracer.StartTransaction(fmt.Sprintf("tx-%d", i), fmt.Sprintf("type-%d", i))
			tx.Duration = time.Duration(duration) * time.Millisecond
			if i%2 == 0 {
				tx.Outcome = "success"
			} else {
				tx.Outcome = "failure"
			}
			tx.End()
		}
	}
	tracer.Flush(nil)
}

Check that event.success_count is correct for transaction event, transaction metrics and service metrics

Transaction Events

Asserted that the metric is recorded as byte.

Success

GET /traces-apm-default*/_search?filter_path=hits.hits
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {"match": {"event.outcome": "success"}},
        {"match": {"service.name": "apm2"}}
      ]
    }
  }, 
  "_source": [false],
  "fields": [
    "event.success_count"
  ]
}

{
  "hits": {
    "hits": [
      {
        "_index": ".ds-traces-apm-default-2023.03.06-000001",
        "_id": "flwTtYYBNtUXE48ncG4T",
        "_score": 4.617365,
        "_source": {},
        "fields": {
          "event.success_count": [
            1
          ]
        }
      }
    ]
  }
}

Failure

GET /traces-apm-default*/_search?filter_path=hits.hits
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {"match": {"event.outcome": "failure"}},
        {"match": {"service.name": "apm2"}}
      ]
    }
  }, 
  "_source": [false],
  "fields": [
    "event.success_count"
  ]
}

{
  "hits": {
    "hits": [
      {
        "_index": ".ds-traces-apm-default-2023.03.06-000001",
        "_id": "gFwTtYYBNtUXE48ncG4T",
        "_score": 9.915409,
        "_source": {},
        "fields": {
          "event.success_count": [
            0
          ]
        }
      }
    ]
  }
}

Transaction Metrics

Asserted that the metric is recorded as aggregate_metric_double with sum and value_count metrics.

Success

GET /metrics-apm.transaction*/_search?filter_path=hits.hits
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {"match": {"event.outcome": "success"}},
        {"match": {"service.name": "apm2"}}
      ]
    }
  }, 
  "_source": [false],
  "fields": [
    "event.success_count"
  ]
}

{
  "hits": {
    "hits": [
      {
        "_index": ".ds-metrics-apm.transaction.1m-default-2023.03.06-000001",
        "_id": "aFwUtYYBNtUXE48nDm9C",
        "_score": 0.69894236,
        "_source": {},
        "fields": {
          "event.success_count": [
            {
              "sum": 2,
              "value_count": 2
            }
          ]
        }
      }
    ]
  }
}

Failure

GET /metrics-apm.transaction*/_search?filter_path=hits.hits
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {"match": {"event.outcome": "failure"}},
        {"match": {"service.name": "apm2"}}
      ]
    }
  }, 
  "_source": [false],
  "fields": [
    "event.success_count"
  ]
}

{
  "hits": {
    "hits": [
      {
        "_index": ".ds-metrics-apm.transaction.1m-default-2023.03.06-000001",
        "_id": "alwUtYYBNtUXE48nDm9C",
        "_score": 0.7566507,
        "_source": {},
        "fields": {
          "event.success_count": [
            {
              "sum": 0,
              "value_count": 2
            }
          ]
        }
      }
    ]
  }
}

Service Transaction Metrics

Asserted that the metric is recorded as aggregate_metric_double with sum and value_count metrics.

Success

GET /metrics-apm.service_transaction*/_search?filter_path=hits.hits
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {"match": {"transaction.type": "type-2"}},
        {"match": {"service.name": "apm2"}}
      ]
    }
  }, 
  "_source": [false],
  "fields": [
    "event.success_count"
  ]
}

{
  "hits": {
    "hits": [
      {
        "_index": ".ds-metrics-apm.service_transaction.1m-default-2023.03.06-000001",
        "_id": "olwUtYYBNtUXE48nDm9C",
        "_score": 4.2731586,
        "_source": {},
        "fields": {
          "event.success_count": [
            {
              "sum": 2,
              "value_count": 2
            }
          ]
        }
      }
    ]
  }
}

Failure

GET /metrics-apm.service_transaction*/_search?filter_path=hits.hits
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {"match": {"transaction.type": "type-1"}},
        {"match": {"service.name": "apm2"}}
      ]
    }
  }, 
  "_source": [false],
  "fields": [
    "event.success_count"
  ]
}

{
  "hits": {
    "hits": [
      {
        "_index": ".ds-metrics-apm.service_transaction.1m-default-2023.03.06-000001",
        "_id": "l1wUtYYBNtUXE48nDm9C",
        "_score": 4.2731586,
        "_source": {},
        "fields": {
          "event.success_count": [
            {
              "sum": 0,
              "value_count": 2
            }
          ]
        }
      }
    ]
  }
}

Check that they work with avg aggregation

Transaction Events

GET /traces-apm-default*/_search?filter_path=aggregations
{
  "size": 1,
  "aggs": {
    "err_rate": {
      "filter": {
        "term": {
          "service.name": "apm2"
        }
      },
      "aggs": {
        "err_rate": {"avg": { "field": "event.success_count" }}
      }
    }
  }
}

{
  "aggregations": {
    "err_rate": {
      "doc_count": 200,
      "err_rate": {
        "value": 0.5
      }
    }
  }
}

Transaction Metrics

GET /metrics-apm.transaction*/_search?filter_path=aggregations
{
  "size": 1,
  "aggs": {
    "err_rate": {
      "filter": {
        "term": {
          "service.name": "apm2"
        }
      },
      "aggs": {
        "err_rate": {"avg": { "field": "event.success_count" }}
      }
    }
  }
}

{
  "aggregations": {
    "err_rate": {
      "doc_count": 200,
      "err_rate": {
        "value": 0.5
      }
    }
  }
}

Service Transaction Metrics

GET /metrics-apm.service_transaction*/_search?filter_path=aggregations
{
  "size": 1,
  "aggs": {
    "err_rate": {
      "filter": {
        "term": {
          "service.name": "apm2"
        }
      },
      "aggs": {
        "err_rate": {"avg": { "field": "event.success_count" }}
      }
    }
  }
}

{
  "aggregations": {
    "err_rate": {
      "doc_count": 200,
      "err_rate": {
        "value": 0.5
      }
    }
  }
}

Ensure event.success_count.sum is rounded under sampling setup

Asserted via test case.

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Dec 15, 2022

carsonip force-pushed the service-metrics-txn-events-txn-metrics branch 2 times, most recently from 190900f to ceeca21 Compare December 16, 2022 10:30

carsonip force-pushed the service-metrics-txn-events-txn-metrics branch from 633ae5a to abfbdbd Compare December 16, 2022 11:17

carsonip requested a review from axw December 16, 2022 12:58

carsonip mentioned this pull request Dec 16, 2022

Move transaction.success_count to event.success_count elastic/apm-data#2

Merged

carsonip force-pushed the service-metrics-txn-events-txn-metrics branch from 28cef68 to a0200d6 Compare December 16, 2022 14:00

axw reviewed Dec 19, 2022

View reviewed changes

apmpackage/apm/data_stream/traces/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

carsonip force-pushed the service-metrics-txn-events-txn-metrics branch from a0200d6 to 5a4373a Compare December 19, 2022 09:35

axw approved these changes Dec 20, 2022

View reviewed changes

simitt reviewed Dec 20, 2022

View reviewed changes

carsonip added 12 commits December 23, 2022 11:29

Transaction event event.success_count

132e828

Add transaction metrics event.success_count

d93c312

Rename to event.success_count for servicemetrics

c24b972

Update docs

c8b1f88

Bump apm-data dependency

e300b0e

Make linter happy

c52a4c3

Bump apm-data

cd264d4

Inline event.success_count ingest pipeline

67fd2c4

Ingest pipeline ignore_failure

1ff8ee5

Bump apm-data

0b52533

Update changelog

665eca4

Fix merge error

47c03e4

carsonip force-pushed the service-metrics-txn-events-txn-metrics branch from 641e5db to 47c03e4 Compare December 23, 2022 12:16

carsonip marked this pull request as ready for review December 23, 2022 15:43

carsonip requested a review from simitt December 23, 2022 15:45

Merge branch 'main' into service-metrics-txn-events-txn-metrics

ddbf7ac

marclop approved these changes Dec 27, 2022

View reviewed changes

x-pack/apm-server/aggregation/txmetrics/aggregator.go Outdated Show resolved Hide resolved

Update x-pack/apm-server/aggregation/txmetrics/aggregator.go

1817be1

Co-authored-by: Marc Lopez Rubio <marc5.12@outlook.com>

simitt approved these changes Dec 27, 2022

View reviewed changes

Merge branch 'main' into service-metrics-txn-events-txn-metrics

975c94d

simitt merged commit b470c3b into elastic:main Dec 27, 2022

lahsivjar added test-plan v8.7.0 labels Feb 9, 2023

marclop mentioned this pull request Feb 20, 2023

Store outcome values in servicemetrics transaction.success_count #9791

Merged

3 tasks

lahsivjar self-assigned this Feb 22, 2023

lahsivjar added the test-plan-ok label Mar 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Service metrics txn events txn metrics #9819

Service metrics txn events txn metrics #9819

carsonip commented Dec 15, 2022 •

edited by lahsivjar

Loading

mergify bot commented Dec 15, 2022

mergify bot commented Dec 15, 2022

apmmachine commented Dec 16, 2022 •

edited

Loading

Build stats

Test stats 🧪

apmmachine commented Dec 16, 2022 •

edited

Loading

carsonip commented Dec 16, 2022

axw left a comment

mergify bot commented Dec 19, 2022

simitt left a comment •

edited

Loading

simitt Dec 20, 2022

carsonip Dec 23, 2022

mergify bot commented Dec 23, 2022

simitt commented Dec 23, 2022

carsonip commented Dec 23, 2022

simitt commented Dec 27, 2022

carsonip commented Dec 27, 2022

lahsivjar commented Mar 6, 2023 •

edited

Loading

Transaction Events

Transaction Metrics

Service Transaction Metrics

Transaction Events

Transaction Metrics

Service Transaction Metrics

Service metrics txn events txn metrics #9819

Service metrics txn events txn metrics #9819

Conversation

carsonip commented Dec 15, 2022 • edited by lahsivjar Loading

Motivation/summary

Checklist

How to test these changes

Related issues

mergify bot commented Dec 15, 2022

mergify bot commented Dec 15, 2022

apmmachine commented Dec 16, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

apmmachine commented Dec 16, 2022 • edited Loading

📚 Go benchmark report

carsonip commented Dec 16, 2022

axw left a comment

Choose a reason for hiding this comment

mergify bot commented Dec 19, 2022

simitt left a comment • edited Loading

Choose a reason for hiding this comment

simitt Dec 20, 2022

Choose a reason for hiding this comment

carsonip Dec 23, 2022

Choose a reason for hiding this comment

mergify bot commented Dec 23, 2022

simitt commented Dec 23, 2022

carsonip commented Dec 23, 2022

simitt commented Dec 27, 2022

carsonip commented Dec 27, 2022

lahsivjar commented Mar 6, 2023 • edited Loading

Transaction Events

Transaction Metrics

Service Transaction Metrics

Transaction Events

Transaction Metrics

Service Transaction Metrics

carsonip commented Dec 15, 2022 •

edited by lahsivjar

Loading

apmmachine commented Dec 16, 2022 •

edited

Loading

apmmachine commented Dec 16, 2022 •

edited

Loading

simitt left a comment •

edited

Loading

lahsivjar commented Mar 6, 2023 •

edited

Loading