From f3d3878cae92affb4e9228908b52fdffd7f6698e Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Tue, 7 Nov 2023 19:54:49 -0800 Subject: [PATCH 01/13] Initial content --- text/0000-Sampling_Configuration.md | 117 ++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 text/0000-Sampling_Configuration.md diff --git a/text/0000-Sampling_Configuration.md b/text/0000-Sampling_Configuration.md new file mode 100644 index 000000000..dbbb497fc --- /dev/null +++ b/text/0000-Sampling_Configuration.md @@ -0,0 +1,117 @@ +# Sampling Configuration + +An attempt to provide a framework for defining sampling configurations. + +Calling this proposal half-baked would be too generous. At this time it just an idea, with many questions unanswered and without any working prototype. Its purpose is to start a discussion on the needs and wants the Open Telemetry community might have on the subject of sampling configuration and a possible way to accomplish them. + +The focus is on head-based sampling, but a similar approach might be used for later sampling stages as well. + +## Motivation + +The need for sampling configuration has been explicitly or implicitly indicated in several discussions, some of them going back a number of years, see for example +issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173), +issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060), +discussion [3725](https://github.com/open-telemetry/opentelemetry-specification/discussions/3725), +issue [3205](https://github.com/open-telemetry/opentelemetry-specification/issues/3205). + +A number of custom samplers are already available as indpendent contributions or just as ideas. They all share the same pain point, which is lack of easy to use configuration mechanism. The goal of this proposal is to create a configuration schema which supports not only the current set of SDK standard samplers, but also non-standard ones, and even samplers that will be added in the future. + +In contrast, the existing configuration schemas, such as [Jaeger sampling](https://www.jaegertracing.io/docs/1.50/sampling/#file-based-sampling-configuration), or Agent [Configuration proposal](https://github.com/open-telemetry/oteps/pull/225) address sampling configuration with a limited known set of samplers only. + +## The basics + +It is assumed that the sampling configuration will be a YAML document, in most cases available as a file. Remote configuration remains an option, as well as dynamic changes to the configuration. + +The configuration file will contain a definition of the sampler to use: + +```yaml +--- +sampler: + SAMPLER +``` +Additional information could be placed there as well. For example, for Java, there could be a location of a jarfile containing any non-standard samplers which have been configured. + +A SAMPLER is described by a structure with two fields: +```yaml + samplerType: TYPE + parameters: # an optional list of parameters for the sampler + - PARAM1 + - PARAM2 + ... + - PARAMn +``` +The mechanism to map the sampler TYPE to the sampler implementation will be platform specific. For Java, the TYPE can be a simple class name, or a part of class name, and the implementing class can be found using reflection. + +There will be a need for each supported sampler to have a documented canonical way of instantiation or initialization, which takes a known list of parameters. The order of the parameters specified in the configuration file will have to match exactly that list. + +A sampler can be passed as an argument for another sampler: +```yaml + samplerType: ParentBased + parameters: + - samplerType: TraceIdRatioBased # for root spans + parameters: + - 0.75 + - samplerType: AlwaysOn # remote parent sampled + - samplerType: AlwaysOff # remote parent not sampled + - samplerType: AlwaysOn # local parent sampled + - samplerType: AlwaysOff # local parent not sampled +``` + +There's no limit on the depth of nesting samplers, which hopefully allows to create complex configurations addressing most of the sampling needs. + +## Composite Samplers + +New composite samplers are proposed to make group sampling decisions. They always ask the child samplers for their decisions, but eventually make the final call. + +### Logical-Or Sampling +```yaml + samplerType: AnyOf + parameters: + - - SAMPLER1 + - SAMPLER2 + ... + - SAMPLERn +``` +The AnyOf sampler takes a list of Samplers as the argument. When making a sampling decision, it goes through the list to find a sampler that decides to sample. If found, this sampling decision is final. If none of the samplers from the list wants to sample the span, the AnyOf sampler drops the span. +If the first child which decided to sample modified the trace state, the effect of the modification remains in effect. + +### Logical-And Sampling +```yaml + samplerType: AllOf + parameters: + - - SAMPLER1 + - SAMPLER2 + ... + - SAMPLERn +``` +The AllOf sampler takes a list of SAMPLERs as the argument. When making a sampling decision, it goes through the list to find a sampler that decides not to sample. If found, thie final sampling decision is not to sample. If all of the samplers from the list want to sample the span, the AllOf sampler samples the span. + +If all of the child samplers agreed to sample, and some of them modified the trace state, the modifications are cumulative as performed in the given order. If the final decision is not to sample, the trace state remains unchanged. + +### Rule based sampling + +For rule-based sampling (e.g. decision depends on Span attributes), we need a RuleBasedSampler, which will take a list of Rules as an argument, an optional Span Kind and a fallback Sampler. Each rule will consist of a Predicate and a Sampler. For a sampling decision, if the Span kind matches the optionally specified kind, the list will be worked through in the declared order. If a Predicate holds, the corresponding Sampler will be called to make the final sampling decision. If the Span Kind does not match the final decision is not to sample. If the Span Kind matches, but none of the Predicates evaluates to True, the fallback sampler makes the final decision. + +Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), with similar functionality. +```yaml + samplerType: RuleBased + spanKind: SERVER | CONSUMER | ... # optional + parameters: + - - RULE1 + - RULE2 + ... + - RULEn + - FALLBACK_SAMPLER +``` +where each RULE is +```yaml + predicate: PREDICATE + sampler: SAMPLER +``` +The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be taken), and perform tests on the accessible values. +For example, one can test if the target URL for a SERVER span matches a given pattern. + +## Open Issues +- How to encode Predicates so they will be readable and yet powerfull and efficiently calculated? +- How to handle RecordOnly (_do not export_) sampling decisions? + From 28eb18ce11e2113ee333785a20a649ad7d665d98 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Tue, 7 Nov 2023 20:00:05 -0800 Subject: [PATCH 02/13] Renaming file --- ...0-Sampling_Configuration.md => 0240-Sampling_Configuration.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-Sampling_Configuration.md => 0240-Sampling_Configuration.md} (100%) diff --git a/text/0000-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md similarity index 100% rename from text/0000-Sampling_Configuration.md rename to text/0240-Sampling_Configuration.md From 0e87ea37f62395c93c915a1a436880c88308a264 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 10 Nov 2023 14:45:47 -0800 Subject: [PATCH 03/13] Updates based on Sampling SIG discussion --- text/0240-Sampling_Configuration.md | 64 +++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 9 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index dbbb497fc..3f4699ed5 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -2,21 +2,40 @@ An attempt to provide a framework for defining sampling configurations. -Calling this proposal half-baked would be too generous. At this time it just an idea, with many questions unanswered and without any working prototype. Its purpose is to start a discussion on the needs and wants the Open Telemetry community might have on the subject of sampling configuration and a possible way to accomplish them. +Calling this proposal half-baked would be too generous. At this time it just a vision, with many questions unanswered and without any working prototype. Its purpose is to start a discussion on the needs and wants the Open Telemetry community might have on the subject of sampling configuration and a possible way to accomplish them. -The focus is on head-based sampling, but a similar approach might be used for later sampling stages as well. +The focus is on head-based sampling, but a similar approach might be used for later sampling stages as well. Most of the technical details presented here assume Java as the platform, but should be general enough to have corresponding concepts and solutions available for other platformstoo. ## Motivation The need for sampling configuration has been explicitly or implicitly indicated in several discussions, some of them going back a number of years, see for example -issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173), -issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060), -discussion [3725](https://github.com/open-telemetry/opentelemetry-specification/discussions/3725), -issue [3205](https://github.com/open-telemetry/opentelemetry-specification/issues/3205). +- issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173): Way to ignore healthcheck traces when using automatic tracer across all languages? +- issue [679](https://github.com/open-telemetry/opentelemetry-specification/issues/679): Configuring a sampler from an environment variable +- issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing +- issue [1844](https://github.com/open-telemetry/opentelemetry-specification/issues/1844): Composite Sampler +- issue [2085](https://github.com/open-telemetry/opentelemetry-specification/issues/2085): Remote controlled sampling rules using attributes +- issue [3205](https://github.com/open-telemetry/opentelemetry-specification/issues/3205): Equivalent of "logLevel" for spans +- discussion [3725](https://github.com/open-telemetry/opentelemetry-specification/discussions/3725): Overriding Decisions of the Built-in Trace Samplers via a "sample.priority" Like Span Attribute -A number of custom samplers are already available as indpendent contributions or just as ideas. They all share the same pain point, which is lack of easy to use configuration mechanism. The goal of this proposal is to create a configuration schema which supports not only the current set of SDK standard samplers, but also non-standard ones, and even samplers that will be added in the future. -In contrast, the existing configuration schemas, such as [Jaeger sampling](https://www.jaegertracing.io/docs/1.50/sampling/#file-based-sampling-configuration), or Agent [Configuration proposal](https://github.com/open-telemetry/oteps/pull/225) address sampling configuration with a limited known set of samplers only. +A number of custom samplers are already available as indpendent contributions +([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), +[LinksBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksBasedSampler.java), +and, of course, the latest and greatest [Consistent Probability Sampling](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/consistent-sampling)) or just as ideas. They all share the same pain point, which is lack of easy to use configuration mechanism. + +Even when the code for these samplers is available, their use is not very simple. In case of Java, they require writing a custom agent [extension](https://opentelemetry.io/docs/instrumentation/java/automatic/extensions/). This can become a hurdle, especially for the Open Telemetry users with no hands-on coding experience. + +## The Goal + +The goal of this proposal is to create an open-ended configuration schema which supports not only the current set of SDK standard samplers, but also non-standard ones, and even samplers that will be added in the future. Furthermore the samplers should be composable together, as it might be required by the users. + +In contrast, the existing configuration schemas, such as [Jaeger sampling](https://www.jaegertracing.io/docs/1.50/sampling/#file-based-sampling-configuration), or Agent [OTEP 225 - Configuration proposal](https://github.com/open-telemetry/oteps/pull/225) address sampling configuration with a limited known set of samplers only. + +## Use cases + +- I want to use one of the samplers from the `opentelemetry-java-contrib` repository, but I do not want to build my own agent extension. I prefer to download one or more jarfiles containing the samplers and configure their use without writing any additional code. +- I want to apply a sampling strategy that combines different samplers depending on the span attributes, such as the URL of the incoming request, and I expect to update the configuration frequently, so I prefer that it is file-based (rather than hardcoded), and better yet, applied dynamically + - I want to write my own sampler with some unique logic, but I want to focus entirely on the sampling algorithm and avoid writing any boilerplate code for instantiating, configuring, and wrapping it up as an agent extension ## The basics @@ -40,7 +59,7 @@ A SAMPLER is described by a structure with two fields: ... - PARAMn ``` -The mechanism to map the sampler TYPE to the sampler implementation will be platform specific. For Java, the TYPE can be a simple class name, or a part of class name, and the implementing class can be found using reflection. +The mechanism to map the sampler TYPE to the sampler implementation will be platform specific. For Java, the TYPE can be a simple class name, or a part of class name, and the implementing class can be found using reflection. Specifying a fully qualified class name should be an option. There will be a need for each supported sampler to have a documented canonical way of instantiation or initialization, which takes a known list of parameters. The order of the parameters specified in the configuration file will have to match exactly that list. @@ -111,6 +130,33 @@ where each RULE is The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be taken), and perform tests on the accessible values. For example, one can test if the target URL for a SERVER span matches a given pattern. +## Strong and Weak Typing + +Constructing a sampler instance may require using some data types which are not strings, numbers, samplers, or lists of those types. Obviously, the beforementioned rule-based sampler needs to get `Rules`, which, in turn, will reference `Predicates`. The knowledge of these types can be built-in to the YAML parser, to ensure proper support. + +If there's a need to support other complex types, the supported parsers can take maps of (key, value) pairs, which will be provided directly by the YAML parser. The samplers will be responsible for converting these values into a suitable internal representation. + +## Suggested deployment pattern + +Using the file-based configuration for sampling as described in this proposal does not require any changes to the OpenTelemetry Java Agent. The YAML file parser and the code to instantiate and configure requested samplers can be provided as an Extension jarfile (`config_based_sampler.jar` below). All samplers from the `opentelemetry-java-contrib` repository can be also made available as a separate jarfile (`all_samplers.jar` below). + +```bash +$ java -javaagent:path/to/opentelemetry-javaagent.jar \ + -Dotel.javaagent.extensions=path/to/config_based_sampler.jar \ + -Dotel.sampling.config.file=path/to/sampling_config.yaml \ + -Dotel.sampling.classpath=path/to/all_samplers.jar \ + ... \ + -jar myapp.jar +``` + +## Compatibility with existing standards and proposals + +Generally, the standard SDK samplers as well as those from the `opentelemetry-java-contrib` repository, with few exceptions, are not prepared to be used directly by this proposal. Even the standard samplers do not have a uniform way of instantiation. For example `ParentBasedSampler` offers only a constructor, while `TraceIdRatioBasedSampler` is typically instantiated using static `create` method. + +However, adding some uniformity there, as well as to the samplers from `opentelemetry-java-contrib` should be quite easy, and hopefully not very controversial. It is also possible to demand a uniform instantiation mechanism only for non-standard samplers; the knowledge about the standard samplers can be built-in. + +Another point of contention is that the existing configuration practices and proposals (see [JSON Schema Definitions for OpenTelemetry File Configuration](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema/tracer_provider.json)) expect very specific knowledge about the samplers, while in this proposal responsibility for matching the samplers' arguments with the samplers signature becomes user's responsibility. However, to decrease the risk of misconfiguration, this proposal can be extended by introducing another configuration section which would describe the number of arguments and their type for non-standard samplers, thus providing some level of consistency checking. + ## Open Issues - How to encode Predicates so they will be readable and yet powerfull and efficiently calculated? - How to handle RecordOnly (_do not export_) sampling decisions? From 26fe48aeae59e7318c100c64416292f53f7065e9 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 10 Nov 2023 17:04:25 -0800 Subject: [PATCH 04/13] Adding example --- text/0240-Sampling_Configuration.md | 52 ++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index 3f4699ed5..093fd424c 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -114,8 +114,8 @@ For rule-based sampling (e.g. decision depends on Span attributes), we need a Ru Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), with similar functionality. ```yaml samplerType: RuleBased - spanKind: SERVER | CONSUMER | ... # optional parameters: + - spanKind: SERVER | CONSUMER | ... # optional - - RULE1 - RULE2 ... @@ -130,6 +130,56 @@ where each RULE is The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be taken), and perform tests on the accessible values. For example, one can test if the target URL for a SERVER span matches a given pattern. +## Example + +Let's assume that a user wants to configure head sampling as follows: +- for root spans: + - drop all `/healthcheck` requests + - capture all `/checkout` requests + - capture 25% of all other requests +- for non-root spans + - follow the parent sampling decision + - however, capture all calls to service `/foo` (even if the trace will be incomplete) +- in any case, do not exceed 1000 spans/second + +Such sampling requirements can be expressed as: +```yaml + samplerType: AllOf + parameters: + - - samplerType: AnyOf + parameters: + - - samplerType: ParentBased + parameters: + - samplerType: RuleBased # for root spans + parameters: + - spanKind: SERVER + - - predicate: http.target == /healtcheck + sampler: + samplerType: AlwaysOff + - predicate: http.target == /checkout + sampler: + samplerType: AlwaysOn + - samplerType: TraceIdRatioBased # fallback sampler + parameters: + - 0.25 + - samplerType: AlwaysOn # remote parent sampled + - samplerType: AlwaysOff # remote parent not sampled + - samplerType: AlwaysOn # local parent sampled + - samplerType: AlwaysOff # local parent not sampled + - samplerType: RuleBased + parameters: + - spanKind: CLIENT + - - predicate: http.url == /foo + sampler: + samplerType: AlwaysOn + - samplerType: AlwaysOff # fallback sampler + - samplerType: RateLimiting + parameters: + - 1000 # spans/second +``` + +The above example uses plain text representation of predicates. Actual representation for predicates is TBD. The example also assumes that a hypothetical `RateLimitingSampler` is available. + ## Strong and Weak Typing Constructing a sampler instance may require using some data types which are not strings, numbers, samplers, or lists of those types. Obviously, the beforementioned rule-based sampler needs to get `Rules`, which, in turn, will reference `Predicates`. The knowledge of these types can be built-in to the YAML parser, to ensure proper support. From 29705836e1b89dd150b742e957a4ad62875f6826 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 13 Nov 2023 09:07:01 -0800 Subject: [PATCH 05/13] simplifying - flattening the RULE structure --- text/0240-Sampling_Configuration.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index 093fd424c..643a26753 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -27,13 +27,13 @@ Even when the code for these samplers is available, their use is not very simple ## The Goal -The goal of this proposal is to create an open-ended configuration schema which supports not only the current set of SDK standard samplers, but also non-standard ones, and even samplers that will be added in the future. Furthermore the samplers should be composable together, as it might be required by the users. +The goal of this proposal is to create an open-ended configuration schema which supports not only the current set of SDK standard samplers, but also non-standard ones, and even samplers that will be added in the future. Furthermore the samplers should be composable together, as it is often required by the users. In contrast, the existing configuration schemas, such as [Jaeger sampling](https://www.jaegertracing.io/docs/1.50/sampling/#file-based-sampling-configuration), or Agent [OTEP 225 - Configuration proposal](https://github.com/open-telemetry/oteps/pull/225) address sampling configuration with a limited known set of samplers only. ## Use cases -- I want to use one of the samplers from the `opentelemetry-java-contrib` repository, but I do not want to build my own agent extension. I prefer to download one or more jarfiles containing the samplers and configure their use without writing any additional code. +- I want to use some of the samplers from the `opentelemetry-java-contrib` repository, but I do not want to build my own agent extension. I prefer to download one or more jarfiles containing the samplers and configure their use without writing any additional code. - I want to apply a sampling strategy that combines different samplers depending on the span attributes, such as the URL of the incoming request, and I expect to update the configuration frequently, so I prefer that it is file-based (rather than hardcoded), and better yet, applied dynamically - I want to write my own sampler with some unique logic, but I want to focus entirely on the sampling algorithm and avoid writing any boilerplate code for instantiating, configuring, and wrapping it up as an agent extension @@ -122,10 +122,15 @@ Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSamp - RULEn - FALLBACK_SAMPLER ``` -where each RULE is +where RULE is an extension of SAMPLER, providing additional predicate ```yaml - predicate: PREDICATE - sampler: SAMPLER + predicate: PREDICATE + samplerType: TYPE + parameters: + - PARAM1 + - PARAM2 + ... + - PARAMn ``` The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be taken), and perform tests on the accessible values. For example, one can test if the target URL for a SERVER span matches a given pattern. @@ -153,12 +158,10 @@ Such sampling requirements can be expressed as: - samplerType: RuleBased # for root spans parameters: - spanKind: SERVER - - - predicate: http.target == /healtcheck - sampler: - samplerType: AlwaysOff + - - predicate: http.target == /healthcheck + samplerType: AlwaysOff - predicate: http.target == /checkout - sampler: - samplerType: AlwaysOn + samplerType: AlwaysOn - samplerType: TraceIdRatioBased # fallback sampler parameters: - 0.25 @@ -170,8 +173,7 @@ Such sampling requirements can be expressed as: parameters: - spanKind: CLIENT - - predicate: http.url == /foo - sampler: - samplerType: AlwaysOn + samplerType: AlwaysOn - samplerType: AlwaysOff # fallback sampler - samplerType: RateLimiting parameters: From 583fccfc95c371afc2c443c3bc2b7ff09b15d54e Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 13 Nov 2023 09:17:50 -0800 Subject: [PATCH 06/13] correcting spelling errors --- text/0240-Sampling_Configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index 643a26753..e0fa5dfb2 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -18,7 +18,7 @@ The need for sampling configuration has been explicitly or implicitly indicated - discussion [3725](https://github.com/open-telemetry/opentelemetry-specification/discussions/3725): Overriding Decisions of the Built-in Trace Samplers via a "sample.priority" Like Span Attribute -A number of custom samplers are already available as indpendent contributions +A number of custom samplers are already available as independent contributions ([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), [LinksBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksBasedSampler.java), and, of course, the latest and greatest [Consistent Probability Sampling](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/consistent-sampling)) or just as ideas. They all share the same pain point, which is lack of easy to use configuration mechanism. @@ -210,6 +210,6 @@ However, adding some uniformity there, as well as to the samplers from `opentele Another point of contention is that the existing configuration practices and proposals (see [JSON Schema Definitions for OpenTelemetry File Configuration](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema/tracer_provider.json)) expect very specific knowledge about the samplers, while in this proposal responsibility for matching the samplers' arguments with the samplers signature becomes user's responsibility. However, to decrease the risk of misconfiguration, this proposal can be extended by introducing another configuration section which would describe the number of arguments and their type for non-standard samplers, thus providing some level of consistency checking. ## Open Issues -- How to encode Predicates so they will be readable and yet powerfull and efficiently calculated? +- How to encode Predicates so they will be readable and yet powerful and efficiently calculated? - How to handle RecordOnly (_do not export_) sampling decisions? From e2d1a844b15964000fd299049d77bcf4a24b1a56 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 17 Nov 2023 13:14:35 -0800 Subject: [PATCH 07/13] Cleanup and small changes --- text/0240-Sampling_Configuration.md | 41 ++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index e0fa5dfb2..244cde237 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -9,6 +9,7 @@ The focus is on head-based sampling, but a similar approach might be used for la ## Motivation The need for sampling configuration has been explicitly or implicitly indicated in several discussions, some of them going back a number of years, see for example + - issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173): Way to ignore healthcheck traces when using automatic tracer across all languages? - issue [679](https://github.com/open-telemetry/opentelemetry-specification/issues/679): Configuring a sampler from an environment variable - issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing @@ -17,11 +18,10 @@ The need for sampling configuration has been explicitly or implicitly indicated - issue [3205](https://github.com/open-telemetry/opentelemetry-specification/issues/3205): Equivalent of "logLevel" for spans - discussion [3725](https://github.com/open-telemetry/opentelemetry-specification/discussions/3725): Overriding Decisions of the Built-in Trace Samplers via a "sample.priority" Like Span Attribute - A number of custom samplers are already available as independent contributions ([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), [LinksBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksBasedSampler.java), -and, of course, the latest and greatest [Consistent Probability Sampling](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/consistent-sampling)) or just as ideas. They all share the same pain point, which is lack of easy to use configuration mechanism. +and, of course, the latest and greatest [Consistent Probability Sampling](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/consistent-sampling)) or just as ideas. They all share the same pain point, which is lack of easy to use configuration mechanism. Even when the code for these samplers is available, their use is not very simple. In case of Java, they require writing a custom agent [extension](https://opentelemetry.io/docs/instrumentation/java/automatic/extensions/). This can become a hurdle, especially for the Open Telemetry users with no hands-on coding experience. @@ -34,8 +34,8 @@ In contrast, the existing configuration schemas, such as [Jaeger sampling](https ## Use cases - I want to use some of the samplers from the `opentelemetry-java-contrib` repository, but I do not want to build my own agent extension. I prefer to download one or more jarfiles containing the samplers and configure their use without writing any additional code. -- I want to apply a sampling strategy that combines different samplers depending on the span attributes, such as the URL of the incoming request, and I expect to update the configuration frequently, so I prefer that it is file-based (rather than hardcoded), and better yet, applied dynamically - - I want to write my own sampler with some unique logic, but I want to focus entirely on the sampling algorithm and avoid writing any boilerplate code for instantiating, configuring, and wrapping it up as an agent extension +- I want to apply a sampling strategy that combines different samplers depending on the span attributes, such as the URL of the incoming request, and I expect to update the configuration frequently, so I prefer that it is file-based (rather than hardcoded), and better yet, applied dynamically. +- I want to write my own sampler with some unique logic, but I want to focus entirely on the sampling algorithm and avoid writing any boilerplate code for instantiating, configuring, and wrapping it up as an agent extension. ## The basics @@ -48,9 +48,11 @@ The configuration file will contain a definition of the sampler to use: sampler: SAMPLER ``` + Additional information could be placed there as well. For example, for Java, there could be a location of a jarfile containing any non-standard samplers which have been configured. A SAMPLER is described by a structure with two fields: + ```yaml samplerType: TYPE parameters: # an optional list of parameters for the sampler @@ -59,11 +61,13 @@ A SAMPLER is described by a structure with two fields: ... - PARAMn ``` + The mechanism to map the sampler TYPE to the sampler implementation will be platform specific. For Java, the TYPE can be a simple class name, or a part of class name, and the implementing class can be found using reflection. Specifying a fully qualified class name should be an option. There will be a need for each supported sampler to have a documented canonical way of instantiation or initialization, which takes a known list of parameters. The order of the parameters specified in the configuration file will have to match exactly that list. A sampler can be passed as an argument for another sampler: + ```yaml samplerType: ParentBased parameters: @@ -83,6 +87,7 @@ There's no limit on the depth of nesting samplers, which hopefully allows to cre New composite samplers are proposed to make group sampling decisions. They always ask the child samplers for their decisions, but eventually make the final call. ### Logical-Or Sampling + ```yaml samplerType: AnyOf parameters: @@ -91,10 +96,12 @@ New composite samplers are proposed to make group sampling decisions. They alway ... - SAMPLERn ``` + The AnyOf sampler takes a list of Samplers as the argument. When making a sampling decision, it goes through the list to find a sampler that decides to sample. If found, this sampling decision is final. If none of the samplers from the list wants to sample the span, the AnyOf sampler drops the span. If the first child which decided to sample modified the trace state, the effect of the modification remains in effect. ### Logical-And Sampling + ```yaml samplerType: AllOf parameters: @@ -103,6 +110,7 @@ If the first child which decided to sample modified the trace state, the effect ... - SAMPLERn ``` + The AllOf sampler takes a list of SAMPLERs as the argument. When making a sampling decision, it goes through the list to find a sampler that decides not to sample. If found, thie final sampling decision is not to sample. If all of the samplers from the list want to sample the span, the AllOf sampler samples the span. If all of the child samplers agreed to sample, and some of them modified the trace state, the modifications are cumulative as performed in the given order. If the final decision is not to sample, the trace state remains unchanged. @@ -112,6 +120,7 @@ If all of the child samplers agreed to sample, and some of them modified the tra For rule-based sampling (e.g. decision depends on Span attributes), we need a RuleBasedSampler, which will take a list of Rules as an argument, an optional Span Kind and a fallback Sampler. Each rule will consist of a Predicate and a Sampler. For a sampling decision, if the Span kind matches the optionally specified kind, the list will be worked through in the declared order. If a Predicate holds, the corresponding Sampler will be called to make the final sampling decision. If the Span Kind does not match the final decision is not to sample. If the Span Kind matches, but none of the Predicates evaluates to True, the fallback sampler makes the final decision. Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), with similar functionality. + ```yaml samplerType: RuleBased parameters: @@ -122,7 +131,9 @@ Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSamp - RULEn - FALLBACK_SAMPLER ``` -where RULE is an extension of SAMPLER, providing additional predicate + +where RULE is an extension of SAMPLER, providing additional predicate: + ```yaml predicate: PREDICATE samplerType: TYPE @@ -132,22 +143,25 @@ where RULE is an extension of SAMPLER, providing additional predicate ... - PARAMn ``` + The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be taken), and perform tests on the accessible values. For example, one can test if the target URL for a SERVER span matches a given pattern. ## Example Let's assume that a user wants to configure head sampling as follows: + - for root spans: - - drop all `/healthcheck` requests - - capture all `/checkout` requests - - capture 25% of all other requests + - drop all `/healthcheck` requests + - capture all `/checkout` requests + - capture 25% of all other requests - for non-root spans - - follow the parent sampling decision - - however, capture all calls to service `/foo` (even if the trace will be incomplete) + - follow the parent sampling decision + - however, capture all calls to service `/foo` (even if the trace will be incomplete) - in any case, do not exceed 1000 spans/second Such sampling requirements can be expressed as: + ```yaml samplerType: AllOf parameters: @@ -201,6 +215,8 @@ $ java -javaagent:path/to/opentelemetry-javaagent.jar \ -jar myapp.jar ``` +If the feature proves popular, other options can be considered. + ## Compatibility with existing standards and proposals Generally, the standard SDK samplers as well as those from the `opentelemetry-java-contrib` repository, with few exceptions, are not prepared to be used directly by this proposal. Even the standard samplers do not have a uniform way of instantiation. For example `ParentBasedSampler` offers only a constructor, while `TraceIdRatioBasedSampler` is typically instantiated using static `create` method. @@ -210,6 +226,7 @@ However, adding some uniformity there, as well as to the samplers from `opentele Another point of contention is that the existing configuration practices and proposals (see [JSON Schema Definitions for OpenTelemetry File Configuration](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema/tracer_provider.json)) expect very specific knowledge about the samplers, while in this proposal responsibility for matching the samplers' arguments with the samplers signature becomes user's responsibility. However, to decrease the risk of misconfiguration, this proposal can be extended by introducing another configuration section which would describe the number of arguments and their type for non-standard samplers, thus providing some level of consistency checking. ## Open Issues -- How to encode Predicates so they will be readable and yet powerful and efficiently calculated? + +- How to encode Predicates so they will be readable and yet powerful, and at the same time calculated efficiently? - How to handle RecordOnly (_do not export_) sampling decisions? - +- How to improve ergonomy of dealing with generic sampler definitions? From 877a05656686787153ea179c04df7617848177e8 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 20 Nov 2023 18:57:25 -0800 Subject: [PATCH 08/13] Removing fallback sampler. Adding more references. --- text/0240-Sampling_Configuration.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index 244cde237..449287ea5 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -4,17 +4,18 @@ An attempt to provide a framework for defining sampling configurations. Calling this proposal half-baked would be too generous. At this time it just a vision, with many questions unanswered and without any working prototype. Its purpose is to start a discussion on the needs and wants the Open Telemetry community might have on the subject of sampling configuration and a possible way to accomplish them. -The focus is on head-based sampling, but a similar approach might be used for later sampling stages as well. Most of the technical details presented here assume Java as the platform, but should be general enough to have corresponding concepts and solutions available for other platformstoo. +The focus is on head-based sampling, but a similar approach might be used for later sampling stages as well. Most of the technical details presented here assume Java as the platform, but should be general enough to have corresponding concepts and solutions available for other platforms too. ## Motivation -The need for sampling configuration has been explicitly or implicitly indicated in several discussions, some of them going back a number of years, see for example +The need for sampling configuration has been explicitly or implicitly indicated in several discussions, both within the [Samplig SIG](https://docs.google.com/document/d/1gASMhmxNt9qCa8czEMheGlUW2xpORiYoD7dBD7aNtbQ) and in the wider community. Some of the discussions are going back a number of years, see for example - issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173): Way to ignore healthcheck traces when using automatic tracer across all languages? - issue [679](https://github.com/open-telemetry/opentelemetry-specification/issues/679): Configuring a sampler from an environment variable - issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing - issue [1844](https://github.com/open-telemetry/opentelemetry-specification/issues/1844): Composite Sampler - issue [2085](https://github.com/open-telemetry/opentelemetry-specification/issues/2085): Remote controlled sampling rules using attributes +- otep [213](https://github.com/open-telemetry/oteps/pull/213): Add Sampling SIG research notes - issue [3205](https://github.com/open-telemetry/opentelemetry-specification/issues/3205): Equivalent of "logLevel" for spans - discussion [3725](https://github.com/open-telemetry/opentelemetry-specification/discussions/3725): Overriding Decisions of the Built-in Trace Samplers via a "sample.priority" Like Span Attribute @@ -34,7 +35,7 @@ In contrast, the existing configuration schemas, such as [Jaeger sampling](https ## Use cases - I want to use some of the samplers from the `opentelemetry-java-contrib` repository, but I do not want to build my own agent extension. I prefer to download one or more jarfiles containing the samplers and configure their use without writing any additional code. -- I want to apply a sampling strategy that combines different samplers depending on the span attributes, such as the URL of the incoming request, and I expect to update the configuration frequently, so I prefer that it is file-based (rather than hardcoded), and better yet, applied dynamically. +- I want to apply a sampling strategy that combines different samplers depending on the span attributes, such as the URL of the incoming request, and I expect to update the configuration frequently, so I prefer that it is file based (rather than hardcoded), and better yet, applied dynamically. - I want to write my own sampler with some unique logic, but I want to focus entirely on the sampling algorithm and avoid writing any boilerplate code for instantiating, configuring, and wrapping it up as an agent extension. ## The basics @@ -117,7 +118,7 @@ If all of the child samplers agreed to sample, and some of them modified the tra ### Rule based sampling -For rule-based sampling (e.g. decision depends on Span attributes), we need a RuleBasedSampler, which will take a list of Rules as an argument, an optional Span Kind and a fallback Sampler. Each rule will consist of a Predicate and a Sampler. For a sampling decision, if the Span kind matches the optionally specified kind, the list will be worked through in the declared order. If a Predicate holds, the corresponding Sampler will be called to make the final sampling decision. If the Span Kind does not match the final decision is not to sample. If the Span Kind matches, but none of the Predicates evaluates to True, the fallback sampler makes the final decision. +For rule-based sampling (e.g. decision depends on Span attributes), we need a RuleBasedSampler, which will take a list of Rules as an argument, and an optional Span Kind. Each rule will consist of a Predicate and a Sampler. For a sampling decision, if the Span kind matches the optionally specified kind, the list will be worked through in the declared order. If a Predicate holds, the corresponding Sampler will be called to make the final sampling decision. If the Span Kind does not match, or none of the Predicates evaluates to True, the final decision is not to sample. Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), with similar functionality. @@ -129,7 +130,6 @@ Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSamp - RULE2 ... - RULEn - - FALLBACK_SAMPLER ``` where RULE is an extension of SAMPLER, providing additional predicate: @@ -176,9 +176,10 @@ Such sampling requirements can be expressed as: samplerType: AlwaysOff - predicate: http.target == /checkout samplerType: AlwaysOn - - samplerType: TraceIdRatioBased # fallback sampler - parameters: - - 0.25 + - predicate: true # works as a fallback sampler + samplerType: TraceIdRatioBased + parameters: + - 0.25 - samplerType: AlwaysOn # remote parent sampled - samplerType: AlwaysOff # remote parent not sampled - samplerType: AlwaysOn # local parent sampled @@ -188,7 +189,6 @@ Such sampling requirements can be expressed as: - spanKind: CLIENT - - predicate: http.url == /foo samplerType: AlwaysOn - - samplerType: AlwaysOff # fallback sampler - samplerType: RateLimiting parameters: - 1000 # spans/second From 079e44d094c1accf1a4e4c454e196678581700f5 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Tue, 21 Nov 2023 21:57:02 -0800 Subject: [PATCH 09/13] Revising the yaml syntax to allow sampler argument identification by parameter names. --- text/0240-Sampling_Configuration.md | 202 ++++++++++++++++------------ 1 file changed, 113 insertions(+), 89 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index 449287ea5..98dad0893 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -42,43 +42,45 @@ In contrast, the existing configuration schemas, such as [Jaeger sampling](https It is assumed that the sampling configuration will be a YAML document, in most cases available as a file. Remote configuration remains an option, as well as dynamic changes to the configuration. -The configuration file will contain a definition of the sampler to use: +The configuration file will contain an actual configuration of the sampler to use. It may also optionally contain definitions of custom samplers. ```yaml --- +sampler_definitions: # optional + + + ... + sampler: - SAMPLER + ``` -Additional information could be placed there as well. For example, for Java, there could be a location of a jarfile containing any non-standard samplers which have been configured. +Each sampler is identified by a unique _sampler_key_, which takes a form of string. +A is described by the sampler_key followed by the sampler arguments, if applicable. -A SAMPLER is described by a structure with two fields: ```yaml - samplerType: TYPE - parameters: # an optional list of parameters for the sampler - - PARAM1 - - PARAM2 - ... - - PARAMn + : + : + : + ... ``` -The mechanism to map the sampler TYPE to the sampler implementation will be platform specific. For Java, the TYPE can be a simple class name, or a part of class name, and the implementing class can be found using reflection. Specifying a fully qualified class name should be an option. - -There will be a need for each supported sampler to have a documented canonical way of instantiation or initialization, which takes a known list of parameters. The order of the parameters specified in the configuration file will have to match exactly that list. - A sampler can be passed as an argument for another sampler: ```yaml - samplerType: ParentBased - parameters: - - samplerType: TraceIdRatioBased # for root spans - parameters: - - 0.75 - - samplerType: AlwaysOn # remote parent sampled - - samplerType: AlwaysOff # remote parent not sampled - - samplerType: AlwaysOn # local parent sampled - - samplerType: AlwaysOff # local parent not sampled + parent_based: + root: + trace_id_ratio_based: + ratio: 0.75 + remote_parent_sampled: + always_on: + remote_parent_not_sampled: + always_off: + local_parent_sampled: + always_on: + local_parent_not_sampled: + always_off: ``` There's no limit on the depth of nesting samplers, which hopefully allows to create complex configurations addressing most of the sampling needs. @@ -90,58 +92,52 @@ New composite samplers are proposed to make group sampling decisions. They alway ### Logical-Or Sampling ```yaml - samplerType: AnyOf - parameters: - - - SAMPLER1 - - SAMPLER2 + any_sampler: + from: + - + - ... - - SAMPLERn + - ``` -The AnyOf sampler takes a list of Samplers as the argument. When making a sampling decision, it goes through the list to find a sampler that decides to sample. If found, this sampling decision is final. If none of the samplers from the list wants to sample the span, the AnyOf sampler drops the span. +The `any_sampler` is a composite sampler which takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list to find a sampler that decides to sample. If found, this sampling decision is final. If none of the samplers from the list wants to sample the span, the composite sampler drops the span. If the first child which decided to sample modified the trace state, the effect of the modification remains in effect. ### Logical-And Sampling ```yaml - samplerType: AllOf - parameters: - - - SAMPLER1 - - SAMPLER2 + all_samplers: + from: + - + - ... - - SAMPLERn + - ``` -The AllOf sampler takes a list of SAMPLERs as the argument. When making a sampling decision, it goes through the list to find a sampler that decides not to sample. If found, thie final sampling decision is not to sample. If all of the samplers from the list want to sample the span, the AllOf sampler samples the span. +The `all_samplers` composite sampler takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list to find a sampler that decides not to sample. If found, the final sampling decision is not to sample. If all of the samplers from the list want to sample the span, the composite sampler samples the span. If all of the child samplers agreed to sample, and some of them modified the trace state, the modifications are cumulative as performed in the given order. If the final decision is not to sample, the trace state remains unchanged. ### Rule based sampling -For rule-based sampling (e.g. decision depends on Span attributes), we need a RuleBasedSampler, which will take a list of Rules as an argument, and an optional Span Kind. Each rule will consist of a Predicate and a Sampler. For a sampling decision, if the Span kind matches the optionally specified kind, the list will be worked through in the declared order. If a Predicate holds, the corresponding Sampler will be called to make the final sampling decision. If the Span Kind does not match, or none of the Predicates evaluates to True, the final decision is not to sample. +For rule-based sampling (e.g. decision depends on Span attributes), we need a `rule_based` sampler, which will take a list of Rules as an argument, and an optional Span Kind. Each rule will consist of a Predicate and a Sampler. For a sampling decision, if the Span kind matches the optionally specified kind, the list will be worked through in the declared order. If a Predicate holds, the corresponding Sampler will be called to make the final sampling decision. If the Span Kind does not match, or none of the Predicates evaluates to True, the final decision is not to sample. -Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), with similar functionality. +Note: The `opentelemetry-java-contrib` repository contains [RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), with similar but a bit different functionality. ```yaml - samplerType: RuleBased - parameters: - - spanKind: SERVER | CONSUMER | ... # optional - - - RULE1 - - RULE2 - ... - - RULEn + rule_based: + span_kind: SERVER | CONSUMER | ... # optional + rules: + - + - + ... ``` -where RULE is an extension of SAMPLER, providing additional predicate: +where RULE is a pair of PREDICATE and SAMPLER: ```yaml predicate: PREDICATE - samplerType: TYPE - parameters: - - PARAM1 - - PARAM2 - ... - - PARAMn + sampler: SAMPLER ``` The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be taken), and perform tests on the accessible values. @@ -163,44 +159,70 @@ Let's assume that a user wants to configure head sampling as follows: Such sampling requirements can be expressed as: ```yaml - samplerType: AllOf - parameters: - - - samplerType: AnyOf - parameters: - - - samplerType: ParentBased - parameters: - - samplerType: RuleBased # for root spans - parameters: - - spanKind: SERVER - - - predicate: http.target == /healthcheck - samplerType: AlwaysOff - - predicate: http.target == /checkout - samplerType: AlwaysOn - - predicate: true # works as a fallback sampler - samplerType: TraceIdRatioBased - parameters: - - 0.25 - - samplerType: AlwaysOn # remote parent sampled - - samplerType: AlwaysOff # remote parent not sampled - - samplerType: AlwaysOn # local parent sampled - - samplerType: AlwaysOff # local parent not sampled - - samplerType: RuleBased - parameters: - - spanKind: CLIENT - - - predicate: http.url == /foo - samplerType: AlwaysOn - - samplerType: RateLimiting - parameters: - - 1000 # spans/second +sampler: + all_samplers: + from: + - any_sampler: + from: + - parent_based: + root: + rule_based: + span_kind: SERVER + rules: + - predicate: http.target == /healthcheck + sampler: always_off + - predicate: http.target == /checkout + sampler: always_on + - predicate: true # works as a fallback sampler + sampler: trace_id_ratio_based + ratio: 0.25 + remote_parent_sampled: + always_on: + remote_parent_not_sampled: + always_off: + local_parent_sampled: + always_on: + local_parent_not_sampled: + always_off: + - rule_based: + span_kind: CLIENT + rules: + - predicate: http.url == /foo + sampler: always_on + - rate_limiting: + max_rate: 1000 # spans/second ``` -The above example uses plain text representation of predicates. Actual representation for predicates is TBD. The example also assumes that a hypothetical `RateLimitingSampler` is available. +The above example uses plain text representation of predicates. Actual representation for predicates is TBD. The example also assumes that a hypothetical `rate_limiting` sampler is available. + +## Custom samplers + +The YAML parser will have built-in knowledge about the standard SDK samplers as well as about the composite samplers described above. +However, one of the goals is to support also other samplers. Before such a custom sampler can be used in the configuration it needs to be introduced to the parser by providing the sampler definition. +Not all platforms are expected to accept custom samplers. + +For example, let's assume that the `rate_limiting` sampler from the previous example is not known to the parser. It's definition can look like follows + +```yaml +sampler_definitions: + rate_limiting: # this is the sampler key + parameters: + - max_rate: number # spans per second + implementation: + class_name: java/io/opentelemetry/contrib/sampler/RateLimitingSampler + instantiation: constructor +``` + +The platform dependant `implementation` part will specify the details allowing the parser to access the sampler code and create the sampler object. +There will be a need for each supported custom sampler to have a documented canonical way of instantiation or initialization, which takes a known list of parameters. The order of the parameters specified in the definition section will have to match exactly that list. + +Specifying the parameter type within the sampler definition can be used to both guide the parser with the correct interpretation of the argument values, or to provide additional verification of the configuration correctness. Definition of more complex types remains to be designed. ## Strong and Weak Typing -Constructing a sampler instance may require using some data types which are not strings, numbers, samplers, or lists of those types. Obviously, the beforementioned rule-based sampler needs to get `Rules`, which, in turn, will reference `Predicates`. The knowledge of these types can be built-in to the YAML parser, to ensure proper support. +Constructing a sampler instance may require using some data types which are not strings, numbers, samplers, or lists of those types. Obviously, the beforementioned rule-based sampler needs to get `SpanKind` and `Rules`, which, in turn, will reference `Predicates`. The knowledge of these types can be built-in to the YAML parser, to ensure proper support. -If there's a need to support other complex types, the supported parsers can take maps of (key, value) pairs, which will be provided directly by the YAML parser. The samplers will be responsible for converting these values into a suitable internal representation. +If there's a need to support other complex types, the supported custom parsers may be required take maps of (key, value) pairs, which will be provided directly by the YAML parser. The samplers will be responsible for converting these values into a suitable internal representation. ## Suggested deployment pattern @@ -219,14 +241,16 @@ If the feature proves popular, other options can be considered. ## Compatibility with existing standards and proposals -Generally, the standard SDK samplers as well as those from the `opentelemetry-java-contrib` repository, with few exceptions, are not prepared to be used directly by this proposal. Even the standard samplers do not have a uniform way of instantiation. For example `ParentBasedSampler` offers only a constructor, while `TraceIdRatioBasedSampler` is typically instantiated using static `create` method. +There already exists a [specification](https://github.com/open-telemetry/oteps/pull/225) for agent configuration which addresses sampling configuration. As long as the standard SDK samplers are used, the YAML representation of sampling configuration in that specification is identical to that of this proposal. +However, here the following extended capabilities are offered: -However, adding some uniformity there, as well as to the samplers from `opentelemetry-java-contrib` should be quite easy, and hopefully not very controversial. It is also possible to demand a uniform instantiation mechanism only for non-standard samplers; the knowledge about the standard samplers can be built-in. - -Another point of contention is that the existing configuration practices and proposals (see [JSON Schema Definitions for OpenTelemetry File Configuration](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema/tracer_provider.json)) expect very specific knowledge about the samplers, while in this proposal responsibility for matching the samplers' arguments with the samplers signature becomes user's responsibility. However, to decrease the risk of misconfiguration, this proposal can be extended by introducing another configuration section which would describe the number of arguments and their type for non-standard samplers, thus providing some level of consistency checking. +- additional composite samplers are built-in +- new (custom) samplers can be defined without adding any code to the SDK or SDK extension +- sampler configuration changes can be applied dynamically upon detecting configuration file modification, and supporting remote configuration via a wire protocol is an option ## Open Issues - How to encode Predicates so they will be readable and yet powerful, and at the same time calculated efficiently? -- How to handle RecordOnly (_do not export_) sampling decisions? -- How to improve ergonomy of dealing with generic sampler definitions? +- How to describe sampler parameter types to arrive at the _right_ balance of complexity and expressiveness? +- How to handle RecordOnly (_do not export_) sampling decisions by the composite samplers? + From 2b66df6dbb76031c20a77b75f948e6f64a4c2ac8 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 27 Nov 2023 19:51:41 -0800 Subject: [PATCH 10/13] Updates after discussion in Configuration SIG. --- text/0240-Sampling_Configuration.md | 54 ++++++++++++++++++----------- 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index 98dad0893..98951ba38 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -1,6 +1,6 @@ # Sampling Configuration -An attempt to provide a framework for defining sampling configurations. +This is an attempt to provide a framework for defining sampling configurations for spans and traces. Calling this proposal half-baked would be too generous. At this time it just a vision, with many questions unanswered and without any working prototype. Its purpose is to start a discussion on the needs and wants the Open Telemetry community might have on the subject of sampling configuration and a possible way to accomplish them. @@ -22,7 +22,7 @@ The need for sampling configuration has been explicitly or implicitly indicated A number of custom samplers are already available as independent contributions ([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), [LinksBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksBasedSampler.java), -and, of course, the latest and greatest [Consistent Probability Sampling](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/consistent-sampling)) or just as ideas. They all share the same pain point, which is lack of easy to use configuration mechanism. +and, of course, the latest and greatest [Consistent Probability Sampling](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/consistent-sampling)) or just as ideas, such as [hintable sampler](https://github.com/open-telemetry/opentelemetry-specification/discussions/3725). They all share the same pain point, which is lack of easy to use configuration mechanism. Even when the code for these samplers is available, their use is not very simple. In case of Java, they require writing a custom agent [extension](https://opentelemetry.io/docs/instrumentation/java/automatic/extensions/). This can become a hurdle, especially for the Open Telemetry users with no hands-on coding experience. @@ -30,13 +30,14 @@ Even when the code for these samplers is available, their use is not very simple The goal of this proposal is to create an open-ended configuration schema which supports not only the current set of SDK standard samplers, but also non-standard ones, and even samplers that will be added in the future. Furthermore the samplers should be composable together, as it is often required by the users. -In contrast, the existing configuration schemas, such as [Jaeger sampling](https://www.jaegertracing.io/docs/1.50/sampling/#file-based-sampling-configuration), or Agent [OTEP 225 - Configuration proposal](https://github.com/open-telemetry/oteps/pull/225) address sampling configuration with a limited known set of samplers only. +In contrast, the existing configuration schemas, such as [Jaeger sampling](https://www.jaegertracing.io/docs/1.50/sampling/#file-based-sampling-configuration), or Agent [OTEP 225 - Configuration proposal](https://github.com/open-telemetry/oteps/pull/225) address sampling configuration with a limited known set of samplers only, and providing extensions requires multiple steps, as shown in this [draft PR](https://github.com/open-telemetry/opentelemetry-java-examples/pull/227). ## Use cases - I want to use some of the samplers from the `opentelemetry-java-contrib` repository, but I do not want to build my own agent extension. I prefer to download one or more jarfiles containing the samplers and configure their use without writing any additional code. - I want to apply a sampling strategy that combines different samplers depending on the span attributes, such as the URL of the incoming request, and I expect to update the configuration frequently, so I prefer that it is file based (rather than hardcoded), and better yet, applied dynamically. - I want to write my own sampler with some unique logic, but I want to focus entirely on the sampling algorithm and avoid writing any boilerplate code for instantiating, configuring, and wrapping it up as an agent extension. +- I want to write my own sampler and deploy it for my already running application, without restarting the application process. ## The basics @@ -58,7 +59,6 @@ sampler: Each sampler is identified by a unique _sampler_key_, which takes a form of string. A is described by the sampler_key followed by the sampler arguments, if applicable. - ```yaml : : @@ -193,7 +193,7 @@ sampler: max_rate: 1000 # spans/second ``` -The above example uses plain text representation of predicates. Actual representation for predicates is TBD. The example also assumes that a hypothetical `rate_limiting` sampler is available. +The above example uses plain text representation for predicates. Actual representation for predicates is TBD. The example also assumes that a hypothetical `rate_limiting` sampler is available. ## Custom samplers @@ -208,25 +208,37 @@ sampler_definitions: rate_limiting: # this is the sampler key parameters: - max_rate: number # spans per second - implementation: + implementation: # Java-specific part class_name: java/io/opentelemetry/contrib/sampler/RateLimitingSampler + class_path: path/to/my/new/sampler.jar # additional classpath, optional instantiation: constructor ``` -The platform dependant `implementation` part will specify the details allowing the parser to access the sampler code and create the sampler object. -There will be a need for each supported custom sampler to have a documented canonical way of instantiation or initialization, which takes a known list of parameters. The order of the parameters specified in the definition section will have to match exactly that list. +The platform dependant `implementation` part will specify the details allowing the parser to access the sampler code and create the sampler object. For Java, reflection can be used to instantiate the sampler object. Other platforms will use other means, if available. + +There will be a need for each custom sampler to have a documented canonical way of instantiation or initialization, which takes a known list of parameters. The order of the parameters specified in the definition section will have to match exactly that list. -Specifying the parameter type within the sampler definition can be used to both guide the parser with the correct interpretation of the argument values, or to provide additional verification of the configuration correctness. Definition of more complex types remains to be designed. +Specifying the parameter type within the sampler definition can be used to both guide the parser with the correct interpretation of the argument values, or to provide additional verification of the configuration correctness. Definition of more complex types remains to be designed. ## Strong and Weak Typing Constructing a sampler instance may require using some data types which are not strings, numbers, samplers, or lists of those types. Obviously, the beforementioned rule-based sampler needs to get `SpanKind` and `Rules`, which, in turn, will reference `Predicates`. The knowledge of these types can be built-in to the YAML parser, to ensure proper support. -If there's a need to support other complex types, the supported custom parsers may be required take maps of (key, value) pairs, which will be provided directly by the YAML parser. The samplers will be responsible for converting these values into a suitable internal representation. +If there's a need to support other complex types, the supported custom samplers may be required to accept maps of (key, value) pairs, which will be provided directly by the YAML parser. The samplers will be responsible for converting these values into a suitable internal representation. + +## Compatibility with existing standards and proposals + +There already exists a [specification](https://github.com/open-telemetry/oteps/blob/main/text/0225-configuration.md) for agent configuration which addresses all configuration needs, including sampling. As long as the standard SDK samplers are used, the YAML representation of sampling configuration in that specification is identical to that of this proposal. + +However, here the following extended capabilities are offered: + +- additional composite samplers are built-in +- new (custom) samplers can be defined without adding any code to the SDK or SDK extension +- sampler configuration changes can be applied dynamically upon detecting configuration file modification, and supporting remote configuration via a wire protocol remains an option ## Suggested deployment pattern -Using the file-based configuration for sampling as described in this proposal does not require any changes to the OpenTelemetry Java Agent. The YAML file parser and the code to instantiate and configure requested samplers can be provided as an Extension jarfile (`config_based_sampler.jar` below). All samplers from the `opentelemetry-java-contrib` repository can be also made available as a separate jarfile (`all_samplers.jar` below). +Using the file-based configuration for sampling as described in this proposal does not require any changes to the OpenTelemetry Java Agent. The YAML file parser and the code to instantiate and configure requested samplers can be provided as an [extension](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/extension) jarfile (`config_based_sampler.jar` below). All samplers from the `opentelemetry-java-contrib` repository can be also made available as a separate jarfile (`all_samplers.jar` below). ```bash $ java -javaagent:path/to/opentelemetry-javaagent.jar \ @@ -237,20 +249,20 @@ $ java -javaagent:path/to/opentelemetry-javaagent.jar \ -jar myapp.jar ``` -If the feature proves popular, other options can be considered. +If the feature proves popular, other options can be considered, including adding this sampler along with its auxiliary composite samplers to the tracing SDK. Assuming that the sampler described in this proposal is called `meta_sampler`, the Java Agent users will be able to configure it as follows, when using [file-based configuration](https://github.com/open-telemetry/oteps/blob/main/text/0225-configuration.md#configuration-file) -## Compatibility with existing standards and proposals - -There already exists a [specification](https://github.com/open-telemetry/oteps/pull/225) for agent configuration which addresses sampling configuration. As long as the standard SDK samplers are used, the YAML representation of sampling configuration in that specification is identical to that of this proposal. -However, here the following extended capabilities are offered: - -- additional composite samplers are built-in -- new (custom) samplers can be defined without adding any code to the SDK or SDK extension -- sampler configuration changes can be applied dynamically upon detecting configuration file modification, and supporting remote configuration via a wire protocol is an option +```yaml +sdk: + tracer_provider: + sampler_config: + meta_sampler: + config_file: path/to/sampling_config.yaml + ... + +``` ## Open Issues - How to encode Predicates so they will be readable and yet powerful, and at the same time calculated efficiently? - How to describe sampler parameter types to arrive at the _right_ balance of complexity and expressiveness? - How to handle RecordOnly (_do not export_) sampling decisions by the composite samplers? - From 76dbe46ff168c0996d9554ce241f4b0d267db486 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 1 Dec 2023 14:41:28 -0800 Subject: [PATCH 11/13] Changing the description of the composite samplers. --- text/0240-Sampling_Configuration.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index 98951ba38..f08ba9ebd 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -100,8 +100,8 @@ New composite samplers are proposed to make group sampling decisions. They alway - ``` -The `any_sampler` is a composite sampler which takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list to find a sampler that decides to sample. If found, this sampling decision is final. If none of the samplers from the list wants to sample the span, the composite sampler drops the span. -If the first child which decided to sample modified the trace state, the effect of the modification remains in effect. +The `any_sampler` is a composite sampler which takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list and asks each sampler for its decision about sampling. If any of these samplers decides to sample, the final decision is to sample. If none of the samplers from the list wants to sample the span, the composite sampler drops the span. +State trace modfications by all the child samplers are cumulative. ### Logical-And Sampling @@ -114,9 +114,8 @@ If the first child which decided to sample modified the trace state, the effect - ``` -The `all_samplers` composite sampler takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list to find a sampler that decides not to sample. If found, the final sampling decision is not to sample. If all of the samplers from the list want to sample the span, the composite sampler samples the span. - -If all of the child samplers agreed to sample, and some of them modified the trace state, the modifications are cumulative as performed in the given order. If the final decision is not to sample, the trace state remains unchanged. +The `all_samplers` composite sampler takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list and asks each sampler for its decision about sampling. If any of these samplers decides not to sample, the final sampling decision is not to sample. If all of the samplers from the list want to sample the span, the composite sampler samples the span. +State trace modfications by all the child samplers are cumulative. ### Rule based sampling From 3fd967780b09abc4a5f3aaaa026957b1d8c2f4be Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Tue, 19 Dec 2023 22:39:15 -0800 Subject: [PATCH 12/13] Added a note on limitations --- text/0240-Sampling_Configuration.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index f08ba9ebd..47685a0ac 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -114,8 +114,8 @@ State trace modfications by all the child samplers are cumulative. - ``` -The `all_samplers` composite sampler takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list and asks each sampler for its decision about sampling. If any of these samplers decides not to sample, the final sampling decision is not to sample. If all of the samplers from the list want to sample the span, the composite sampler samples the span. -State trace modfications by all the child samplers are cumulative. +The `all_samplers` composite sampler takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list and asks each sampler for its decision about sampling. If any of these samplers decides not to sample, the final sampling decision is not to sample, and the remaining samplers are skipped. If all of the samplers from the list decide to sample the span, the composite sampler samples the span. +State trace modfications by all the referenced child samplers are cumulative. ### Rule based sampling @@ -142,6 +142,15 @@ where RULE is a pair of PREDICATE and SAMPLER: The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be taken), and perform tests on the accessible values. For example, one can test if the target URL for a SERVER span matches a given pattern. +## Limitations of composite samplers + +Not all samplers can participate as components of composite samplers without undesired or unexpected effects. In general, samplers can update trace-state regardless of their sampling decision, and can maintain internal state as well. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. +Such samplers assume that their sampling decisions will be honored by the tracer at the face value in all cases. + +A special attention is required for [consistent probability](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP) samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. Mixing CP samplers with other types of samplers in most cases will lead to incorrect adjusted counts. The family of CP samplers has its own [composition rules](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#composition-rules), which correctly handle composing multiple CP samplers. + +There are other choices for defining the behavior of Logical-Or and Logical-And composite samplers. One such option would be to allow consequent component samplers of Logical-And to see the trace-state as modified by their predecessors in the list. However, this would seem to violate the expectation that the trace-state as accessible via the `Context` parameter for [`shouldSample`](https://opentelemetry.io/docs/specs/otel/trace/sdk/#shouldsample) represents the values as left by the span parent rather than the predecessor samplers. + ## Example Let's assume that a user wants to configure head sampling as follows: @@ -153,7 +162,7 @@ Let's assume that a user wants to configure head sampling as follows: - for non-root spans - follow the parent sampling decision - however, capture all calls to service `/foo` (even if the trace will be incomplete) -- in any case, do not exceed 1000 spans/second +- in any case, do not exceed 1000 spans/minute Such sampling requirements can be expressed as: @@ -189,7 +198,7 @@ sampler: - predicate: http.url == /foo sampler: always_on - rate_limiting: - max_rate: 1000 # spans/second + max_rate: 1000 # spans/minute ``` The above example uses plain text representation for predicates. Actual representation for predicates is TBD. The example also assumes that a hypothetical `rate_limiting` sampler is available. @@ -206,7 +215,7 @@ For example, let's assume that the `rate_limiting` sampler from the previous exa sampler_definitions: rate_limiting: # this is the sampler key parameters: - - max_rate: number # spans per second + - max_rate: number # spans per minute implementation: # Java-specific part class_name: java/io/opentelemetry/contrib/sampler/RateLimitingSampler class_path: path/to/my/new/sampler.jar # additional classpath, optional From fe4d4ab7bc74e615fa69b40cd4974a3123f3e244 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 5 Jan 2024 11:07:47 -0800 Subject: [PATCH 13/13] Adding some clarifications. --- text/0240-Sampling_Configuration.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/text/0240-Sampling_Configuration.md b/text/0240-Sampling_Configuration.md index 47685a0ac..b954a2b4b 100644 --- a/text/0240-Sampling_Configuration.md +++ b/text/0240-Sampling_Configuration.md @@ -101,7 +101,9 @@ New composite samplers are proposed to make group sampling decisions. They alway ``` The `any_sampler` is a composite sampler which takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list and asks each sampler for its decision about sampling. If any of these samplers decides to sample, the final decision is to sample. If none of the samplers from the list wants to sample the span, the composite sampler drops the span. -State trace modfications by all the child samplers are cumulative. +[Trace state](https://opentelemetry.io/docs/specs/otel/trace/api/#tracestate) modfications by all the child samplers are cumulative. + +Each direct component of `any_sampler` is guaranteed to participate in the sampling decision (be invoked) and will see the same _parent_ state. The order of the component samplers in the list does not matter, as long as there's no overlap in the trace state keys they use. ### Logical-And Sampling @@ -114,8 +116,7 @@ State trace modfications by all the child samplers are cumulative. - ``` -The `all_samplers` composite sampler takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list and asks each sampler for its decision about sampling. If any of these samplers decides not to sample, the final sampling decision is not to sample, and the remaining samplers are skipped. If all of the samplers from the list decide to sample the span, the composite sampler samples the span. -State trace modfications by all the referenced child samplers are cumulative. +The `all_samplers` composite sampler takes a non-empty list of Samplers as the argument. When making a sampling decision, it goes through the list and asks each sampler for its decision about sampling. If any of these samplers decides not to sample, the final sampling decision is not to sample, and the remaining samplers are skipped. If all of the samplers from the list decide to sample the span, the composite sampler samples the span and the trace state modfications by all the referenced child samplers become cumulative. The trace state does not change if any of the child samplers decided not to sample. ### Rule based sampling @@ -144,12 +145,12 @@ For example, one can test if the target URL for a SERVER span matches a given pa ## Limitations of composite samplers -Not all samplers can participate as components of composite samplers without undesired or unexpected effects. In general, samplers can update trace-state regardless of their sampling decision, and can maintain internal state as well. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. +Not all samplers can participate as components of composite samplers without undesired or unexpected effects. In general, samplers can update trace state regardless of their sampling decision, and can maintain internal state as well. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. Such samplers assume that their sampling decisions will be honored by the tracer at the face value in all cases. A special attention is required for [consistent probability](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP) samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. Mixing CP samplers with other types of samplers in most cases will lead to incorrect adjusted counts. The family of CP samplers has its own [composition rules](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#composition-rules), which correctly handle composing multiple CP samplers. -There are other choices for defining the behavior of Logical-Or and Logical-And composite samplers. One such option would be to allow consequent component samplers of Logical-And to see the trace-state as modified by their predecessors in the list. However, this would seem to violate the expectation that the trace-state as accessible via the `Context` parameter for [`shouldSample`](https://opentelemetry.io/docs/specs/otel/trace/sdk/#shouldsample) represents the values as left by the span parent rather than the predecessor samplers. +There are other choices for defining the behavior of Logical-Or and Logical-And composite samplers. One such option would be to allow consequent component samplers of Logical-And to see the trace-state as modified by their predecessors in the list. However, this would seem to violate the expectation that the trace state as accessible via the `Context` parameter for [`shouldSample`](https://opentelemetry.io/docs/specs/otel/trace/sdk/#shouldsample) represents the values as left by the span parent rather than the predecessor samplers. ## Example