Support OTEL_PROPAGATORS in distro (#295)

* Support OTEL_PROPAGATORS in distro

* Isolate distro pkg to fix test failure

* Add changes to changelog

* Add distro README

* Formatting and grammar fixes for distro/README.md

* Lazy load default propagator if unspecified

* Fix WithEndpoint and WithAccessToken docs to match

* Revert Validate comment

* Simplify WithEndpoint default value doc

* Update CHANGELOG.md

Co-authored-by: Robert Pająk <pellared@hotmail.com>

* Ensure nonePropagator is not wrapped in composite

* Include baggage context propagation in README docs

* Explain WithEndpoint envar opts

* go mod tidy

* Apply suggestions from code review

Co-authored-by: Robert Pająk <rpajak@splunk.com>

* Apply feedback

Co-authored-by: Robert Pająk <pellared@hotmail.com>
Co-authored-by: Robert Pająk <rpajak@splunk.com>

CHANGELOG.md

@@ -61,6 +61,10 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - Add the
   `github.com/signalfx/splunk-otel-go/instrumentation/github.com/gomodule/redigo/splunkredigo`
   instrumentation for the `github.com/gomodule/redigo` package. (#288)
+- Add the `WithPropagator` option to
+  `github.com/signalfx/splunk-otel-go/distro` along with parsing of the
+  `OTEL_PROPAGATORS` environment variable to set the global OpenTelemetry
+  `TextMapPropagator`. (#295)
 - Add the
   `github.com/signalfx/splunk-otel-go/instrumentation/gopkg.in/olivere/elastic/splunkelastic`
   instrumentation for the `gopkg.in/olivere/elastic` package. (#311)

README.md

@@ -33,7 +33,8 @@ Table of Contents:
 
 This Splunk distribution comes with the following defaults:
 
-- [B3 context propagation](https://github.com/openzipkin/b3-propagation).
+- [W3C tracecontext](https://www.w3.org/TR/trace-context/) and
+  [W3C baggage](https://www.w3.org/TR/baggage/) context propagation.
 - [Jaeger Thrift over HTTP
   exporter](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/jaeger)
   configured to send spans to a locally running [Splunk OpenTelemetry Connector](https://github.com/signalfx/splunk-otel-collector)
@@ -93,6 +94,8 @@ As well as in Go code before executing `distro.Run()`:
 os.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.name=my-app,service.version=1.2.3,deployment.environment=development")
 ```
 
+For advanced configuration options, refer to the [`distro` package documentation](./distro/README.md#Configuration).
+
 ## Library Instrumentation
 
 Supported libraries are listed

distro/README.md

@@ -0,0 +1,79 @@
+# Package `github.com/signalfx/splunk-otel-go/distro`
+
+This package provides a Splunk distribution of the OpenTelemetry Go SDK. It is
+designed to provide an SDK properly configured to be used with the Splunk
+platform out-of-the-box.
+
+## Getting Started
+
+The main entry point for the package is the [`Run`] function. Use this
+function to create an SDK that is ready to be used with OpenTelemetry and
+forward all telemetry to Splunk. See [`example_test.go`](./example_test.go) for
+a complete example.
+
+## Configuration
+
+The [`SDK`] is configured with the following options.
+
+| Option Name | Default Value | Environment Variable |
+| ---| --- | --- |
+| `WithAccessToken` | `""` | `SPLUNK_ACCESS_TOKEN` |
+| `WithEndpoint` | (1) | (2) |
+| `WithPropagator` | `tracecontext,baggage` | `OTEL_PROPAGATORS` |
+
+(1): The default value depends on the exporter used. See the
+[`WithEndpoint`](#withendpoint) section for more details.
+(2): The environment variable depends on the exporter used. See the
+[`WithEndpoint`](#withendpoint) section for more details.
+
+Environment variable can be used to set related option values, but the value
+set in code will take precedence. This is the same behavior the default
+OpenTelemetry SDK has.
+
+The following sections contain specific information for each option.
+
+### `WithAccessToken`
+
+`WithAccessToken` configures the authentication token used to authenticate
+telemetry sent directly to Splunk Observability Cloud.
+
+- Default value: empty (i.e. `""`)
+- Environment variable: `SPLUNK_ACCESS_TOKEN`
+
+### `WithEndpoint`
+
+`WithEndpoint` configures the Splunk endpoint that telemetry is sent to.
+
+- Default value: depends on the exporter used.
+  - For the `otlp` over gRPC exporter: `"localhost:4317"`
+  - For the `jaeger-thrift-splunk` exporter: `"http://127.0.0.1:9080/v1/trace"`
+- Environment variable: depends on the exporter used.
+  - For the `otlp` over gRPC exporter:
+    - `OTEL_EXPORTER_OTLP_ENDPOINT`
+    - `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`
+  - For the `jaeger-thrift-splunk` exporter: `OTEL_EXPORTER_JAEGER_ENDPOINT`
+
+### `WithPropagator`
+
+`WithPropagator` configures the OpenTelemetry `TextMapPropagator` set as the
+global `TextMapPropagator`. Setting to `nil` will prevent any global
+`TextMapPropagator` from being set.
+
+- Default value: A W3C tracecontext and baggage `TextMapPropagator`
+- Environment variable: `OTEL_PROPAGATORS`
+
+  The environment variable values are restricted to the following.
+  - `"tracecontext"`: W3C tracecontext
+  - `"baggage"`: W3C baggage
+  - `"b3"`: B3 single-header format
+  - `"b3multi"`: B3 multi-header format
+  - `"jaeger"`: Jaeger
+  - `"xray"`: AWS X-Ray
+  - `"ottrace"`: OpenTracing
+  - `"none"`: None; explicitly do not set any global propagator
+
+  Values can be joined with a comma (`","`) to produce a composite
+  `TextMapPropagator`.
+
+[`Run`]: https://pkg.go.dev/github.com/signalfx/splunk-otel-go/distro#Run
+[`SDK`]: https://pkg.go.dev/github.com/signalfx/splunk-otel-go/distro#SDK

distro/config.go

@@ -18,17 +18,33 @@ import (
 	"fmt"
 	"net/url"
 	"os"
+	"strings"
+
+	"go.opentelemetry.io/contrib/propagators/aws/xray"
+	"go.opentelemetry.io/contrib/propagators/b3"
+	"go.opentelemetry.io/contrib/propagators/jaeger"
+	"go.opentelemetry.io/contrib/propagators/ot"
+	"go.opentelemetry.io/otel/propagation"
 )
 
 // Environment variable keys that set values of the configuration.
 const (
+	// Access token added to exported data.
 	accessTokenKey = "SPLUNK_ACCESS_TOKEN"
+
+	// OpenTelemetry TextMapPropagator to set as global.
+	otelPropagatorsKey = "OTEL_PROPAGATORS"
+
+	// FIXME: support OTEL_SPAN_LINK_COUNT_LIMIT
+	// FIXME: support OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT
+	// FIXME: support OTEL_TRACES_EXPORTER
 )
 
 // config is the configuration used to create and operate an SDK.
 type config struct {
 	AccessToken string
 	Endpoint    string
+	Propagator  propagation.TextMapPropagator
 }
 
 // newConfig returns a validated config with Splunk defaults.
@@ -40,14 +56,22 @@ func newConfig(opts ...Option) (*config, error) {
 	for _, o := range opts {
 		o.apply(c)
 	}
+
+	// Apply default field values if they were not set.
+	if c.Propagator == nil {
+		c.Propagator = loadPropagator(
+			envOr(otelPropagatorsKey, "tracecontext,baggage"),
+		)
+	}
+
 	if err := c.Validate(); err != nil {
 		return nil, err
 	}
 	return c, nil
 }
 
 // Validate ensures c is valid, otherwise returning an appropriate error.
-func (c config) Validate() error {
+func (c *config) Validate() error {
 	var errs []string
 
 	if c.Endpoint != "" {
@@ -62,6 +86,82 @@ func (c config) Validate() error {
 	return nil
 }
 
+type nonePropagatorType struct{ propagation.TextMapPropagator }
+
+// nonePropagator signals the disablement of setting a global
+// TextMapPropagator.
+var nonePropagator = nonePropagatorType{}
+
+// propagators maps environment variable values to TextMapPropagator creation
+// functions.
+var propagators = map[string]func() propagation.TextMapPropagator{
+	// W3C Trace Context.
+	"tracecontext": func() propagation.TextMapPropagator {
+		return propagation.TraceContext{}
+	},
+	// W3C Baggage
+	"baggage": func() propagation.TextMapPropagator {
+		return propagation.Baggage{}
+	},
+	// B3 Single
+	"b3": func() propagation.TextMapPropagator {
+		return b3.New(b3.WithInjectEncoding(b3.B3SingleHeader))
+	},
+	// B3 Multi
+	"b3multi": func() propagation.TextMapPropagator {
+		return b3.New(b3.WithInjectEncoding(b3.B3MultipleHeader))
+	},
+	// Jaeger
+	"jaeger": func() propagation.TextMapPropagator {
+		return jaeger.Jaeger{}
+	},
+	// AWS X-Ray.
+	"xray": func() propagation.TextMapPropagator {
+		return xray.Propagator{}
+	},
+	// OpenTracing Trace
+	"ottrace": func() propagation.TextMapPropagator {
+		return ot.OT{}
+	},
+	// None, explicitly do not set a global propagator.
+	"none": func() propagation.TextMapPropagator {
+		return nonePropagator
+	},
+}
+
+func loadPropagator(name string) propagation.TextMapPropagator {
+	var props []propagation.TextMapPropagator
+	for _, part := range strings.Split(name, ",") {
+		factory, ok := propagators[part]
+		if !ok {
+			// Skip invalid data.
+			// TODO: log this.
+			continue
+		}
+
+		p := factory()
+		if p == nonePropagator {
+			// Make sure the disablement of the global propagator does not get
+			// lost as a composite below.
+			return p
+		}
+		props = append(props, p)
+	}
+
+	switch len(props) {
+	case 0:
+		// Default to "tracecontext,baggage".
+		return propagation.NewCompositeTextMapPropagator(
+			propagation.TraceContext{},
+			propagation.Baggage{},
+		)
+	case 1:
+		return props[0]
+	default:
+		return propagation.NewCompositeTextMapPropagator(props...)
+	}
+}
+
 // envOr returns the environment variable value associated with key if it
 // exists, otherwise it returns alt.
 func envOr(key, alt string) string {
@@ -84,19 +184,45 @@ func (fn optionFunc) apply(c *config) {
 	fn(c)
 }
 
-// WithEndpoint configures the endpoint telemetry is sent to.
-// Setting empty string results in no operation.
+// WithEndpoint configures the endpoint telemetry is sent to. Passing an empty
+// string results in the default value being used.
 func WithEndpoint(endpoint string) Option {
 	return optionFunc(func(c *config) {
 		c.Endpoint = endpoint
 	})
 }
 
-// WithAccessToken configures the authentication token
-// allowing exporters to send data directly to a Splunk back-end.
-// Setting empty string results in no operation.
+// WithAccessToken configures the authentication token used to authenticate
+// telemetry sent directly to Splunk Observability Cloud. Passing an empty
+// string results in no authentication token being used, and assumes
+// authentication is handled by another system.
+//
+// The SPLUNK_ACCESS_TOKEN environment variable value is used if this Option
+// is not provided.
 func WithAccessToken(accessToken string) Option {
 	return optionFunc(func(c *config) {
 		c.AccessToken = accessToken
 	})
 }
+
+// WithPropagator configures the OpenTelemetry TextMapPropagator set as the
+// global TextMapPropagator. Passing nil will prevent any TextMapPropagator
+// from being set.
+//
+// The OTEL_PROPAGATORS environment variable value is used if this Option is
+// not provided.
+//
+// By default, a tracecontext and baggage TextMapPropagator is set as the
+// global TextMapPropagator if this is not provided or the OTEL_PROPAGATORS
+// environment variable is not set.
+func WithPropagator(p propagation.TextMapPropagator) Option {
+	return optionFunc(func(c *config) {
+		if p == nil {
+			// Set to nonePropagator so when environment variable overrides
+			// are applied this is distinguishable from no WithPropagator
+			// option being passed.
+			p = nonePropagator
+		}
+		c.Propagator = p
+	})
+}

distro/config_test.go

@@ -19,6 +19,11 @@ import (
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	"go.opentelemetry.io/contrib/propagators/aws/xray"
+	"go.opentelemetry.io/contrib/propagators/b3"
+	"go.opentelemetry.io/contrib/propagators/jaeger"
+	"go.opentelemetry.io/contrib/propagators/ot"
+	"go.opentelemetry.io/otel/propagation"
 )
 
 type KeyValue struct {
@@ -90,6 +95,41 @@ var ConfigurationTests = []*ConfigFieldTest{
 			},
 		},
 	},
+	{
+		Name: "Propagator",
+		ValueFunc: func(c *config) interface{} {
+			return c.Propagator
+		},
+		DefaultValue: propagation.NewCompositeTextMapPropagator(
+			propagation.TraceContext{},
+			propagation.Baggage{},
+		),
+		EnvironmentTests: []KeyValue{
+			{Key: otelPropagatorsKey, Value: "tracecontext"},
+		},
+		OptionTests: []OptionTest{
+			{
+				Name: "nil propagator",
+				Options: []Option{
+					WithPropagator(nil),
+				},
+				AssertionFunc: func(t *testing.T, c *config, e error) {
+					assert.NoError(t, e)
+					assert.Equal(t, nonePropagator, c.Propagator)
+				},
+			},
+			{
+				Name: "set to tracecontext",
+				Options: []Option{
+					WithPropagator(propagation.TraceContext{}),
+				},
+				AssertionFunc: func(t *testing.T, c *config, e error) {
+					assert.NoError(t, e)
+					assert.Equal(t, propagation.TraceContext{}, c.Propagator)
+				},
+			},
+		},
+	},
 }
 
 func TestConfig(t *testing.T) {
@@ -145,3 +185,33 @@ func testOptions(t *testing.T, tc *ConfigFieldTest) {
 		}(t, o)
 	}
 }
+
+func TestLoadPropagatorComposite(t *testing.T) {
+	assert.Equal(t, propagation.NewCompositeTextMapPropagator(
+		propagation.TraceContext{},
+		propagation.Baggage{},
+		b3.New(b3.WithInjectEncoding(b3.B3SingleHeader)),
+		b3.New(b3.WithInjectEncoding(b3.B3MultipleHeader)),
+		jaeger.Jaeger{},
+		xray.Propagator{},
+		ot.OT{},
+	), loadPropagator("tracecontext,baggage,b3,b3multi,jaeger,xray,ottrace,garbage"))
+}
+
+func TestLoadPropagatorDefault(t *testing.T) {
+	assert.Equal(t, propagation.NewCompositeTextMapPropagator(
+		propagation.TraceContext{},
+		propagation.Baggage{},
+	), loadPropagator(""))
+}
+
+func TestLoadPropagatorCompositeWithNone(t *testing.T) {
+	// Assumes specification as stated:
+	//
+	//   "none": No automatically configured propagator.
+	//
+	// means if "none" is included in the value, no propagator should be
+	// configured. Therefore, loadPropagator needs to return just the
+	// nonePropagator value to signal this behavior.
+	assert.Equal(t, nonePropagator, loadPropagator("tracecontext,baggage,none"))
+}