Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS X-Ray Exporter #41

Merged
merged 68 commits into from
Dec 16, 2019
Merged
Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
c0d6c5b
initial def of aws xray exporter
kbrockhoff Nov 8, 2019
f5079de
initial dev of xray data structures and converters
kbrockhoff Nov 13, 2019
b0b6901
initial dev of xray data structures and converters
kbrockhoff Nov 14, 2019
72a0683
initial dev of xray data structures and converters
kbrockhoff Nov 14, 2019
9dee5a1
initial dev of xray data structures and converters
kbrockhoff Nov 15, 2019
5e22689
initial dev of xray data structures and converters
kbrockhoff Nov 16, 2019
2dc3c2c
initial dev of xray data structures and converters
kbrockhoff Nov 18, 2019
e98cd2e
initial dev of xray data structures and converters
kbrockhoff Nov 19, 2019
31e123e
initial dev of export handler
kbrockhoff Nov 20, 2019
bccb369
Merge remote-tracking branch 'upstream/master'
kbrockhoff Nov 20, 2019
7bea1a5
fix formatting and lint errors
kbrockhoff Nov 20, 2019
e50934c
initial dev of export handler
kbrockhoff Nov 20, 2019
7548f34
added to component list
kbrockhoff Nov 20, 2019
1e3e047
fix issues raised during code review
kbrockhoff Nov 21, 2019
c92085c
switch user attribute name to constant
kbrockhoff Nov 21, 2019
7d62e94
Merge remote-tracking branch 'upstream/master'
kbrockhoff Nov 21, 2019
27d9962
fixed additional code review issues
kbrockhoff Nov 26, 2019
dfa1427
Merge remote-tracking branch 'upstream/master'
kbrockhoff Nov 27, 2019
b201406
fixed additional code review issues
kbrockhoff Nov 27, 2019
c1c8b9c
temporarily change package name
kbrockhoff Nov 27, 2019
338caca
temporarily change package name
kbrockhoff Nov 27, 2019
a47e06b
revert temporarily change package name
kbrockhoff Nov 27, 2019
3f4154a
fixed additional code review issues
kbrockhoff Nov 27, 2019
87c63cb
Merge remote-tracking branch 'upstream/master'
kbrockhoff Dec 4, 2019
7d0c7df
switched to constants defined in collector
kbrockhoff Dec 4, 2019
aba55dc
Merge remote-tracking branch 'upstream/master'
kbrockhoff Dec 5, 2019
d0f2748
switched to status conversion functions defined in collector
kbrockhoff Dec 5, 2019
f41e865
Merge remote-tracking branch 'upstream/master'
kbrockhoff Dec 7, 2019
c7abdda
fix latest code review issues
kbrockhoff Dec 7, 2019
7e732c0
Merge remote-tracking branch 'upstream/master'
kbrockhoff Dec 11, 2019
9189a5a
fix new static check issues
kbrockhoff Dec 11, 2019
851184f
Merge remote-tracking branch 'upstream/master'
kbrockhoff Dec 11, 2019
e8ac424
fix test that breaks if no valid aws session available
kbrockhoff Dec 12, 2019
e900d2d
Update Collector Core dependency to latest `master` (#61)
tigrannajaryan Dec 12, 2019
af8a37b
Ported kinesis exporter from Omnition (#60)
owais Dec 13, 2019
9d176f5
fix pull request issues
kbrockhoff Dec 13, 2019
4d8548a
Updated SAPM exporter dependencies (#59)
owais Dec 13, 2019
55b8658
Add E2E test for SAPM Receiver (#63)
tigrannajaryan Dec 13, 2019
e747a7e
Updated component dependencies (#64)
owais Dec 13, 2019
0d77b22
Fix build for go 1.13 (#65)
rghetia Dec 13, 2019
7f4c30a
Add E2E test for SAPM Exporter (#66)
tigrannajaryan Dec 13, 2019
4f69d2c
initial def of aws xray exporter
kbrockhoff Nov 8, 2019
65c5a41
initial dev of xray data structures and converters
kbrockhoff Nov 13, 2019
30506d5
initial dev of xray data structures and converters
kbrockhoff Nov 14, 2019
029eba8
initial dev of xray data structures and converters
kbrockhoff Nov 14, 2019
22c2ae6
initial dev of xray data structures and converters
kbrockhoff Nov 15, 2019
73e2541
initial dev of xray data structures and converters
kbrockhoff Nov 16, 2019
b6b1198
initial dev of xray data structures and converters
kbrockhoff Nov 18, 2019
42435c8
initial dev of xray data structures and converters
kbrockhoff Nov 19, 2019
b6c606b
initial dev of export handler
kbrockhoff Nov 20, 2019
4f0086a
fix formatting and lint errors
kbrockhoff Nov 20, 2019
3fb89a0
initial dev of export handler
kbrockhoff Nov 20, 2019
56623ed
added to component list
kbrockhoff Nov 20, 2019
42a3d06
fix issues raised during code review
kbrockhoff Nov 21, 2019
34442e6
switch user attribute name to constant
kbrockhoff Nov 21, 2019
2b6a513
fixed additional code review issues
kbrockhoff Nov 26, 2019
30566de
fixed additional code review issues
kbrockhoff Nov 27, 2019
142de99
temporarily change package name
kbrockhoff Nov 27, 2019
101ea0e
temporarily change package name
kbrockhoff Nov 27, 2019
5989f2b
revert temporarily change package name
kbrockhoff Nov 27, 2019
935e60c
fixed additional code review issues
kbrockhoff Nov 27, 2019
6f86a33
switched to constants defined in collector
kbrockhoff Dec 4, 2019
47c88e2
switched to status conversion functions defined in collector
kbrockhoff Dec 5, 2019
3915e0f
fix latest code review issues
kbrockhoff Dec 7, 2019
ddc4777
fix new static check issues
kbrockhoff Dec 11, 2019
e76baa4
fix test that breaks if no valid aws session available
kbrockhoff Dec 12, 2019
2ae696b
fix pull request issues
kbrockhoff Dec 13, 2019
ef18288
Merge branch 'master' of github.com:kbrockhoff/opentelemetry-collecto…
kbrockhoff Dec 14, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
switch user attribute name to constant
kbrockhoff committed Nov 21, 2019
commit c92085c45408146f86a1c2aff4ce55f209e47972
5 changes: 2 additions & 3 deletions exporter/awsxrayexporter/README.md
Original file line number Diff line number Diff line change
@@ -39,7 +39,8 @@ by the Span Resource object. X-Ray uses this data to generate inferred segments

## Exporter Configuration

The following exporter configuration parameters are supported.
The following exporter configuration parameters are supported. They mirror and have the same affect as the
comparable AWS X-Ray Daemon configuration values.

| Name | Description | Default |
| :---------------- | :--------------------------------------------------------------------- | ------- |
@@ -52,8 +53,6 @@ The following exporter configuration parameters are supported.
| `local_mode` | Local mode to skip EC2 instance metadata check. | false |
| `resource_arn` | Amazon Resource Name (ARN) of the AWS resource running the collector. | |
| `role_arn` | IAM role to upload segments to a different account. | |
| `user_attribute` | Span attribute name which holds the originating user's login. | |


## AWS Credential Configuration

8 changes: 3 additions & 5 deletions exporter/awsxrayexporter/awsxray.go
Original file line number Diff line number Diff line change
@@ -32,7 +32,6 @@ import (
func NewTraceExporter(config configmodels.Exporter, logger *zap.Logger, cn connAttr) (exporter.TraceExporter, error) {
typeLog := zap.String("type", config.Type())
nameLog := zap.String("name", config.Name())
userAttribute := config.(*Config).UserAttribute
awsConfig, session, err := GetAWSConfigSession(logger, cn, config.(*Config))
if err != nil {
return nil, err
@@ -42,7 +41,7 @@ func NewTraceExporter(config configmodels.Exporter, logger *zap.Logger, cn connA
config,
func(ctx context.Context, td consumerdata.TraceData) (int, error) {
logger.Debug("TraceExporter", typeLog, nameLog, zap.Int("#spans", len(td.Spans)))
droppedSpans, input := assembleRequest(userAttribute, td, logger)
droppedSpans, input := assembleRequest(td, logger)
logger.Debug("request: " + input.String())
output, err := xrayClient.PutTraceSegments(input)
logger.Debug("response: " + output.String())
@@ -57,8 +56,7 @@ func NewTraceExporter(config configmodels.Exporter, logger *zap.Logger, cn connA
)
}

func assembleRequest(userAttribute string, td consumerdata.TraceData,
logger *zap.Logger) (int, *xray.PutTraceSegmentsInput) {
func assembleRequest(td consumerdata.TraceData, logger *zap.Logger) (int, *xray.PutTraceSegmentsInput) {
documents := make([]*string, len(td.Spans))
droppedSpans := int(0)
for i, span := range td.Spans {
@@ -67,7 +65,7 @@ func assembleRequest(userAttribute string, td consumerdata.TraceData,
continue
}
spanName := span.Name.Value
jsonStr, err := translator.MakeSegmentDocumentString(spanName, span, userAttribute)
jsonStr, err := translator.MakeSegmentDocumentString(spanName, span)
if err != nil {
droppedSpans++
logger.Warn("Unable to convert span", zap.Error(err))
2 changes: 0 additions & 2 deletions exporter/awsxrayexporter/config.go
Original file line number Diff line number Diff line change
@@ -37,6 +37,4 @@ type Config struct {
ResourceARN string `mapstructure:"resource_arn"`
// IAM role to upload segments to a different account.
RoleARN string `mapstructure:"role_arn"`
// Span attribute name which holds the originating user's login.
UserAttribute string `mapstructure:"user_attribute"`
}
1 change: 0 additions & 1 deletion exporter/awsxrayexporter/config_test.go
Original file line number Diff line number Diff line change
@@ -56,6 +56,5 @@ func TestLoadConfig(t *testing.T) {
LocalMode: false,
ResourceARN: "arn:aws:ec2:us-east1:123456789:instance/i-293hiuhe0u",
RoleARN: "arn:aws:iam::123456789:role/monitoring-EKS-NodeInstanceRole",
UserAttribute: "user.id",
})
}
1 change: 0 additions & 1 deletion exporter/awsxrayexporter/factory.go
Original file line number Diff line number Diff line change
@@ -51,7 +51,6 @@ func (f *Factory) CreateDefaultConfig() configmodels.Exporter {
LocalMode: false,
ResourceARN: "",
RoleARN: "",
UserAttribute: "",
}
}

1 change: 0 additions & 1 deletion exporter/awsxrayexporter/testdata/config.yaml
Original file line number Diff line number Diff line change
@@ -10,7 +10,6 @@ exporters:
region: eu-west-1
resource_arn: "arn:aws:ec2:us-east1:123456789:instance/i-293hiuhe0u"
role_arn: "arn:aws:iam::123456789:role/monitoring-EKS-NodeInstanceRole"
user_attribute: "user.id"
awsxray/disabled: # will be ignored
disabled: true

17 changes: 8 additions & 9 deletions exporter/awsxrayexporter/translator/segment.go
Original file line number Diff line number Diff line change
@@ -38,6 +38,7 @@ const (
PeerIpv6Attribute = "peer.ipv6"
PeerPortAttribute = "peer.port"
PeerServiceAttribute = "peer.service"
IAMUserAttribute = "iam.user"
)

// OpenTelemetry Semantic Convention values for component attribute values.
@@ -147,8 +148,8 @@ var (
)

// MakeSegmentDocumentString converts an Otel Span to an X-Ray Segment and then serialzies to JSON
kbrockhoff marked this conversation as resolved.
Show resolved Hide resolved
func MakeSegmentDocumentString(name string, span *tracepb.Span, userAttribute string) (string, error) {
segment := MakeSegment(name, span, userAttribute)
func MakeSegmentDocumentString(name string, span *tracepb.Span) (string, error) {
segment := MakeSegment(name, span)
w := writers.borrow()
if err := w.Encode(segment); err != nil {
return "", err
@@ -159,7 +160,7 @@ func MakeSegmentDocumentString(name string, span *tracepb.Span, userAttribute st
}

// MakeSegment converts an Otel Span to an X-Ray Segment
func MakeSegment(name string, span *tracepb.Span, userAttribute string) Segment {
func MakeSegment(name string, span *tracepb.Span) Segment {
var (
traceID = convertToAmazonTraceID(span.TraceId)
startTime = timestampToFloatSeconds(span.StartTime, span.StartTime)
@@ -171,7 +172,7 @@ func MakeSegment(name string, span *tracepb.Span, userAttribute string) Segment
awsfiltered, aws = makeAws(causefiltered, span.Resource)
service = makeService(span.Resource)
sqlfiltered, sql = makeSQL(awsfiltered)
user, annotations = makeAnnotations(sqlfiltered, userAttribute)
user, annotations = makeAnnotations(sqlfiltered)
namespace string
)

@@ -318,16 +319,14 @@ func mergeAnnotations(dest map[string]interface{}, src map[string]string) {
}
}

func makeAnnotations(attributes map[string]string, userAttribute string) (string, map[string]interface{}) {
func makeAnnotations(attributes map[string]string) (string, map[string]interface{}) {
var (
result = map[string]interface{}{}
user string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

user is never set. why return the user?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by removing

)

if userAttribute != "" {
user = attributes[userAttribute]
delete(attributes, userAttribute)
}
user = attributes[IAMUserAttribute]
delete(attributes, IAMUserAttribute)
delete(attributes, ComponentAttribute)
mergeAnnotations(result, attributes)

57 changes: 8 additions & 49 deletions exporter/awsxrayexporter/translator/segment_test.go
Original file line number Diff line number Diff line change
@@ -46,7 +46,7 @@ func TestClientSpanWithGrpcComponent(t *testing.T) {
timeEvents := constructTimedEventsWithSentMessageEvent(span.StartTime)
span.TimeEvents = &timeEvents

jsonStr, err := MakeSegmentDocumentString(spanName, span, "user.id")
jsonStr, err := MakeSegmentDocumentString(spanName, span)

assert.NotNil(t, jsonStr)
assert.Nil(t, err)
@@ -56,8 +56,7 @@ func TestClientSpanWithGrpcComponent(t *testing.T) {
func TestClientSpanWithAwsSdkClient(t *testing.T) {
spanName := "AmazonDynamoDB.getItem"
parentSpanID := NewSegmentID()
userAttribute := "originating.user"
user := "testing"
user := "testingT"
attributes := make(map[string]interface{})
attributes[ComponentAttribute] = HTTPComponentType
attributes[MethodAttribute] = "POST"
@@ -68,11 +67,11 @@ func TestClientSpanWithAwsSdkClient(t *testing.T) {
attributes[AWSOperationAttribute] = "GetItem"
attributes[AWSRequestIDAttribute] = "18BO1FEPJSSAOGNJEDPTPCMIU7VV4KQNSO5AEMVJF66Q9ASUAAJG"
attributes[AWSTableNameAttribute] = "otel-dev-Testing"
attributes[userAttribute] = user
attributes[IAMUserAttribute] = user
labels := constructDefaultResourceLabels()
span := constructClientSpan(parentSpanID, spanName, 0, "OK", attributes, labels)

jsonStr, err := MakeSegmentDocumentString(spanName, span, userAttribute)
jsonStr, err := MakeSegmentDocumentString(spanName, span)

assert.NotNil(t, jsonStr)
assert.Nil(t, err)
@@ -84,7 +83,6 @@ func TestClientSpanWithAwsSdkClient(t *testing.T) {
func TestServerSpanWithInternalServerError(t *testing.T) {
spanName := "/api/locations"
parentSpanID := NewSegmentID()
userAttribute := "originating.user"
user := "testing"
errorMessage := "java.lang.NullPointerException"
attributes := make(map[string]interface{})
@@ -93,14 +91,14 @@ func TestServerSpanWithInternalServerError(t *testing.T) {
attributes[URLAttribute] = "https://api.example.org/api/locations"
attributes[TargetAttribute] = "/api/locations"
attributes[StatusCodeAttribute] = 500
attributes[userAttribute] = user
attributes[IAMUserAttribute] = user
attributes[ErrorKindAttribute] = "java.lang.NullPointerException"
labels := constructDefaultResourceLabels()
span := constructServerSpan(parentSpanID, spanName, tracetranslator.OCInternal, errorMessage, attributes, labels)
timeEvents := constructTimedEventsWithSentMessageEvent(span.StartTime)
span.TimeEvents = &timeEvents

segment := MakeSegment(spanName, span, "originating.user")
segment := MakeSegment(spanName, span)

assert.NotNil(t, segment)
assert.NotNil(t, segment.Cause)
@@ -114,7 +112,6 @@ func TestServerSpanWithInternalServerError(t *testing.T) {
testWriters.release(w)
assert.True(t, strings.Contains(jsonStr, spanName))
assert.True(t, strings.Contains(jsonStr, errorMessage))
assert.False(t, strings.Contains(jsonStr, userAttribute))
}

func TestClientSpanWithDbComponent(t *testing.T) {
@@ -133,45 +130,7 @@ func TestClientSpanWithDbComponent(t *testing.T) {
labels := constructDefaultResourceLabels()
span := constructClientSpan(nil, spanName, 0, "OK", attributes, labels)

segment := MakeSegment(spanName, span, "originating.user")

assert.NotNil(t, segment)
assert.NotNil(t, segment.SQL)
assert.NotNil(t, segment.Service)
assert.NotNil(t, segment.AWS)
assert.NotNil(t, segment.Annotations)
assert.Nil(t, segment.Cause)
assert.Nil(t, segment.HTTP)
assert.Equal(t, spanName, segment.Name)
assert.False(t, segment.Fault)
assert.False(t, segment.Error)
w := testWriters.borrow()
if err := w.Encode(segment); err != nil {
assert.Fail(t, "invalid json")
}
jsonStr := w.String()
testWriters.release(w)
assert.True(t, strings.Contains(jsonStr, spanName))
assert.True(t, strings.Contains(jsonStr, enterpriseAppID))
}

func TestClientSpanWithBlankUserAttribute(t *testing.T) {
spanName := "call update_user_preference( ?, ?, ? )"
enterpriseAppID := "25F2E73B-4769-4C79-9DF3-7EBE85D571EA"
attributes := make(map[string]interface{})
attributes[ComponentAttribute] = DbComponentType
attributes[DbTypeAttribute] = "sql"
attributes[DbInstanceAttribute] = "customers"
attributes[DbStatementAttribute] = spanName
attributes[DbUserAttribute] = "userprefsvc"
attributes[PeerAddressAttribute] = "mysql://db.dev.example.com:3306"
attributes[PeerHostAttribute] = "db.dev.example.com"
attributes[PeerPortAttribute] = "3306"
attributes["enterprise.app.id"] = enterpriseAppID
labels := constructDefaultResourceLabels()
span := constructClientSpan(nil, spanName, 0, "OK", attributes, labels)

segment := MakeSegment(spanName, span, "")
segment := MakeSegment(spanName, span)

assert.NotNil(t, segment)
assert.NotNil(t, segment.SQL)
@@ -208,7 +167,7 @@ func TestSpanWithInvalidTraceId(t *testing.T) {
span.TimeEvents = &timeEvents
span.TraceId[0] = 0x11

jsonStr, err := MakeSegmentDocumentString(spanName, span, "user.id")
jsonStr, err := MakeSegmentDocumentString(spanName, span)

assert.NotNil(t, jsonStr)
assert.Nil(t, err)