Skip to content

Commit

Permalink
Add Datadog exporter overall structure (#1142)
Browse files Browse the repository at this point in the history
* Add DataDog exporter back from old fork
commit 99129fb96e29e9c1a92da00b7e3f8efcae8a31e8
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Thu Sep 3 18:10:28 2020 +0200

    Handle namespace at initialization time

commit babca25927926a60c0c416294af3aadf784d41b9
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Thu Sep 3 17:23:53 2020 +0200

    Initialize on a separate function
    This way the variables can be checked without worrying about the env

commit 24d0cb4cc566fa5313a8650c904a27bea68bf555
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Thu Sep 3 14:30:35 2020 +0200

    Check environment variables for unified service tagging

commit 6695f8297ab8b1fcae71b05acb027c4a0992e3a0
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Wed Sep 2 14:57:37 2020 +0200

    Add support for sending metrics through the API
    - Use datadog.Metric type for simplicity
    - Get host if unset

commit c366603
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Wed Sep 2 09:56:24 2020 +0200

    Disable Queue and Retry settings (#72)

    These are handled by the statsd package.
    OpenTelemetry docs are confusing and the default configuration (disabled)
    is different from the one returned by "GetDefault..." functions

commit a660b56
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Tue Sep 1 15:26:14 2020 +0200

    Add support for summary and distribution metric types (#65)

    * Add support for summary metric type

    * Add support for distribution metrics

    * Refactor metrics construction
    - Drop name in Metrics (now they act as Metric values)
    - Refactor constructor so that errors happen at compile-time

    * Report Summary total sum and count values
    Snapshot values are not filled in by OpenTelemetry

    * Report p00 and p100 as `.min` and `.max`
    This is more similar to what we do for our own non-additive type

    * Keep hostname if it has not been overridden

commit c95adc4
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Thu Aug 27 13:00:02 2020 +0200

    Update dependencies and `make gofmt`
    The collector was updated to 0.9.0 upstream

commit 20afb0e
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Wed Aug 26 18:25:49 2020 +0200

    Refactor configuration (#45)

    * Refactor configuration
    * Implement telemetry and tags configuration handling
    * Update example configuration and README file
    Co-authored-by: Kylian Serrania <kylian.serrania@datadoghq.com>

commit fdc98b5
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Fri Aug 21 11:09:08 2020 +0200

    Initial DogStatsD implementation (#15)

    Initial metrics exporter through DogStatsD with support for all metric types but summary and distribution

commit e848a60
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Fri Aug 21 10:42:45 2020 +0200

    Bump collector version

commit 58be9a4
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Thu Aug 6 10:04:32 2020 +0200

    Address linter

commit 695430c
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Tue Aug 4 13:28:01 2020 +0200

    Fix field name error

    MetricsEndpoint was renamed to MetricsURL

commit 168b319
Author: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Date:   Mon Aug 3 11:05:01 2020 +0200

    Create initial outline for Datadog exporter (#1)

    * Add support for basic configuration options
    * Documents configuration options

* go mod tidy

* Address feedback from upstream PR we did not merge (#1)

* Backport changes from upstream PR

Remove `err` from MapMetrics

* Remove usage of pdatautil

* Fix tests

* Use TCPAddr

* Review which functions should be private

* Remove DogStatsD mode (#2)

* Remove DogStatsD mode

* go mod tidy

* Remove mentions to DogStatSD

* Improve test coverage (#3)

* Improve test coverage

Added unit tests for
- API key censoring
- Hostname
- Metrics exporter

Renamed test and implementation files for consistency

* Add one additional test

* Remove client validation (#6)

The zorkian API does not validate the API key unless you also have
an application key, even though the endpoint works without it.

I am removing this validation until this gets fixed on the zorkian library

* Keep only configuration and factory methods

Following the contribution guidelines we need to make a first PR
with this

* Use latest version of collector

* Remove `report_percentiles` option
It is not being used as of now until the OTLP metrics format
stabilizes and we have a Summary type metric again

* Correct configuration

The API key is now a required setting

* Remove test not relevant for this PR

* Remove unnecessary imports after removing test

* Address review comment

* Apply suggestions from code review

Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>

* Separate documentation into two examples
One example with the minimal configuration, for sending to `datadoghq.com`
and a second one for sending to `datadoghq.eu`

Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>
  • Loading branch information
mx-psi and tigrannajaryan authored Sep 30, 2020
1 parent 7f52fd7 commit 3e375ad
Show file tree
Hide file tree
Showing 12 changed files with 2,177 additions and 0 deletions.
1 change: 1 addition & 0 deletions exporter/datadogexporter/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include ../../Makefile.Common
24 changes: 24 additions & 0 deletions exporter/datadogexporter/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Datadog Exporter

This exporter sends metric data to [Datadog](https://datadoghq.com).

## Configuration

The only required setting is a [Datadog API key](https://app.datadoghq.com/account/settings#api).
```yaml
datadog:
api:
key: "<API key>"
```
To send data to the Datadog EU site, set the `api.site` parameter to `datadoghq.eu`:
```yaml
datadog:
api:
key: "<API key>"
site: datadoghq.eu
```

The hostname, environment, service and version can be set in the configuration for unified service tagging.

See the sample configuration file under the `example` folder for other available options.
145 changes: 145 additions & 0 deletions exporter/datadogexporter/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package datadogexporter

import (
"errors"
"fmt"
"strings"

"go.opentelemetry.io/collector/config/configmodels"
"go.opentelemetry.io/collector/config/confignet"
)

var (
errUnsetAPIKey = errors.New("api.key is not set")
)

// APIConfig defines the API configuration options
type APIConfig struct {
// Key is the Datadog API key to associate your Agent's data with your organization.
// Create a new API key here: https://app.datadoghq.com/account/settings
Key string `mapstructure:"key"`

// Site is the site of the Datadog intake to send data to.
// The default value is "datadoghq.com".
Site string `mapstructure:"site"`
}

// GetCensoredKey returns the API key censored for logging purposes
func (api *APIConfig) GetCensoredKey() string {
if len(api.Key) <= 5 {
return api.Key
}
return strings.Repeat("*", len(api.Key)-5) + api.Key[len(api.Key)-5:]
}

// MetricsConfig defines the metrics exporter specific configuration options
type MetricsConfig struct {
// Namespace is the namespace under which the metrics are sent
// By default metrics are not namespaced
Namespace string `mapstructure:"namespace"`

// Buckets states whether to report buckets from distribution metrics
Buckets bool `mapstructure:"report_buckets"`

// TCPAddr.Endpoint is the host of the Datadog intake server to send metrics to.
// If unset, the value is obtained from the Site.
confignet.TCPAddr `mapstructure:",squash"`
}

// TagsConfig defines the tag-related configuration
// It is embedded in the configuration
type TagsConfig struct {
// Hostname is the host name for unified service tagging.
// If unset, it is determined automatically.
// See https://docs.datadoghq.com/agent/faq/how-datadog-agent-determines-the-hostname
// for more details.
Hostname string `mapstructure:"hostname"`

// Env is the environment for unified service tagging.
// It can also be set through the `DD_ENV` environment variable.
Env string `mapstructure:"env"`

// Service is the service for unified service tagging.
// It can also be set through the `DD_SERVICE` environment variable.
Service string `mapstructure:"service"`

// Version is the version for unified service tagging.
// It can also be set through the `DD_VERSION` version variable.
Version string `mapstructure:"version"`

// Tags is the list of default tags to add to every metric or trace.
Tags []string `mapstructure:"tags"`
}

// GetTags gets the default tags extracted from the configuration
func (t *TagsConfig) GetTags(addHost bool) []string {
tags := make([]string, 0, 4)

vars := map[string]string{
"env": t.Env,
"service": t.Service,
"version": t.Version,
}

if addHost {
vars["host"] = t.Hostname
}

for name, val := range vars {
if val != "" {
tags = append(tags, fmt.Sprintf("%s:%s", name, val))
}
}

tags = append(tags, t.Tags...)

return tags
}

// Config defines configuration for the Datadog exporter.
type Config struct {
configmodels.ExporterSettings `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.

TagsConfig `mapstructure:",squash"`

// API defines the Datadog API configuration.
API APIConfig `mapstructure:"api"`

// Metrics defines the Metrics exporter specific configuration
Metrics MetricsConfig `mapstructure:"metrics"`
}

// Sanitize tries to sanitize a given configuration
func (c *Config) Sanitize() error {
// Add '.' at the end of namespace
if c.Metrics.Namespace != "" && !strings.HasSuffix(c.Metrics.Namespace, ".") {
c.Metrics.Namespace = c.Metrics.Namespace + "."
}

if c.API.Key == "" {
return errUnsetAPIKey
}

c.API.Key = strings.TrimSpace(c.API.Key)

// Set the endpoint based on the Site
if c.Metrics.TCPAddr.Endpoint == "" {
c.Metrics.TCPAddr.Endpoint = fmt.Sprintf("https://api.%s", c.API.Site)
}

return nil
}
136 changes: 136 additions & 0 deletions exporter/datadogexporter/config_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package datadogexporter

import (
"path"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.opentelemetry.io/collector/component/componenttest"
"go.opentelemetry.io/collector/config/configmodels"
"go.opentelemetry.io/collector/config/confignet"
"go.opentelemetry.io/collector/config/configtest"
)

// TestLoadConfig tests that the configuration is loaded correctly
func TestLoadConfig(t *testing.T) {
factories, err := componenttest.ExampleComponents()
assert.NoError(t, err)

factory := NewFactory()
factories.Exporters[typeStr] = factory
cfg, err := configtest.LoadConfigFile(t, path.Join(".", "testdata", "config.yaml"), factories)

require.NoError(t, err)
require.NotNil(t, cfg)

apiConfig := cfg.Exporters["datadog/api"].(*Config)
err = apiConfig.Sanitize()

require.NoError(t, err)
assert.Equal(t, &Config{
ExporterSettings: configmodels.ExporterSettings{
NameVal: "datadog/api",
TypeVal: typeStr,
},

TagsConfig: TagsConfig{
Hostname: "customhostname",
Env: "prod",
Service: "myservice",
Version: "myversion",
Tags: []string{"example:tag"},
},

API: APIConfig{
Key: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
Site: "datadoghq.eu",
},

Metrics: MetricsConfig{
Namespace: "opentelemetry.",
TCPAddr: confignet.TCPAddr{
Endpoint: "https://api.datadoghq.eu",
},
},
}, apiConfig)

invalidConfig2 := cfg.Exporters["datadog/invalid"].(*Config)
err = invalidConfig2.Sanitize()
require.Error(t, err)

}

func TestTags(t *testing.T) {
tc := TagsConfig{
Hostname: "customhost",
Env: "customenv",
Service: "customservice",
Version: "customversion",
Tags: []string{"key1:val1", "key2:val2"},
}

assert.ElementsMatch(t,
[]string{
"host:customhost",
"env:customenv",
"service:customservice",
"version:customversion",
"key1:val1",
"key2:val2",
},
tc.GetTags(true), // get host
)
}

// TestOverrideMetricsURL tests that the metrics URL is overridden
// correctly when set manually.
func TestOverrideMetricsURL(t *testing.T) {

const DebugEndpoint string = "http://localhost:8080"

cfg := Config{
API: APIConfig{Key: "notnull", Site: DefaultSite},
Metrics: MetricsConfig{
TCPAddr: confignet.TCPAddr{
Endpoint: DebugEndpoint,
},
},
}

err := cfg.Sanitize()
require.NoError(t, err)
assert.Equal(t, cfg.Metrics.Endpoint, DebugEndpoint)
}

func TestAPIKeyUnset(t *testing.T) {
cfg := Config{}
err := cfg.Sanitize()
assert.Equal(t, err, errUnsetAPIKey)
}

func TestCensorAPIKey(t *testing.T) {
cfg := APIConfig{
Key: "ddog_32_characters_long_api_key1",
}

assert.Equal(
t,
"***************************_key1",
cfg.GetCensoredKey(),
)
}
81 changes: 81 additions & 0 deletions exporter/datadogexporter/example/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
receivers:
examplereceiver:

processors:
exampleprocessor:

exporters:
datadog:
## @param hostname - string - optional
## A custom hostname.
## If unset this will be determined automatically if possible.
#
# hostname: customhostname

## @param env - string - optional
## The environment for unified service tagging.
## If unset it will be determined from the `DD_ENV` environment variable.
#
# env: prod

## @param service - string - optional
## The service for unified service tagging.
## If unset it will be determined from the `DD_SERVICE` environment variable.
#
# service: myservice

## @param version - string - optional
## The version for unified service tagging.
## If unset it will be determined from the `DD_VERSION` environment variable.
#
# version: myversion

## @param tags - list of strings - optional - default: []
## The list of default tags to add to every metric or trace.
#
# tags: []

## @param api - custom object - required.
## Specific API configuration.
#
api:
## @ param key - string - required
## The Datadog API key to associate your Agent's data with your organization.
## Create a new API key here: https://app.datadoghq.com/account/settings
#
key: "<YOUR API KEY>"

## @param site - string - optional - default: datadoghq.com
## The site of the Datadog intake to send Agent data to.
## Set to 'datadoghq.eu' to send data to the EU site.
#
# site: datadoghq.com

## @param metrics - custom object - optional
## Metric exporter specific configuration.
#
# metrics:
## @param namespace - string - optional
## The namespace with which to prefix all metrics.
## By default metrics are not namespaced.
#
# namespace: ""

## @param report_buckets - boolean - optional - default: false
## Whether to report bucket counts for distribution metric types.
## Enabling this will increase the number of custom metrics.
#
# report_buckets: false

## @param endpoint - string - optional
## The host of the Datadog intake server to send metrics to.
## If unset the value is obtained through the `site` parameter in the `api` section.
#
# endpoint: https://api.datadoghq.com

service:
pipelines:
traces:
receivers: [examplereceiver]
processors: [exampleprocessor]
exporters: [datadog]
Loading

0 comments on commit 3e375ad

Please sign in to comment.