Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graphite Input Protocol Parsing #3125

Merged
merged 29 commits into from
Jun 25, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
c130efb
Add fields to config metric name schema of graphite
cannium Jun 12, 2015
2a383e6
Fix unit tests for graphite
cannium Jun 12, 2015
d539b23
Move graphite parser to separate file
jwilder Jun 22, 2015
d132263
Add support for measurement*
jwilder Jun 22, 2015
a7d4d97
Convert template matching to matcher struct
jwilder Jun 22, 2015
d69a21c
Allow skipping fields in templates
jwilder Jun 22, 2015
9bcbbd9
Rename matcher to template
jwilder Jun 23, 2015
46046c6
Add matcher for match a metric to a template
jwilder Jun 23, 2015
e5fbf24
Remove DecodeNameAndTags func
jwilder Jun 23, 2015
cab9e36
Add basic template filtering support
jwilder Jun 23, 2015
9cd82ae
Add support for global tags
jwilder Jun 23, 2015
b55981f
Add support for per-template default tags
jwilder Jun 23, 2015
fed8d67
Add validation for graphite config templates and tags
jwilder Jun 23, 2015
dd0e6e5
Use strings.Fields to bef more forgiving of whitespace
jwilder Jun 23, 2015
ea348dd
Add sample graphite config to default config
jwilder Jun 23, 2015
1ecf9b5
Fix validation failing when using a default template
jwilder Jun 23, 2015
b294930
Add graphite plugin readme
jwilder Jun 23, 2015
98cbfdc
Update tempalte format comment
jwilder Jun 24, 2015
a2a1956
Use search tree for filter matching
jwilder Jun 24, 2015
f70eee6
Add support for multiple measurement fields in templates
jwilder Jun 24, 2015
613b1d2
Prevent duplicate filters in config
jwilder Jun 24, 2015
9ed71ad
Add test for matching similar patterns
jwilder Jun 24, 2015
a76e812
Add graphite parser benchmark
jwilder Jun 24, 2015
ba7187f
Add comments to graphite parser
jwilder Jun 24, 2015
320a951
Fix default template being returned when partially matching
jwilder Jun 24, 2015
b0cda03
Update changelog
jwilder Jun 24, 2015
fbfb90d
Code review fixes
jwilder Jun 25, 2015
c5a10cf
Use raw metric name when default template fails to match
jwilder Jun 25, 2015
562d7cd
Handle timestamp special cases
jwilder Jun 25, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

- [2650](https://github.com/influxdb/influxdb/pull/2650): Add SHOW GRANTS FOR USER statement. Thanks @n1tr0g.
- [3013](https://github.com/influxdb/influxdb/issues/3013): Panic error with inserting values with commas
- [3125](https://github.com/influxdb/influxdb/pull/3125): Graphite Input Protocol Parsing

### Bugfixes

Expand All @@ -28,6 +29,8 @@
- [#2944](https://github.com/influxdb/influxdb/issues/2944): Don't require "WHERE time" when creating continuous queries.
- [#3075](https://github.com/influxdb/influxdb/pull/3075): GROUP BY correctly when different tags have same value.
- [#3078](https://github.com/influxdb/influxdb/pull/3078): Fix CLI panic on malformed INSERT.
- [2102](https://github.com/influxdb/influxdb/issues/2102): Re-work Graphite input and metric processing
- [2996](https://github.com/influxdb/influxdb/issues/2996): Graphite Input Parsing

## v0.9.0 [2015-06-11]

Expand Down
7 changes: 7 additions & 0 deletions cmd/influxd/run/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ func NewConfig() *Config {
c.HTTPD = httpd.NewConfig()
c.Collectd = collectd.NewConfig()
c.OpenTSDB = opentsdb.NewConfig()
c.Graphites = append(c.Graphites, graphite.NewConfig())

c.Monitoring = monitor.NewConfig()
c.ContinuousQuery = continuous_querier.NewConfig()
Expand Down Expand Up @@ -97,5 +98,11 @@ func (c *Config) Validate() error {
} else if c.HintedHandoff.Dir == "" {
return errors.New("HintedHandoff.Dir must be specified")
}

for _, g := range c.Graphites {
if err := g.Validate(); err != nil {
return fmt.Errorf("invalid graphite config: %v", err)
}
}
return nil
}
21 changes: 20 additions & 1 deletion etc/config.sample.toml
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,26 @@ reporting-disabled = false
# protocol = "tcp"
# consistency-level = "one"
# name-separator = "."
# name-position = "last"

## "name-schema" configures tag names for parsing the metric name from graphite protocol;
## separated by `name-separator`.
## The "measurement" tag is special and the corresponding field will become
## the name of the metric.
## e.g. "type.host.measurement.device" will parse "server.localhost.cpu.cpu0" as
## {
## measurement: "cpu",
## tags: {
## "type": "server",
## "host": "localhost,
## "device": "cpu0"
## }
## }
# name-schema = "type.host.measurement.device"

## If set to true, when the input metric name has more fields than `name-schema` specified,
## the extra fields will be ignored.
## Otherwise an error will be logged and the metric rejected.
# ignore-unnamed = true

###
### [collectd]
Expand Down
124 changes: 124 additions & 0 deletions services/graphite/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
## Introduction

The graphite plugin allows measurements to be saved using the graphite line protocol. By default, enabling the graphite plugin will allow you to collect metrics and store them using the metric name as the measurement. If you send a metric named `servers.localhost.cpu.loadavg.10`, it will store the full metric name as the measurement with no extracted tags.

While this default setup works, it is not the ideal way to store measurements in InfluxDB since it does not take advantage of tags. It also will not perform optimally with a large dataset sizes since queries will be forced to use regexes which is known to not scale well.

To extract tags from metrics, one or more templates must be configured to parse metrics into tags and measurements.

## Templates

Templates allow matching parts of a metric name to be used as tag names in the stored metric. They have a similar format to graphite metric names. The values in between the separators are used as the tag name. The location of the tag name that matches the same position as the graphite metric section is used as the value. If there is no value, the graphite portion is skipped.

The special value _measurement_ is used to define the measurement name. It can have a trailing `*` to indicate that the remainder of the metric should be used. If a _measurement_ is not specified, the full metric name is used.

### Basic Matching

`servers.localhost.cpu.loadavg.10`
* Template: `.host.resource.measurement*`
* Output: _measurement_ =`loading.10` _tags_ =`host=localhost resource=cpu`

### Multiple Measurement Matching

The _measurement_ can be specified multiple times in a template to provide more control over the measurement name. Multiple values
will be joined together using the _Separator_ config variable. By default, this value is `.`.

`servers.localhost.cpu.cpu0.user`
* Template: `.host.measurement.cpu.measurement`
* Output: _measurement_ = `cpu.user` _tags_ = `host=localhost cpu=cpu0`

Since '.' requires queries on measurements to be double-quoted, you may want to set this to `_` to simplify querying parsed metrics.

`servers.localhost.cpu.cpu0.user`
* Separator: `_`
* Template: `.host.measurement.cpu.measurement`
* Output: _measurement_ = `cpu_user` _tags_ = `host=localhost cpu=cpu0`

### Adding Tags

Additional tags can be added to a metric that don't exist on the received metric. You can add additional tags by specifying them after the pattern. Tags have the same format as the line protocol. Multiple tags are separated by commas.

`servers.localhost.cpu.loadavg.10`
* Template: `.host.resource.measurement* region=us-west,zone=1a`
* Output: _measurement_ = `loading.10` _tags_ = `host=localhost resource=cpu region=us-west zone=1a`

## Multiple Templates

One template may not match all metrics. For example, using multiple plugins with diamond will produce metrics in different formats. If you need to use multiple templates, you'll need to define a prefix filter that must match before the template can be applied.

### Filters

Filters have a similar format to templates but work more like wildcard expressions. When multiple filters would match a metric, the more specific one is chosen. Filters are configured by adding them before the template.

For example,

```
servers.localhost.cpu.loadavg.10
servers.host123.elasticsearch.cache_hits 100
servers.host456.mysql.tx_count 10
```
* `servers.*` would match all values
* `servers.*.mysql` would match `servers.host456.mysql.tx_count 10`
* `servers.localhost.*` would match `servers.localhost.cpu.loadavg`

## Default Templates

If no template filters are defined or you want to just have one basic template, you can define a default template. This template will apply to any metric that has not already matched a filter.

```
dev.http.requests.200
prod.myapp.errors.count
dev.db.queries.count
```

* `env.app.measurement*` would create
* _measurement_=`requests.200` _tags_=`env=dev,app=http`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are tags comma- or space-separated? (see "Adding Tags" above)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When used in the template, the should be comma separated. The tags should be separated from the template which a space though.

* _measurement_= `errors.count` _tags_=`env=prod,app=myapp`
* _measurement_=`queries.count` _tags_=`env=dev,app=db`

## Global Tags

If you need to add the same set of tags to all metrics, you can define them globally at the plugin level and not within each template description.

## Minimal Config
```
[[graphite]]
enabled = true
# bind-address = ":2003"
# protocol = "tcp"
# consistency-level = "one"

### If matching multiple measurement files, this string will be used to join the matched values.
# separator = "."

### Default tags that will be added to all metrics. These can be overridden at the template level
### or by tags extracted from metric
# tags = ["region=us-east", "zone=1c"]

### Each template line requires a template pattern. It can have an optional
### filter before the template and separated by spaces. It can also have optional extra
### tags following the template. Multiple tags should be separated by commas and no spaces
### similar to the line protocol format. The can be only one default template.
# templates = [
# "*.app env.service.resource.measurement",
# # Default template
# "server.*",
#]
```

## Customized Config ```
[[graphite]]
enabled = true
separator = "_"
tags = ["region=us-east", "zone=1c"]
templates = [
# filter + template
"*.app env.service.resource.measurement",

# filter + template + extra tag
"stats.* .host.measurement* region=us-west,agent=sensu",

# default template. Ignore the first graphite component "servers"
".measurement*",
]
```
Loading