Skip to content

Commit

Permalink
Loki: Modified heroku drain target to make any url query parameters a…
Browse files Browse the repository at this point in the history 
…vailable as labels (#7619)

**What this PR does / why we need it**:

This PR reads the url query parameters included in Heroku drain POST
request and creates `__param_<name>` labels similar to how the
parameters work with prometheus metrics and metric endpoints as a way to
allow setting the true application name when ingesting logs from a
Heroku drain.

**Which issue(s) this PR fixes**:
Fixes #<issue number>

**Special notes for your reviewer**:

The existing `app` field coming from Heroku only specifies application
source (either the application itself or Heroku) and thus is
insufficient for uniquely identifying a log lines source.

**Checklist**
- [x] Reviewed the `CONTRIBUTING.md` guide
- [x] Documentation added
- [x] Tests updated
- [x] `CHANGELOG.md` updated
- [x] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
  • Loading branch information
@cadrake
cadrake authored Dec 1, 2022
1 parent eb7fbfe commit ea9ad33
Show file tree
Hide file tree
Showing 5 changed files with 151 additions and 6 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@

#### Promtail

* [7619](https://github.com/grafana/loki/pull/7619) **cadrake**: Add ability to pass query params to heroku drain targets for relabelling.

##### Enhancements

Expand Down
6 changes: 6 additions & 0 deletions clients/pkg/promtail/targets/heroku/target.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,12 @@ func (h *Target) drain(w http.ResponseWriter, r *http.Request) {
ts = message.Timestamp
}

// Create __heroku_drain_param_<name> labels from query parameters
params := r.URL.Query()
for k, v := range params {
lb.Set(fmt.Sprintf("__heroku_drain_param_%s", k), strings.Join(v, ","))
}

tenantIDHeaderValue := r.Header.Get("X-Scope-OrgID")
if tenantIDHeaderValue != "" {
// If present, first inject the tenant ID in, so it can be relabeled if necessary
Expand Down
124 changes: 119 additions & 5 deletions clients/pkg/promtail/targets/heroku/target_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"fmt"
"net"
"net/http"
"net/url"
"os"
"strings"
"testing"
Expand Down Expand Up @@ -33,14 +34,23 @@ const testLogLine1Timestamp = "2022-06-13T14:52:23.621815+00:00"
const testLogLine2 = `156 <190>1 2022-06-13T14:52:23.827271+00:00 host app web.1 - [GIN] 2022/06/13 - 14:52:23 | 200 | 163.92µs | 181.167.87.140 | GET "/static/main.css"
`

func makeDrainRequest(host string, bodies ...string) (*http.Request, error) {
func makeDrainRequest(host string, params map[string][]string, bodies ...string) (*http.Request, error) {
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/heroku/api/v1/drain", host), strings.NewReader(strings.Join(bodies, "")))
if err != nil {
return nil, err
}

drainToken := uuid.New().String()
frameID := uuid.New().String()

values := url.Values{}
for name, params := range params {
for _, p := range params {
values.Add(name, p)
}
}
req.URL.RawQuery = values.Encode()

req.Header.Set("Content-Type", "application/heroku_drain-1")
req.Header.Set("Logplex-Drain-Token", fmt.Sprintf("d.%s", drainToken))
req.Header.Set("Logplex-Frame-Id", frameID)
Expand All @@ -59,6 +69,7 @@ func TestHerokuDrainTarget(t *testing.T) {
}
type args struct {
RequestBodies []string
RequestParams map[string][]string
RelabelConfigs []*relabel.Config
Labels model.LabelSet
}
Expand All @@ -70,6 +81,27 @@ func TestHerokuDrainTarget(t *testing.T) {
"heroku request with a single log line, internal labels dropped, and fixed are propagated": {
args: args{
RequestBodies: []string{testPayload},
RequestParams: map[string][]string{},
Labels: model.LabelSet{
"job": "some_job_name",
},
},
expectedEntries: []expectedEntry{
{
labels: model.LabelSet{
"job": "some_job_name",
},
line: `at=info method=GET path="/" host=cryptic-cliffs-27764.herokuapp.com request_id=59da6323-2bc4-4143-8677-cc66ccfb115f fwd="181.167.87.140" dyno=web.1 connect=0ms service=3ms status=200 bytes=6979 protocol=https
`,
},
},
},
"heroku request with a single log line and query parameters, internal labels dropped, and fixed are propagated": {
args: args{
RequestBodies: []string{testPayload},
RequestParams: map[string][]string{
"some_query_param": []string{"app_123", "app_456"},
},
Labels: model.LabelSet{
"job": "some_job_name",
},
Expand All @@ -84,9 +116,37 @@ func TestHerokuDrainTarget(t *testing.T) {
},
},
},
"heroku request with a two log lines, internal labels dropped, and fixed are propagated": {
"heroku request with two log lines, internal labels dropped, and fixed are propagated": {
args: args{
RequestBodies: []string{testLogLine1, testLogLine2},
RequestParams: map[string][]string{},
Labels: model.LabelSet{
"job": "multiple_line_job",
},
},
expectedEntries: []expectedEntry{
{
labels: model.LabelSet{
"job": "multiple_line_job",
},
line: `[GIN] 2022/06/13 - 14:52:23 | 200 | 1.428101ms | 181.167.87.140 | GET "/"
`,
},
{
labels: model.LabelSet{
"job": "multiple_line_job",
},
line: `[GIN] 2022/06/13 - 14:52:23 | 200 | 163.92µs | 181.167.87.140 | GET "/static/main.css"
`,
},
},
},
"heroku request with two log lines and query parameters, internal labels dropped, and fixed are propagated": {
args: args{
RequestBodies: []string{testLogLine1, testLogLine2},
RequestParams: map[string][]string{
"some_query_param": []string{"app_123", "app_456"},
},
Labels: model.LabelSet{
"job": "multiple_line_job",
},
Expand All @@ -111,6 +171,7 @@ func TestHerokuDrainTarget(t *testing.T) {
"heroku request with a single log line, with internal labels relabeled, and fixed labels": {
args: args{
RequestBodies: []string{testLogLine1},
RequestParams: map[string][]string{},
Labels: model.LabelSet{
"job": "relabeling_job",
},
Expand Down Expand Up @@ -150,6 +211,59 @@ func TestHerokuDrainTarget(t *testing.T) {
},
},
},
"heroku request with a single log line and query parameters, with internal labels relabeled, and fixed labels": {
args: args{
RequestBodies: []string{testLogLine1},
RequestParams: map[string][]string{
"some_query_param": []string{"app_123", "app_456"},
},
Labels: model.LabelSet{
"job": "relabeling_job",
},
RelabelConfigs: []*relabel.Config{
{
SourceLabels: model.LabelNames{"__heroku_drain_host"},
TargetLabel: "host",
Replacement: "$1",
Action: relabel.Replace,
Regex: relabel.MustNewRegexp("(.*)"),
},
{
SourceLabels: model.LabelNames{"__heroku_drain_app"},
TargetLabel: "app",
Replacement: "$1",
Action: relabel.Replace,
Regex: relabel.MustNewRegexp("(.*)"),
},
{
SourceLabels: model.LabelNames{"__heroku_drain_proc"},
TargetLabel: "procID",
Replacement: "$1",
Action: relabel.Replace,
Regex: relabel.MustNewRegexp("(.*)"),
},
{
SourceLabels: model.LabelNames{"__heroku_drain_param_some_query_param"},
TargetLabel: "query_param",
Replacement: "$1",
Action: relabel.Replace,
Regex: relabel.MustNewRegexp("(.*)"),
},
},
},
expectedEntries: []expectedEntry{
{
line: `[GIN] 2022/06/13 - 14:52:23 | 200 | 1.428101ms | 181.167.87.140 | GET "/"
`,
labels: model.LabelSet{
"host": "host",
"app": "app",
"procID": "web.1",
"query_param": "app_123,app_456",
},
},
},
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
Expand Down Expand Up @@ -179,7 +293,7 @@ func TestHerokuDrainTarget(t *testing.T) {
// Send some logs
ts := time.Now()

req, err := makeDrainRequest(fmt.Sprintf("http://%s:%d", localhost, port), tc.args.RequestBodies...)
req, err := makeDrainRequest(fmt.Sprintf("http://%s:%d", localhost, port), tc.args.RequestParams, tc.args.RequestBodies...)
require.NoError(t, err, "expected test drain request to be successfully created")
res, err := http.DefaultClient.Do(req)
require.NoError(t, err)
Expand Down Expand Up @@ -237,7 +351,7 @@ func TestHerokuDrainTarget_UseIncomingTimestamp(t *testing.T) {
// Clear received lines after test case is ran
defer eh.Clear()

req, err := makeDrainRequest(fmt.Sprintf("http://%s:%d", localhost, port), testLogLine1)
req, err := makeDrainRequest(fmt.Sprintf("http://%s:%d", localhost, port), make(map[string][]string), testLogLine1)
require.NoError(t, err, "expected test drain request to be successfully created")
res, err := http.DefaultClient.Do(req)
require.NoError(t, err)
Expand Down Expand Up @@ -315,7 +429,7 @@ func TestHerokuDrainTarget_UseTenantIDHeaderIfPresent(t *testing.T) {
// Clear received lines after test case is ran
defer eh.Clear()

req, err := makeDrainRequest(fmt.Sprintf("http://%s:%d", localhost, port), testLogLine1)
req, err := makeDrainRequest(fmt.Sprintf("http://%s:%d", localhost, port), make(map[string][]string), testLogLine1)
require.NoError(t, err, "expected test drain request to be successfully created")
req.Header.Set("X-Scope-OrgID", "42")
res, err := http.DefaultClient.Do(req)
Expand Down
5 changes: 5 additions & 0 deletions docs/sources/clients/promtail/configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1311,6 +1311,11 @@ The Heroku Drain target exposes for each log entry the received syslog fields wi
- `__heroku_drain_proc`: The [PROCID](https://tools.ietf.org/html/rfc5424#section-6.2.6) field parsed from the message.
- `__heroku_drain_log_id`: The [MSGID](https://tools.ietf.org/html/rfc5424#section-6.2.7) field parsed from the message.

Additionally, the Heroku drain target will read all url query parameters from the
configured drain target url and make them available as
`__heroku_drain_param_<name>` labels, multiple instances of the same parameter
will appear as comma separated strings

### relabel_configs

Relabeling is a powerful tool to dynamically rewrite the label set of a target
Expand Down
21 changes: 20 additions & 1 deletion docs/sources/clients/promtail/scraping.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -446,7 +446,7 @@ Configuration is specified in a`heroku_drain` block within the Promtail `scrape_
- source_labels: ['__heroku_drain_host']
target_label: 'host'
- source_labels: ['__heroku_drain_app']
target_label: 'app'
target_label: 'source'
- source_labels: ['__heroku_drain_proc']
target_label: 'proc'
- source_labels: ['__heroku_drain_log_id']
Expand All @@ -466,6 +466,25 @@ with a command like the following:
heroku drains:add [http|https]://HOSTNAME:8080/heroku/api/v1/drain -a HEROKU_APP_NAME
```

### Getting the Heroku application name

Note that the `__heroku_drain_app` label will contain the source of the log line, either `app` or `heroku` and not the name of the heroku application.

The easiest way to provide the actual application name is to include a query parameter when creating the heroku drain and then relabel that parameter in your scraping config, for example:

```
heroku drains:add [http|https]://HOSTNAME:8080/heroku/api/v1/drain?app_name=HEROKU_APP_NAME -a HEROKU_APP_NAME
```

And then in a relabel_config:

```yaml
relabel_configs:
- source_labels: ['__heroku_drain_param_app_name']
target_label: 'app'
```

It also supports `relabeling` and `pipeline` stages just like other targets.

When Promtail receives Heroku Drain logs, various internal labels are made available for [relabeling](#relabeling):
Expand Down

0 comments on commit ea9ad33

Please sign in to comment.