-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cloud Foundry metrics receiver #1 - config, factory, docs #4626
Changes from 7 commits
ce06827
372b7fe
41e017e
bc4b7a7
0eed6dc
d744688
2780141
340bd01
22764b0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
include ../../Makefile.Common |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
# Cloud Foundry Receiver | ||
|
||
The Cloud Foundry receiver connects to the RLP (Reverse Log Proxy) Gateway of the Cloud Foundry installation, typically | ||
available at the URL `https://log-stream.<cf-system-domain>`. | ||
|
||
## Authentication | ||
|
||
RLP Gateway authentication is performed by adding the Oauth2 token as the `Authorization` header. To obtain an OAuth2 | ||
token to use for the RLP Gateway, a request is made to the UAA component which acts as the OAuth2 provider (URL | ||
specified by `uaa_url` configuration option, which typically is `https://uaa.<cf-system-domain>`). To authenticate with | ||
UAA, username and password/secret combination is used (`uaa_username` and `uaa_password` configuration options). This | ||
UAA user must have the `client_credentials` and `refresh_token` authorized grant types, and `logs.admin` authority. | ||
|
||
The following is an example sequence of commands to create the UAA user using the `uaac` command line utility: | ||
* `uaac target https://uaa.<cf-system-domain>` | ||
* `uaac token client get identity -s <identity-user-secret>` | ||
* `uaac client add <uaa_username> --name opentelemetry --secret <uaa_password> --authorized_grant_types client_credentials,refresh_token --authorities logs.admin` | ||
|
||
The `<uaa_username>` and `<uaa_password>` above can be set to anything as long as they match the values provided to the | ||
receiver configuration. The admin account (which is `identity` here) which has the permissions to create new clients may | ||
have a different name on different setups. The value of `--name` is not used for receiver configuration. | ||
|
||
## Configuration | ||
|
||
The receiver takes the following configuration options: | ||
* `rlp_gateway_url` - URL of the RLP gateway, required, typically `https://log-stream.<cf-system-domain>` | ||
* `rlp_gateway_skip_tls_verify` - whether to skip TLS verify for the RLP Gateway endpoint, default `false` | ||
* `rlp_gateway_shard_id` - metrics are load balanced among receivers that use the same shard ID, therefore this must | ||
only be set if there are multiple receivers which must both receive all the metrics instead of them being balanced | ||
between them. Default value is `opentelemetry` | ||
* `uaa_url` - URL of the UAA provider, required, typically `https://uaa.<cf-system-domain>` | ||
* `uaa_skip_tls_verify` - whether to skip TLS verify for the UAA endpoint, default `false` | ||
* `uaa_username` - name of the UAA user (required grant types/authorities described above) | ||
* `uaa_username` - password of the UAA user | ||
* `http_timeout` - HTTP socket timeout used for RLP Gateway, default `10s` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you format it to make it more readable? It should be easy to find if an option is required and what is its default value. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please resolve |
||
|
||
Example: | ||
|
||
```yaml | ||
receivers: | ||
cloudfoundry: | ||
rlp_gateway_url: "https://log-stream.sys.example.internal" | ||
rlp_gateway_skip_tls_verify: false | ||
rlp_gateway_shard_id: "opentelemetry" | ||
uaa_url: "https://uaa.sys.example.internal" | ||
uaa_skip_tls_verify: false | ||
uaa_username: "otelclient" | ||
uaa_password: "changeit" | ||
http_timeout: "20s" | ||
``` | ||
|
||
The full list of settings exposed for this receiver are documented [here](./config.go) | ||
with detailed sample configurations [here](./testdata/config.yaml). | ||
|
||
## Metrics | ||
|
||
Reported metrics are grouped under an instrumentation library named `otelcol/cloudfoundry`. Metric names are as | ||
specified by [Cloud Foundry metrics documentation](https://docs.cloudfoundry.org/running/all_metrics.html), but the | ||
origin name is prepended to the metric name with `.` separator. All metrics either of type `Gauge` or `Sum`. | ||
|
||
### Attributes | ||
|
||
All the metrics have the following attributes: | ||
* `origin` - origin name as documented by Cloud Foundry | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is origin both an attribute and also prepended to the metric name? (I don't know if this is desirable or no, just double checking that the docs are correct). |
||
* `source` - for applications, the GUID of the application, otherwise equal to `origin` | ||
|
||
For CF/TAS deployed in BOSH, the following attributes are also present, using their canonical BOSH meanings: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: it may be useful to link abbreviations to their definitions somewhere. I don't know what TAS or BOSH and being able to quickly find out can help maintaining the component in the future. |
||
* `deployment` - BOSH deployment name | ||
* `index` - BOSH instance ID (GUID) | ||
* `ip` - BOSH instance IP | ||
* `job` - BOSH job name | ||
|
||
For metrics originating with `rep` origin name (specific to applications), the following metrics are present: | ||
* `instance_id` - numerical index of the application instance. However, also present for `bbs` origin, where it matches | ||
the value of `index` | ||
* `process_id` - process ID (GUID). For a process of type "web" which is the main process of an application, this is | ||
equal to `source_id` and `app_id` | ||
* `process_instance_id` - unique ID of a process instance, should be treated as an opaque string | ||
* `process_type` - process type. Each application has exactly one process of type `web`, but many have any number of | ||
other processes | ||
|
||
On TAS/PCF versions 2.8.0+ and cf-deployment versions v11.1.0+, the following additional attributes are present for | ||
application metrics: `app_id`, `app_name`, `space_id`, `space_name`, `organization_id`, `organization_name` which | ||
provide the GUID and name of application, space and organization respectively. | ||
|
||
This might not be a comprehensive list of attributes, as the receiver passes on whatever attributes the gateway | ||
provides, which may include some that are specific to TAS and possibly new ones in future Cloud Foundry versions as | ||
well. |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,68 @@ | ||||||
// Copyright 2019, OpenTelemetry Authors | ||||||
// | ||||||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||||||
// you may not use this file except in compliance with the License. | ||||||
// You may obtain a copy of the License at | ||||||
// | ||||||
// http://www.apache.org/licenses/LICENSE-2.0 | ||||||
// | ||||||
// Unless required by applicable law or agreed to in writing, software | ||||||
// distributed under the License is distributed on an "AS IS" BASIS, | ||||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||||||
// See the License for the specific language governing permissions and | ||||||
// limitations under the License. | ||||||
|
||||||
package cloudfoundryreceiver | ||||||
|
||||||
import ( | ||||||
"fmt" | ||||||
"net/url" | ||||||
"time" | ||||||
|
||||||
"go.opentelemetry.io/collector/config" | ||||||
) | ||||||
|
||||||
// Config defines configuration for Collectd receiver. | ||||||
type Config struct { | ||||||
config.ReceiverSettings `mapstructure:",squash"` | ||||||
|
||||||
RLPGatewayURL string `mapstructure:"rlp_gateway_url"` | ||||||
RLPGatewaySkipTLSVerify bool `mapstructure:"rlp_gateway_skip_tls_verify"` | ||||||
RLPGatewayShardID string `mapstructure:"rlp_gateway_shard_id"` | ||||||
Comment on lines
+29
to
+31
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would it be nicer to make |
||||||
UAAUrl string `mapstructure:"uaa_url"` | ||||||
UAASkipTLSVerify bool `mapstructure:"uaa_skip_tls_verify"` | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider using the generic |
||||||
UAAUsername string `mapstructure:"uaa_username"` | ||||||
UAAPassword string `mapstructure:"uaa_password"` | ||||||
Comment on lines
+32
to
+35
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would it be nicer to make |
||||||
HTTPTimeout time.Duration `mapstructure:"http_timeout"` | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider using |
||||||
} | ||||||
|
||||||
func (c *Config) Validate() error { | ||||||
err := validateURLOption("rlp_gateway_url", c.RLPGatewayURL) | ||||||
if err != nil { | ||||||
return err | ||||||
} | ||||||
|
||||||
err = validateURLOption("uaa_url", c.UAAUrl) | ||||||
if err != nil { | ||||||
return err | ||||||
} | ||||||
|
||||||
if c.UAAUsername == "" { | ||||||
return fmt.Errorf("username not specified") | ||||||
} | ||||||
|
||||||
return nil | ||||||
} | ||||||
|
||||||
func validateURLOption(name string, value string) error { | ||||||
if value == "" { | ||||||
return fmt.Errorf("%s not specified", name) | ||||||
} | ||||||
|
||||||
_, err := url.Parse(value) | ||||||
if err != nil { | ||||||
return fmt.Errorf("failed to parse %s value %s as url: %v", name, value, err) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is no need to pass the value. See https://play.golang.org/p/XcW50z01QNf
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please resolve |
||||||
} | ||||||
|
||||||
return nil | ||||||
} |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,58 @@ | ||||||
// Copyright 2019, OpenTelemetry Authors | ||||||
// | ||||||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||||||
// you may not use this file except in compliance with the License. | ||||||
// You may obtain a copy of the License at | ||||||
// | ||||||
// http://www.apache.org/licenses/LICENSE-2.0 | ||||||
// | ||||||
// Unless required by applicable law or agreed to in writing, software | ||||||
// distributed under the License is distributed on an "AS IS" BASIS, | ||||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||||||
// See the License for the specific language governing permissions and | ||||||
// limitations under the License. | ||||||
|
||||||
package cloudfoundryreceiver | ||||||
|
||||||
import ( | ||||||
"path" | ||||||
"testing" | ||||||
"time" | ||||||
|
||||||
"github.com/stretchr/testify/assert" | ||||||
"github.com/stretchr/testify/require" | ||||||
"go.opentelemetry.io/collector/component/componenttest" | ||||||
"go.opentelemetry.io/collector/config" | ||||||
"go.opentelemetry.io/collector/config/configtest" | ||||||
) | ||||||
|
||||||
func TestLoadConfig(t *testing.T) { | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I suggest adding a TODO for the next PR: add negative tests e.g. in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please resolve |
||||||
factories, err := componenttest.NopFactories() | ||||||
assert.Nil(t, err) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please resolve |
||||||
|
||||||
factory := NewFactory() | ||||||
factories.Receivers[typeStr] = factory | ||||||
cfg, err := configtest.LoadConfigAndValidate(path.Join(".", "testdata", "config.yaml"), factories) | ||||||
|
||||||
require.NoError(t, err) | ||||||
require.NotNil(t, cfg) | ||||||
|
||||||
assert.Equal(t, len(cfg.Receivers), 2) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How about changing this line to:
Suggested change
so that it does not panic later if the length would be 0 or 1? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please resolve |
||||||
|
||||||
r0 := cfg.Receivers[config.NewID(typeStr)] | ||||||
assert.Equal(t, r0, factory.CreateDefaultConfig()) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Expected is first, actual is second.
Suggested change
See: https://pkg.go.dev/github.com/stretchr/testify@v1.7.0/assert#Equal There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please resolve |
||||||
|
||||||
r1 := cfg.Receivers[config.NewIDWithName(typeStr, "one")].(*Config) | ||||||
assert.Equal(t, r1, | ||||||
&Config{ | ||||||
ReceiverSettings: config.NewReceiverSettings(config.NewIDWithName(typeStr, "one")), | ||||||
RLPGatewayURL: "https://log-stream.sys.example.internal", | ||||||
RLPGatewaySkipTLSVerify: true, | ||||||
RLPGatewayShardID: "otel-test", | ||||||
UAAUrl: "https://uaa.sys.example.internal", | ||||||
UAASkipTLSVerify: true, | ||||||
UAAUsername: "admin", | ||||||
UAAPassword: "test", | ||||||
HTTPTimeout: time.Second * 20, | ||||||
}) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Expected is first, actual is second. See: https://pkg.go.dev/github.com/stretchr/testify@v1.7.0/assert#Equal There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please resolve |
||||||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
// Copyright 2019, OpenTelemetry Authors | ||
// | ||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
// Package cloudfoundryreceiver implements a receiver that can be used by the | ||
// Opentelemetry collector to receive Cloud Foundry metrics via its Reverse | ||
// Log Proxy (RLP) Gateway component. The protocol is handled by the | ||
// go-loggregator library, which uses HTTP to connect to the gateway and receive | ||
// JSON-protobuf encoded v2 Envelope messages as documented by loggregator-api. | ||
package cloudfoundryreceiver |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
// Copyright 2019, OpenTelemetry Authors | ||
// | ||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
package cloudfoundryreceiver | ||
|
||
import ( | ||
"context" | ||
|
||
"go.opentelemetry.io/collector/component" | ||
"go.opentelemetry.io/collector/config" | ||
"go.opentelemetry.io/collector/consumer" | ||
"go.opentelemetry.io/collector/receiver/receiverhelper" | ||
) | ||
|
||
// This file implements factory for Cloud Foundry receiver. | ||
|
||
const ( | ||
typeStr = "cloudfoundry" | ||
defaultUAAUsername = "admin" | ||
defaultRLPGatewayShardID = "opentelemetry" | ||
defaultURL = "https://localhost" | ||
) | ||
|
||
// NewFactory creates a factory for collectd receiver. | ||
func NewFactory() component.ReceiverFactory { | ||
return receiverhelper.NewFactory( | ||
typeStr, | ||
createDefaultConfig, | ||
receiverhelper.WithMetrics(createMetricsReceiver)) | ||
} | ||
|
||
func createDefaultConfig() config.Receiver { | ||
return &Config{ | ||
ReceiverSettings: config.NewReceiverSettings(config.NewID(typeStr)), | ||
RLPGatewayURL: defaultURL, | ||
RLPGatewaySkipTLSVerify: false, | ||
RLPGatewayShardID: defaultRLPGatewayShardID, | ||
UAAUsername: defaultUAAUsername, | ||
UAAPassword: "", | ||
UAAUrl: defaultURL, | ||
UAASkipTLSVerify: false, | ||
} | ||
} | ||
|
||
func createMetricsReceiver( | ||
_ context.Context, | ||
params component.ReceiverCreateSettings, | ||
cfg config.Receiver, | ||
nextConsumer consumer.Metrics, | ||
) (component.MetricsReceiver, error) { | ||
c := cfg.(*Config) | ||
return newCloudFoundryReceiver(params.Logger, *c, nextConsumer) | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
// Copyright 2019, OpenTelemetry Authors | ||
// | ||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
package cloudfoundryreceiver | ||
|
||
import ( | ||
"context" | ||
"testing" | ||
|
||
"github.com/stretchr/testify/assert" | ||
"go.opentelemetry.io/collector/component/componenttest" | ||
"go.opentelemetry.io/collector/config/configcheck" | ||
"go.opentelemetry.io/collector/consumer/consumertest" | ||
) | ||
|
||
func TestCreateDefaultConfig(t *testing.T) { | ||
factory := NewFactory() | ||
cfg := factory.CreateDefaultConfig() | ||
assert.NotNil(t, cfg, "failed to create default config") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should it not be There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please resolve |
||
assert.NoError(t, configcheck.ValidateConfig(cfg)) | ||
} | ||
|
||
func TestCreateReceiver(t *testing.T) { | ||
factory := NewFactory() | ||
cfg := factory.CreateDefaultConfig() | ||
|
||
params := componenttest.NewNopReceiverCreateSettings() | ||
tReceiver, err := factory.CreateMetricsReceiver(context.Background(), params, cfg, consumertest.NewNop()) | ||
assert.NoError(t, err) | ||
assert.NotNil(t, tReceiver, "receiver creation failed") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
markdownlint
wants to have a blank line between a paragraph and a listThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the same comment applies to other places in the
README.md