Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

receiver/azureblobreceiver First PR #11872

Merged
merged 32 commits into from
Aug 11, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
4a9659d
blob receiver
eedorenko Mar 3, 2022
57e44a6
blobclient
eedorenko Mar 4, 2022
2aa1998
working receiver
eedorenko Mar 4, 2022
4c0aa91
traces receiver
eedorenko Mar 5, 2022
b511219
two receivers
eedorenko Mar 5, 2022
a21d8c9
blob
eedorenko Mar 7, 2022
bbef295
sync
eedorenko Mar 24, 2022
6a3ae74
linting
eedorenko Mar 24, 2022
5599fad
sync
eedorenko Mar 24, 2022
b17653d
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
eedorenko Mar 24, 2022
aa93eee
linting
eedorenko Mar 24, 2022
2fa2b96
linting
eedorenko Mar 24, 2022
32eda09
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
eedorenko Apr 11, 2022
f69a5fb
Merge branch 'main' into eedorenko/azureblobreceiver-1
eedorenko Jul 1, 2022
40fa787
readme comments
eedorenko Aug 8, 2022
c7b2559
factory comment
eedorenko Aug 8, 2022
5d5cda7
Merge branch 'eedorenko/azureblobreceiver-1' of https://github.com/ee…
eedorenko Aug 8, 2022
f85b246
comments
eedorenko Aug 8, 2022
6e773ce
linting
eedorenko Aug 8, 2022
b03f0fa
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
eedorenko Aug 8, 2022
a5eb45e
bump up to 1.18
eedorenko Aug 8, 2022
d6c5645
readme
eedorenko Aug 8, 2022
cf671aa
config
eedorenko Aug 8, 2022
0744f67
comments
eedorenko Aug 9, 2022
2ff5d50
private factory method
eedorenko Aug 9, 2022
a020e55
end_point -> endpoint
eedorenko Aug 9, 2022
187748e
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
eedorenko Aug 9, 2022
11c3d49
cleaning
eedorenko Aug 10, 2022
c9e84f3
codeowners
eedorenko Aug 10, 2022
afdfdd8
versions
eedorenko Aug 10, 2022
dc4f4e2
Update .github/CODEOWNERS
eedorenko Aug 10, 2022
929156d
Merge branch 'eedorenko/azureblobreceiver-1' of https://github.com/ee…
eedorenko Aug 10, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ receiver/awscontainerinsightreceiver/ @open-telemetry/collector-c
receiver/awsecscontainermetricsreceiver/ @open-telemetry/collector-contrib-approvers @Aneurysm9
receiver/awsfirehosereceiver/ @open-telemetry/collector-contrib-approvers @Aneurysm9
receiver/awsxrayreceiver/ @open-telemetry/collector-contrib-approvers @willarmiros
receiver/azureblobreceiver/ @open-telemetry/collector-contrib-approvers @eedorenko @mx-psi
receiver/bigipreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski @StefanKurek
receiver/carbonreceiver/ @open-telemetry/collector-contrib-approvers @pjanotti
receiver/chronyreceiver/ @open-telemetry/collector-contrib-approvers @MovieStoreGuy @jamesmoessis
Expand Down
1 change: 1 addition & 0 deletions receiver/azureblobreceiver/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include ../../Makefile.Common
41 changes: 41 additions & 0 deletions receiver/azureblobreceiver/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Azure Blob Receiver

| Status | |
| ------------------------ | ----------- |
| Stability |[development]|
| Supported pipeline types | logs,traces |
| Distributions | [contrib] |


This receiver reads logs and trace data from [Azure Blob Storage](https://azure.microsoft.com/services/storage/blobs/).

## Configuration

The following settings are required:

- `connection_string:` (no default): Azure Blob Storage connection key, which can be found in the Azure Blob Storage resource on the Azure Portal.
- `event_hub:`
` endpoint:` (no default): Azure Event Hub endpoint triggering on the `Blob Create` event

The following settings can be optionally configured:

- `logs:`
` container_name:` (default = "logs"): Name of the blob container with the logs
- `traces:`
` container_name:` (default = "traces"): Name of the blob container with the traces

Example:

```yaml
receivers:
azureblob:
connection_string: DefaultEndpointsProtocol=https;AccountName=accountName;AccountKey=+idLkHYcL0MUWIKYHm2j4Q==;EndpointSuffix=core.windows.net
event_hub:
endpoint: Endpoint=sb://oteldata.servicebus.windows.net/;SharedAccessKeyName=otelhubbpollicy;SharedAccessKey=mPJVubIK5dJ6mLfZo1ucsdkLysLSQ6N7kddvsIcmoEs=;EntityPath=otellhub
```

The receiver subscribes [on the events](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview) published by Azure Blob Storage and handled by Azure Event Hub. When it receives `Blob Create` event, it reads the logs or traces from a corresponding blob and deletes it after processing.




39 changes: 39 additions & 0 deletions receiver/azureblobreceiver/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
// Copyright OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package azureblobreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/azureblobreceiver"

import (
"go.opentelemetry.io/collector/config"
)

type Config struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you open a follow up PR to document the fields directly in the Go struct? This is not urgent, but will be useful in the future for automatic configuration schema generation

config.ReceiverSettings `mapstructure:",squash"`
ConnectionString string `mapstructure:"connection_string"`
EventHub EventHubConfig `mapstructure:"event_hub"`
Logs LogsConfig `mapstructure:"logs"`
Traces TracesConfig `mapstructure:"traces"`
}

type EventHubConfig struct {
EndPoint string `mapstructure:"endpoint"`
}

type LogsConfig struct {
ContainerName string `mapstructure:"container_name"`
}

type TracesConfig struct {
ContainerName string `mapstructure:"container_name"`
}
121 changes: 121 additions & 0 deletions receiver/azureblobreceiver/factory.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
// Copyright OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package azureblobreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/azureblobreceiver"

import (
"context"
"errors"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config"
"go.opentelemetry.io/collector/consumer"

"github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent"
)

const (
// The value of "type" key in configuration.
typeStr = "azureblob"
logsContainerName = "logs"
tracesContainerName = "traces"
)

var (
errUnexpectedConfigurationType = errors.New("failed to cast configuration to Azure Blob Config")
)

type blobReceiverFactory struct {
receivers *sharedcomponent.SharedComponents
}

// NewFactory returns a factory for Azure Blob receiver.
func NewFactory() component.ReceiverFactory {
f := &blobReceiverFactory{
receivers: sharedcomponent.NewSharedComponents(),
}

return component.NewReceiverFactory(
typeStr,
f.createDefaultConfig,
component.WithTracesReceiver(f.createTracesReceiver, component.StabilityLevelBeta),
component.WithLogsReceiver(f.createLogsReceiver, component.StabilityLevelBeta))
}

func (f *blobReceiverFactory) createDefaultConfig() config.Receiver {
return &Config{
ReceiverSettings: config.NewReceiverSettings(config.NewComponentID(typeStr)),
Logs: LogsConfig{ContainerName: logsContainerName},
Traces: TracesConfig{ContainerName: tracesContainerName},
}
}

func (f *blobReceiverFactory) createLogsReceiver(
ctx context.Context,
set component.ReceiverCreateSettings,
cfg config.Receiver,
nextConsumer consumer.Logs,
) (component.LogsReceiver, error) {

receiver, err := f.getReceiver(set, cfg)

if err != nil {
set.Logger.Error(err.Error())
return nil, err
}

return receiver, nil
}

func (f *blobReceiverFactory) createTracesReceiver(
ctx context.Context,
set component.ReceiverCreateSettings,
cfg config.Receiver,
nextConsumer consumer.Traces,
) (component.TracesReceiver, error) {

receiver, err := f.getReceiver(set, cfg)

if err != nil {
set.Logger.Error(err.Error())
return nil, err
}

return receiver, nil
}

func (f *blobReceiverFactory) getReceiver(
mx-psi marked this conversation as resolved.
Show resolved Hide resolved
set component.ReceiverCreateSettings,
cfg config.Receiver) (component.Receiver, error) {

var err error
r := f.receivers.GetOrAdd(cfg, func() component.Component {
receiverConfig, ok := cfg.(*Config)

if !ok {
err = errUnexpectedConfigurationType
return nil
}

var receiver component.Receiver
receiver, err = newReceiver(*receiverConfig, set)
return receiver
})

if err != nil {
return nil, err
}

return r.Unwrap(), err
}
33 changes: 33 additions & 0 deletions receiver/azureblobreceiver/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
module github.com/open-telemetry/opentelemetry-collector-contrib/receiver/azureblobreceiver

go 1.18

require (
github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent v0.57.2
go.opentelemetry.io/collector v0.57.2
)

require (
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/knadh/koanf v1.4.2 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
go.opentelemetry.io/collector/pdata v0.57.2 // indirect
go.opentelemetry.io/otel v1.8.0 // indirect
go.opentelemetry.io/otel/metric v0.31.0 // indirect
go.opentelemetry.io/otel/trace v1.8.0 // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
go.uber.org/zap v1.21.0 // indirect
golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4 // indirect
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
golang.org/x/text v0.3.7 // indirect
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa // indirect
google.golang.org/grpc v1.48.0 // indirect
google.golang.org/protobuf v1.28.1 // indirect
)
Loading