SC-20508 #2

ghost · 2024-10-07T10:27:51Z

This PR adds metrics support for Sematext exporter

If a user enters the wrong region/ app_token it will throw an error

… with a different error. For some reason the recorded request body is nil

…xt format

otisg · 2024-10-09T06:27:25Z

@Eromosele-SM "This branch has conflicts that must be resolved"

exporter/sematextexporter/config.go

cmd/otelcontribcol/go.mod

exporter/sematextexporter/Makefile

exporter/sematextexporter/config.go

exporter/sematextexporter/writer.go

exporter/sematextexporter/writer_test.go

otisg · 2024-10-09T12:12:32Z

FWIW I saw 2 instances of the following in your merge of main into your fork:
c64a75e [chore] [exporter/sumologic] Use NewDefaultClientConfig instead of manually creating struct (open-telemetry#35547)

(this one for SL, and another for Mezmo)
Maybe something new to look into/use?

ghost · 2024-10-09T13:40:44Z

FWIW I saw 2 instances of the following in your merge of main into your fork: c64a75e [chore] [exporter/sumologic] Use NewDefaultClientConfig instead of manually creating struct (open-telemetry#35547)

(this one for SL, and another for Mezmo) Maybe something new to look into/use?

Their createDefaultClientConfig() function returns a confighttp.ClientConfig but what our createDefaultClientConfig() function returns is a component.Config and there is no default for it

exporter/sematextexporter/writer.go

jackgopack4

I don't see any major issues here, some small suggestions. Make sure you get some pipelines/checks set up (you may be able to piggyback on upstream CI if you open a draft PR on open-telemetry/opentelemetry-collector-contrib

jackgopack4 · 2024-10-11T19:13:43Z

exporter/sematextexporter/writer_test.go

no specific comments on any of your tests, but it might be useful to open up a draft pr on the contrib repo to benefit from their pipelines they have set up for linting, test coverage, etc.

Yeah i will, once i am done with the logs support

exporter/sematextexporter/writer_test.go

exporter/sematextexporter/writer.go

exporter/sematextexporter/factory.go

…sent

otisg · 2024-10-25T08:02:44Z

exporter/sematextexporter/README.md

+* `endpoint` (required) HTTP/S destination for metric receivers. It is dependent on the region:
+    - US: `spm-receiver.sematext.com`
+    - EU: `spm-receiver.eu.sematext.com`
+* `timeout` (default = 5s) Timeout for requests


@Eromosele-SM make it match STA timeout?

exporter/sematextexporter/README.md

otisg · 2024-10-25T08:04:21Z

exporter/sematextexporter/README.md

+  * `US`
+  * `EU`
+* `payload_max_lines` (default = 10_000) Maximum number of lines allowed per HTTP POST request
+* `payload_max_bytes` (default = 10_000_000) Maximum number of bytes allowed per HTTP POST request


@Eromosele-SM I'd check these defaults internally with Sematext first.

Is this it?
https://sematext.com/docs/monitoring/custom-metrics/#:~:text=We%20recommend%20sending%20batches%20of%20between%20100%20and%20500%20lines%20of%20metrics.

That's the suggest for custom metrics. I'd ask STA devs what STA uses.

otisg · 2024-10-25T08:05:32Z

exporter/sematextexporter/README.md

+* `payload_max_bytes` (default = 10_000_000) Maximum number of bytes allowed per HTTP POST request
+* `metrics_schema` (default = telegraf-prometheus-v2) The chosen metrics schema to write; must be one of:
+    * `otel-v1`
+    * `telegraf-prometheus-v2`


@Eromosele-SM probably only one schema in the end. Also, this Exporter will end up shipping telemetry other than metrics. So maybe you want to structure the README in preparation for that. For example, this particular schema will be only for metrics.

I can always edit the readme when we have logs support ready?

Ofc. I hope the YAML is also structured in anticipation of Exporter sending other telemetry later (so other App tokens, other endpoints, although the region could probably remain general).

otisg

A few suggestions made...

ghost · 2024-10-30T22:48:56Z

@boratanrikulu @otisg
Can you please take a look?
I made some changes

otisg

Just did a quick skip over the surface stuff, not actual code.

exporter/sematextexporter/README.md

exporter/sematextexporter/config.go

otisg · 2024-10-30T23:02:13Z

exporter/sematextexporter/config.go

+
+// Validate checks for invalid or missing entries in the configuration.
+func (cfg *Config) Validate() error {
+	if strings.ToLower(cfg.Region) != "eu" && strings.ToLower(cfg.Region) != "us" && strings.ToLower(cfg.Region) != "custom"{


Still not sure what "custom" is for. Esp. since in the line below you say only US and EU are allowed.

The custom is so that we can run tests on the exporter with a mockserver

#### Description Generates simple histograms using telemetrygen  #### Link to tracking issue Fixes  #### Testing Test with a local otel collector with debug output ``` bin/telemetrygen metrics --metrics 5 --otlp-http --otlp-endpoint "localhost:4318" --metric-type Histogram --otlp-insecure ``` Output from debug Exporter: ``` Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.633365 +0000 UTC Timestamp: 2024-11-13 16:22:51.633367 +0000 UTC Count: 0 Sum: 0.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #1 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.639942 +0000 UTC Timestamp: 2024-11-13 16:22:51.639942 +0000 UTC Count: 1 Sum: 1.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 1 Buckets #2, Count: 0 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #2 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.6404 +0000 UTC Timestamp: 2024-11-13 16:22:51.640401 +0000 UTC Count: 2 Sum: 4.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 1 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 ResourceMetrics #3 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.640729 +0000 UTC Timestamp: 2024-11-13 16:22:51.640729 +0000 UTC Count: 3 Sum: 3.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 1 Buckets #1, Count: 1 Buckets #2, Count: 1 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #4 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.641073 +0000 UTC Timestamp: 2024-11-13 16:22:51.641073 +0000 UTC Count: 4 Sum: 12.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 1 Buckets #3, Count: 2 Buckets #4, Count: 1 {"kind": "exporter", "data_type": "metrics", "name": "debug"} ```  #### Documentation  --------- Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>

#### Description Vulnerability #1: GO-2025-3420 Sensitive headers incorrectly sent after cross-domain redirect in net/http More info: https://pkg.go.dev/vuln/GO-2025-3420 Standard library Found in: net/http@go1.22.8 Fixed in: net/http@go1.22.11 Example traces found: Error: #1: codeowners.go:212:55: githubgen.codeownersGenerator.getGithubMembers calls github.OrganizationsService.ListMembers, which eventually calls http.Client.Do Vulnerability #2: GO-[20](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/13003223509/job/36265594395?pr=37492#step:6:21)25-3373 Usage of IPv6 zone IDs can bypass URI name constraints in crypto/x509 More info: https://pkg.go.dev/vuln/GO-2025-3373 Standard library Found in: crypto/x509@go1.[22](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/13003223509/job/36265594395?pr=37492#step:6:23).8 Fixed in: crypto/x509@go1.22.11 Example traces found: Related: open-telemetry/opentelemetry-collector#12197

AkhigbeEromo added 12 commits October 4, 2024 00:01

Worked on metrics support

7be09e3

Fixed all otel-related issues

56075f9

Add region to sematextexporter config

6cad40b

Rewrite test for config

8ed1a5a

Add comments for the region

af5ba97

Fix linter issues

6d2eff4

Vaidate app_token and region

929070f

If a user enters the wrong region/ app_token it will throw an error

Add sematext specifics for the metrics line

f958e8f

SC-20508 #comment Fix the timestamp issue but now the test is failing…

abf2930

… with a different error. For some reason the recorded request body is nil

Fix the issue with the test to ensure metrics line are sent in semate…

bf6e81b

…xt format

Fix lint issues

9d02ad7

Delete logging for test

2146176

ghost marked this pull request as ready for review October 8, 2024 21:56

boratanrikulu requested changes Oct 9, 2024

View reviewed changes

ghost force-pushed the add-sematext-exporter branch from bf683ff to 2146176 Compare October 9, 2024 11:35

boratanrikulu and others added 4 commits October 9, 2024 14:41

Merge remote-tracking branch 'origin' into add-sematext-exporter

6af2ce6

remove go.mod files

602ecbf

fix new-line issue

0290804

Update App_token to AppToken

b4788ae

AkhigbeEromo added 2 commits October 9, 2024 13:42

Clean up dirty code

649215f

Remove "influx" occurences

2e17dd5

Remove unneccessary comments

e358f65

jackgopack4 reviewed Oct 11, 2024

View reviewed changes

exporter/sematextexporter/writer.go Show resolved Hide resolved

jackgopack4 reviewed Oct 11, 2024

View reviewed changes

exporter/sematextexporter/writer.go Outdated Show resolved Hide resolved

jackgopack4 reviewed Oct 11, 2024

View reviewed changes

exporter/sematextexporter/factory.go Outdated Show resolved Hide resolved

ghost changed the title ~~Add sematext exporter~~ SC:20508 Add sematext exporter Oct 14, 2024

AkhigbeEromo and others added 5 commits October 24, 2024 00:54

Fix test because hostname changed

a8346ed

Reduce tags to 20 and make sure os.host and token tags are always pre…

02fec37

…sent

Make os.host dynamic in test

a85429a

Add README file

9317219

Change endpoint to metrics receiver endpoint

1f32a87

otisg reviewed Oct 25, 2024

View reviewed changes

exporter/sematextexporter/README.md Show resolved Hide resolved

otisg reviewed Oct 25, 2024

View reviewed changes

otisg approved these changes Oct 25, 2024

View reviewed changes

AkhigbeEromo added 8 commits October 25, 2024 12:11

Region is now used to determine the endpoint

8d81a7d

Edit README.md

245725c

Update README.md

eeb42af

Refactor Exporter so that endpoints can be seperate

5d17fd7

Refactor Exporter

b4505b0

Correct a comment

98d51f4

Update README.md

1c87f19

Change defaults of payload

fcd0f90

ghost requested review from otisg, jackgopack4 and boratanrikulu October 30, 2024 22:49

otisg reviewed Oct 30, 2024

View reviewed changes

AkhigbeEromo added 2 commits November 5, 2024 09:10

Fix indentation issues

40581e7

Change go.mod version to 1.22.0

6495e65

AkhigbeEromo changed the title ~~SC-20508 Add sematext exporter~~ SC-20508 Dec 18, 2024

AkhigbeEromo closed this Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SC-20508 #2

SC-20508 #2

ghost commented Oct 7, 2024 •

edited by ghost

Loading

otisg commented Oct 9, 2024

otisg commented Oct 9, 2024

ghost commented Oct 9, 2024

jackgopack4 left a comment

jackgopack4 Oct 11, 2024

ghost Oct 15, 2024

otisg Oct 25, 2024

otisg Oct 25, 2024

ghost Oct 25, 2024

otisg Oct 25, 2024

otisg Oct 25, 2024

ghost Oct 25, 2024

otisg Oct 25, 2024

otisg left a comment

ghost commented Oct 30, 2024

otisg left a comment

otisg Oct 30, 2024

ghost Nov 5, 2024

SC-20508 #2

SC-20508 #2

Conversation

ghost commented Oct 7, 2024 • edited by ghost Loading

otisg commented Oct 9, 2024

otisg commented Oct 9, 2024

ghost commented Oct 9, 2024

jackgopack4 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

otisg left a comment

Choose a reason for hiding this comment

ghost commented Oct 30, 2024

otisg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Oct 7, 2024 •

edited by ghost

Loading