Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pkg/stanza] container: remove time field from attributes after parsing #33389

Open
ChrsMark opened this issue Jun 5, 2024 · 4 comments
Open
Assignees
Labels
enhancement New feature or request never stale Issues marked with this label will be never staled and automatically removed pkg/stanza

Comments

@ChrsMark
Copy link
Member

ChrsMark commented Jun 5, 2024

Component(s)

pkg/stanza

Is your feature request related to a problem? Please describe.

This issue keeps track of what was suggested at #33353 (comment).

At the moment when the container parser parses the logs, it parses the time field moving the result to the Timestamp.
However the original time field is preserved in the Attributes. Example:

2024-06-04T06:40:08.219Z	info	ResourceLog #0
Resource SchemaURL: 
Resource attributes:
     -> k8s.pod.uid: Str(d5ecc924-e255-4525-b5be-6437939b1e4d)
     -> k8s.container.name: Str(busybox)
     -> k8s.namespace.name: Str(default)
     -> k8s.pod.name: Str(daemonset-logs-dhzcq)
     -> k8s.container.restart_count: Str(0)
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2024-06-04 06:40:08.007370503 +0000 UTC
Timestamp: 2024-06-04 06:40:07.855932421 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(otel logs at 06:40:07)
Attributes:
     -> logtag: Str(F)
     -> log.file.path: Str(/var/log/pods/default_daemonset-logs-dhzcq_d5ecc924-e255-4525-b5be-6437939b1e4d/busybox/0.log)
     -> time: Str(2024-06-04T06:40:07.855932421Z)
     -> log.iostream: Str(stdout)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #1
ObservedTimestamp: 2024-06-04 06:40:08.007451031 +0000 UTC
Timestamp: 2024-06-04 06:40:07.957875321 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(otel logs at 06:40:07)
Attributes:
     -> log.file.path: Str(/var/log/pods/default_daemonset-logs-dhzcq_d5ecc924-e255-4525-b5be-6437939b1e4d/busybox/0.log)
     -> log.iostream: Str(stdout)
     -> time: Str(2024-06-04T06:40:07.957875321Z)
     -> logtag: Str(F)
Trace ID: 
Span ID: 
Flags: 0

Describe the solution you'd like

As mentioned at #33353 (comment), it be would reasonable to remove the original field once its parsed.

One thing to consider here is the possibility of this field being used by some users today. This could happen for users that use the existing Helm preset.
Since the container parser will be a replacement for this preset we need to take this into account.
Maybe we can do this removal through a setting/feature-flag to not aggressively break any users that might use the original time field.

Describe alternatives you've considered

No response

Additional context

No response

@ChrsMark ChrsMark added enhancement New feature or request needs triage New item requiring triage labels Jun 5, 2024
Copy link
Contributor

github-actions bot commented Jun 5, 2024

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@djaglowski djaglowski removed the needs triage New item requiring triage label Jun 5, 2024
andrzej-stencel added a commit that referenced this issue Jul 10, 2024
…logs (#33946)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
This PR adds support for removing the original `time` attribute/field
from the parsed log record as it was suggested at
#33389.
Users should use the `Timestamp` field which holds the parsed time.

This patch introduces this change behind a feature flag called
`filelog.container.removeOriginalTimeField` which is disabled by default
following the [feature
lifecycle](https://github.com/open-telemetry/opentelemetry-collector/tree/main/featuregate#feature-lifecycle).

**Link to tracking Issue:** <Issue number if applicable>
#33389

**Testing:** <Describe what testing was performed and which tests were
added.>

1. Added unit-test
2. Run with `./bin/otelcontribcol_linux_amd64 --config
container_config.yaml
--feature-gates=filelog.container.removeOriginalTimeField
` and verify the output:

```console
2024-07-08T14:26:50.936+0300	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 2}
2024-07-08T14:26:50.936+0300	info	ResourceLog #0
Resource SchemaURL: 
Resource attributes:
     -> k8s.namespace.name: Str(some)
     -> k8s.pod.name: Str(kube-controller-kind-control-plane)
     -> k8s.container.restart_count: Str(1)
     -> k8s.pod.uid: Str(49cc7c1fd3702c40b2686ea7486091d6)
     -> k8s.container.name: Str(kube-controller)
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2024-07-08 11:26:50.836517638 +0000 UTC
Timestamp: 2029-03-30 08:31:20.545192187 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(INFO: log line here)
Attributes:
     -> logtag: Str(F)
     -> log.file.path: Str(/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log)
     -> log.iostream: Str(stdout)
Trace ID: 
Span ID: 
Flags: 0
```

**Documentation:** <Describe the documentation added.> Added.

---------

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Co-authored-by: Andrzej Stencel <andrzej.stencel@elastic.co>
@ChrsMark
Copy link
Member Author

Since #33946 was merged with the feature gate, let's keep this issue open to track the graduation of this feature gate.
I will take care of it.

Copy link
Contributor

github-actions bot commented Sep 9, 2024

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Sep 9, 2024
@ChrsMark ChrsMark removed the Stale label Sep 9, 2024
djaglowski pushed a commit that referenced this issue Sep 13, 2024
…ate to beta (#35169)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

This PR moves the feature gate to delete the original time field from
parsed container logs to beta as part of
#33389.

**Link to tracking Issue:** <Issue number if applicable>

**Testing:** <Describe what testing was performed and which tests were
added.>

**Documentation:** <Describe the documentation added.>

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
jriguera pushed a commit to springernature/opentelemetry-collector-contrib that referenced this issue Oct 4, 2024
…ate to beta (open-telemetry#35169)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

This PR moves the feature gate to delete the original time field from
parsed container logs to beta as part of
open-telemetry#33389.

**Link to tracking Issue:** <Issue number if applicable>

**Testing:** <Describe what testing was performed and which tests were
added.>

**Documentation:** <Describe the documentation added.>

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Nov 11, 2024
@ChrsMark ChrsMark added never stale Issues marked with this label will be never staled and automatically removed and removed Stale labels Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request never stale Issues marked with this label will be never staled and automatically removed pkg/stanza
Projects
None yet
Development

No branches or pull requests

2 participants