Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Synthtrace] Support Non-ECS Logs #191086

Merged

Conversation

mohamedhamed-ahmed
Copy link
Contributor

@mohamedhamed-ahmed mohamedhamed-ahmed commented Aug 22, 2024

closes 3759

📝 Summary

This PR creates a new scenario with different non ECS fields as below :

  • log.level-> severity
  • message -> msg
  • service.name -> svc
  • host.name -> hostname
  • New field: thisisaverylongfieldnamethatevendoesnotcontainanyspaceswhyitcouldpotentiallybreakouruiinseveralplaces

The above fields are applied with different variances as below :

  • In DSNS data stream with @timestamp
  • Outside of DSNS with @timestamp (e.g. cloud-logs-*, etc.)
  • Outside of DSNS without @timestamp (replaced by “date”)

🎥 Demo

node scripts/synthtrace simple_non_ecs_logs.ts

nonEcs.mov

@mohamedhamed-ahmed mohamedhamed-ahmed added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting Team:obs-ux-logs Observability Logs User Experience Team labels Aug 22, 2024
@obltmachine
Copy link

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@mohamedhamed-ahmed
Copy link
Contributor Author

/ci

@mohamedhamed-ahmed mohamedhamed-ahmed marked this pull request as ready for review August 22, 2024 15:28
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

@botelastic botelastic bot added ci:project-deploy-observability Create an Observability project Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Aug 23, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

Copy link
Contributor

@cauemarcondes cauemarcondes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -53,6 +53,17 @@ export class SynthtraceEsClient<TFields extends Fields> {
)}"`
);

const resolvedIndices = this.indices.length
? (
await this.client.indices.resolveIndex({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for that, I was doing the same thing on another PR that hasn't been merged yet 😆

await this.client.indices.resolveIndex({
name: this.indices.join(','),
expand_wildcards: ['open', 'hidden'],
// @ts-expect-error ignore_unavailable is not in the type definition, but it is accepted by es
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: If you want you can create a PR to fix this type issue like i did once upon a time - elastic/elasticsearch-specification#2653

Copy link
Contributor

@achyutjhunjhunwala achyutjhunjhunwala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kibana-ci
Copy link
Collaborator

kibana-ci commented Sep 3, 2024

💛 Build succeeded, but was flaky

  • Buildkite Build
  • Commit: fbb0255
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-191086-fbb025531b45

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #15 / When rendering PolicySettingsLayout and user has Edit permissions should show success toast on update success

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/apm-synthtrace 53 56 +3
@kbn/apm-synthtrace-client 204 205 +1
total +4
Unknown metric groups

API count

id before after diff
@kbn/apm-synthtrace 53 56 +3
@kbn/apm-synthtrace-client 204 205 +1
total +4

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@mohamedhamed-ahmed mohamedhamed-ahmed enabled auto-merge (squash) September 3, 2024 13:53
@mohamedhamed-ahmed mohamedhamed-ahmed merged commit 913b8f3 into elastic:main Sep 3, 2024
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
apm:review backport:skip This commit does not require backporting ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team Team:obs-ux-logs Observability Logs User Experience Team v8.16.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants