Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor add_cloud_metadata to handle ECS fields easier #26438

Merged
merged 6 commits into from
Jun 29, 2021
Merged

Refactor add_cloud_metadata to handle ECS fields easier #26438

merged 6 commits into from
Jun 29, 2021

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented Jun 23, 2021 •
edited
Loading

What does this PR do?

This PR refactors add_cloud_metadata processor as a follow-up of #26056 so as to provide a better handling for ECS fields which will be reported under a different namespace other than cloud.*. See more at #26337

Why is it important?

So as to provide a better experience for ECS fields handling by the add_cloud_metadata processor.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. Unit test should be fine: go test -v github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/...
  2. Test using the processor in an actual cloud env and verify events report fields properly.

Related issues

@ChrsMark
Refactor add_cloud_metadata to handle ECS fields easier
d187355 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark added the Team:Integrations Label for the Integrations team label Jun 23, 2021
@ChrsMark ChrsMark self-assigned this Jun 23, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 23, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 23, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26438 updated

  • Start Time: 2021-06-29T07:35:16.885+0000

  • Duration: 128 min 5 sec

  • Commit: 4e117f5

Test stats 🧪

Test Results
Failed 0
Passed 47893
Skipped 5347
Total 53240

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 47893
Skipped 5347
Total 53240

kaiyan-sheng
kaiyan-sheng approved these changes Jun 24, 2021
View reviewed changes
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me besides one naming issue :)

@ChrsMark ChrsMark mentioned this pull request Jun 28, 2021
Merged
jsoriano
jsoriano reviewed Jun 28, 2021
View reviewed changes
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not blocking this PR as I think that the refactor in addMeta would be enough to avoid the different code to handle overwrites. 👍

But in the fetcher I still see some special code to handle cloud and orchestrator fields, I wonder if this could be unified too.

libbeat/processors/add_cloud_metadata/provider_google_gce.go Outdated
Comment on lines 134 to 136
extractECSField("orchestrator", out, meta)

meta.DeepUpdate(common.MapStr{"cloud": out})
Copy link
Member

@jsoriano jsoriano Jun 28, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It still seems that some special handling is done for orchestrator vs cloud fields. And it may be a bit confusing to have an out, that is not an output of the function at the end.

Wdyt about setting the orchestrator values directly on meta, so they don't have to be moved from out to meta?

And in the same way, perhaps cloud fields could be directly set on meta["cloud"], so this intermediary out is not needed.

Copy link
Member Author

@ChrsMark ChrsMark Jun 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼 Good point! I tried to get rid of the extra "cooking" and applied the schema for orchestrator fields on a different map, since unfortunately Apply cannot work in a "DeepUpdate" way so it's not possible to write directly to cloud.* in multiple places since cloud* will be overridden each time Apply is called.

@ChrsMark
Merge remote-tracking branch 'upstream/master' into improve_cloud_met…
5310ef6 
…a_handling
@ChrsMark
Minor improvements
564d4d8 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Improve more
4e117f5 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark added the backport-v7.14.0 Automated backport with mergify label Jun 29, 2021
@ChrsMark ChrsMark merged commit f8bb3a2 into elastic:master Jun 29, 2021
mergify bot pushed a commit that referenced this pull request Jun 29, 2021
@ChrsMark
Refactor add_cloud_metadata to handle ECS fields easier (#26438)
244833d 
(cherry picked from commit f8bb3a2)
@mergify mergify bot mentioned this pull request Jun 29, 2021
Merged
v1v added a commit to v1v/beats that referenced this pull request Jun 29, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/support-d…
2c9f25c 
…arwin-arm64

* upstream/master: (295 commits)
  Update urllib to 1.26.5. (elastic#26380)
  Update golang.org/x/crypto (elastic#26448)
  [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816)
  Move parsers outside of filestream input so others can use them as well (elastic#26541)
  [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508)
  [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620)
  Logging code cleanup related to Nomad auto-discovery (elastic#26498)
  [Metricbeat] Add Couchbase's Sync Gateway module (elastic#25599)
  Refactor add_cloud_metadata to handle ECS fields easier (elastic#26438)
  [Elastic Agent] Improper casting of int64 (elastic#26520)
  [Elastic Agent] Enable configuring monitoring namespace (elastic#26439)
  [Heartbeat] configure permissions for synthetics config (elastic#26393)
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  ...
ChrsMark added a commit that referenced this pull request Jun 29, 2021
@mergify @ChrsMark
Refactor add_cloud_metadata to handle ECS fields easier (#26438) (#26556
6452107 
)

(cherry picked from commit f8bb3a2)

Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
@jsoriano jsoriano mentioned this pull request Aug 16, 2021
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsoriano jsoriano jsoriano left review comments

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Assignees

@ChrsMark ChrsMark

Labels
backport-v7.14.0 Automated backport with mergify Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Refactor add_cloud_metadata processor to improve ECS fields handling
4 participants
@ChrsMark @elasticmachine @jsoriano @kaiyan-sheng