Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GCP] Add cloudsql data streams #4126

Merged
merged 29 commits into from
Jul 20, 2023
Merged

[GCP] Add cloudsql data streams #4126

merged 29 commits into from
Jul 20, 2023

Conversation

gpop63
Copy link
Contributor

@gpop63 gpop63 commented Sep 5, 2022 •
edited
Loading

What does this PR do?

Adds CloudSQL data streams:

  • MySQL
  • PostgreSQL
  • SQLServer

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@elasticmachine
Copy link

elasticmachine commented Sep 5, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-07-20T22:02:17.948+0000

  • Duration: 18 min 46 sec

Test stats 🧪

Test Results
Failed 0
Passed 64
Skipped 0
Total 64

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Sep 5, 2022 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (6/6) 💚
Files 100.0% (6/6) 💚
Classes 100.0% (6/6) 💚
Methods 87.826% (101/115) 👎 -12.174
Lines 96.0% (1464/1525) 👍 9.169
Conditionals 100.0% (0/0) 💚

@gpop63 gpop63 mentioned this pull request Sep 14, 2022
Merged
6 tasks
@gpop63 gpop63 marked this pull request as ready for review September 14, 2022 11:03
@gpop63 gpop63 requested review from a team as code owners September 14, 2022 11:03
@gpop63
Copy link
Contributor Author

gpop63 commented Sep 16, 2022

/test

@elasticmachine
Copy link

elasticmachine commented Sep 16, 2022 •
edited
Loading

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@endorama endorama added Integration:gcp Google Cloud Platform Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Sep 23, 2022
@endorama endorama mentioned this pull request Sep 23, 2022
Closed
8 tasks
@botelastic
Copy link

botelastic bot commented Nov 7, 2022

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Nov 7, 2022
@endorama
Copy link
Member

@gpop63 does this PR requires elastic/beats#33066?

@botelastic botelastic bot removed the Stalled label Nov 14, 2022
@gpop63
Copy link
Contributor Author

gpop63 commented Nov 14, 2022

Yes, we should first merge the CloudSQL Metadata PR (elastic/beats#33066)) and then this one, because we need the metadata to identify which database type generated events.

@endorama
Copy link
Member

elastic/beats#33066 has been merged, so we can move forward with this once 8.6.0 is out.

@botelastic
Copy link

botelastic bot commented Jan 14, 2023

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Jan 14, 2023
@botelastic
Copy link

botelastic bot commented Feb 13, 2023

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

@botelastic botelastic bot closed this Feb 13, 2023
@endorama endorama reopened this Feb 13, 2023
@botelastic botelastic bot removed the Stalled label Feb 13, 2023
@botelastic
Copy link

botelastic bot commented Mar 15, 2023

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Mar 15, 2023
@botelastic botelastic bot removed the Stalled label Mar 23, 2023
@botelastic
Copy link

botelastic bot commented Jun 23, 2023

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Jun 23, 2023
@botelastic botelastic bot removed the Stalled label Jul 17, 2023
endorama
endorama approved these changes Jul 19, 2023
View reviewed changes
Copy link
Member

@endorama endorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a minor comment but looks good!

packages/gcp/data_stream/cloudsql_mysql/elasticsearch/ingest_pipeline/default.yml
processors:
- drop:
description: Drop if database is not MySQL.
if: "ctx?.gcp?.labels?.cloudsql?.name != 'mysql'"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I think will be useful in the future is to document how we built this check. Is not very clear where this label come from, if it can change or not and where to check in case this pipeline stops working.

Please document this in the cloudsql.md file for future reference.

packages/gcp/data_stream/cloudsql_postgresql/elasticsearch/ingest_pipeline/default.yml
processors:
- drop:
description: Drop if database is not PostgreSQL.
if: "ctx?.gcp?.labels?.cloudsql?.name != 'postgres'"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

packages/gcp/data_stream/cloudsql_sqlserver/elasticsearch/ingest_pipeline/default.yml
processors:
- drop:
description: Drop if database is not SQLServer.
if: "ctx?.gcp?.labels?.cloudsql?.name != 'sqlserver'"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

@gpop63 gpop63 merged commit 27e77c6 into elastic:main Jul 20, 2023
@elasticmachine
Copy link

Package gcp - 2.24.0 containing this change is available at https://epr.elastic.co/search?package=gcp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@endorama endorama endorama approved these changes

Assignees

@gpop63 gpop63

Labels
Integration:gcp Google Cloud Platform Team:Cloud-Monitoring Label for the Cloud Monitoring team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

GCP CloudSQL integration
3 participants
@gpop63 @elasticmachine @endorama