Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] add collecting tags for rds metricset #16605

Merged
merged 18 commits into from
Mar 10, 2020
Merged

[Metricbeat] add collecting tags for rds metricset #16605

merged 18 commits into from
Mar 10, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Feb 26, 2020

What does this PR do?

This PR adds tag collection in rds metricset by default.

Why is it important?

Tags are important part of metadata for RDS instances. With this PR, users can collect metrics from RDS and use tags for filtering.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

Create RDS with tags first, then use config below to start aws module:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  metricsets:
    - rds

You should see tags collected in events, for example:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "aws": {
        "rds": {
            "aurora_bin_log_replica_lag": 0,
            "aurora_replica.lag_max.ms": 19.108999252319336,
            "aurora_replica.lag_min.ms": 19.108999252319336,
            "cache_hit_ratio.buffer": 100,
            "cache_hit_ratio.result_set": 0,
            "cpu": {
                "total": {
                    "pct": 0.04
                }
            },
            "database_connections": 0,
            "db_instance": {
                "arn": "arn:aws:rds:us-east-1:428152502467:db:database-1-instance-1",
                "class": "db.r5.large",
                "identifier": "database-1-instance-1",
                "status": "available"
            },
            "db_instance.identifier": "database-1-instance-1",
            "deadlocks": 0,
            "disk_usage": {
                "bin_log.bytes": 0
            },
            "engine_uptime.sec": 2704277,
            "free_local_storage.bytes": 31745863680,
            "freeable_memory.bytes": 4634234880,
            "latency": {
                "commit": 5.270933333333333,
                "ddl": 0,
                "delete": 0,
                "dml": 0.1624,
                "insert": 0.1624,
                "select": 0.17333862433862435,
                "update": 0
            },
            "login_failures": 0,
            "queries": 9.11833836203304,
            "throughput": {
                "commit": 0.5000916834753039,
                "ddl": 0,
                "delete": 0,
                "dml": 0.5000916834753039,
                "insert": 0.5000916834753039,
                "network": 1.4,
                "network_receive": 0.7,
                "network_transmit": 0.7,
                "select": 3.150577605894414,
                "update": 0
            },
            "transactions": {
                "active": 0,
                "blocked": 0
            }
        },
        "tags": {
            "created-by": "ks",
            "dept": "engr"
        }
    },
    "cloud": {
        "account": {
            "id": "428152502467",
            "name": "elastic-beats"
        },
        "availability_zone": "us-east-1b",
        "provider": "aws",
        "region": "us-east-1"
    },
    "event": {
        "dataset": "aws.rds",
        "duration": 115000,
        "module": "aws"
    },
    "metricset": {
        "name": "rds",
        "period": 10000
    },
    "service": {
        "type": "aws"
    }
}

Also test with tags_filter:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  tags_filter:
      - key: "dept"
        value: "eng"
  metricsets:
    - rds

Related issues

@kaiyan-sheng kaiyan-sheng self-assigned this Feb 26, 2020
@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review February 26, 2020 19:26
@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner February 26, 2020 19:26
@kaiyan-sheng kaiyan-sheng added enhancement needs_backport PR is waiting to be backported to other branches. review Team:Platforms Label for the Integrations - Platforms team labels Feb 26, 2020
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks good to me. I left just one small comment :)

@kaiyan-sheng kaiyan-sheng added the test-plan Add this PR to be manual test plan label Mar 10, 2020
@kaiyan-sheng kaiyan-sheng merged commit df3eb93 into elastic:master Mar 10, 2020
@kaiyan-sheng kaiyan-sheng deleted the rds_tags branch March 10, 2020 19:52
@kaiyan-sheng kaiyan-sheng added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 10, 2020
kaiyan-sheng added a commit that referenced this pull request Mar 11, 2020
* add ListTagsForResourceRequest for rds metricset

(cherry picked from commit df3eb93)
@andresrc andresrc added test-plan-added This PR has been added to the test plan and removed [zube]: Done labels Mar 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement review Team:Platforms Label for the Integrations - Platforms team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.7.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support to collects tags in RDS (AWS module)
3 participants