Incorrect Fields Leveraged for AWS RDS Dashboard

[TristanMa]

I believe that the OOTB AWS RDS Dashboard is currently levering the below incorrect fields for it's visualizations.

aws.rds.db_instance.identifier is currently being used as a breakdown to identify the different RDS instances but our OOTB AWS integration does not pull in this field. I believe the correct field should be aws.dimensions.DBInstanceIdentifier as it holds the same value as defined in our Integration docs and values are populated via our OOTB AWS Integration.

Affected Visualizations:

1. Free Storae
2. Read Latency
3. Write Latency
4. Insert Throughput
5. Select Throughput
6. CPU Total PCT
7. Disk Queueu Depth
8. Write IOPS
9. Read IOPS

aws.rds.write_io.ops_per_sec is being used for the WriteOps visualization but this value is not populated by the OOTB AWS integration. I believe the correct field should be aws.rds.write.iops as it returns the expected values

aws.rds.read_io.ops_per_sec is being used for the WriteOps visualization but this value is not populated by the OOTB AWS integration. I believe the correct field should be aws.rds.read.iops as it returns the expected values

[endorama]

Hi @TristanMa, reviewing the fields that the RDS integration should output aws.rds.db_instance.identifier should be present.

See https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/rds/fields/fields.yml#L53-L56

I'm working on reproducing the 3 issues you mention.

[endorama]

I backtracked the changes adding dimension fields to this PR #204. The intention for that PR was to fix the dashboards adding dimensions fields but I'm not able to confirm if the fix has been applied to all the needed dashboards.

I also confirmed that aws.rds.write_io.ops_per_sec and aws.rds.read_io.ops_per_sec are not populated, while aws.rds.write.iops and aws.rds.read.iops are populated from the event in the data stream ingest pipeline.

[aspacca]

@TristanMa what version of the agent did you use when hit the bug?

aws.rds.db_instance.identifier is currently being used as a breakdown to identify the different RDS instances but our OOTB AWS integration does not pull in this field

this is a problem of the underlaying metricbeat version, and it come from a bug introduced in 8.6.0 and fixed here

the same bug produced different separated events for each of the metrics (so for example, one with aws.rds.write.iops and one with aws.rds.read.iops) instead of merging all metrics in a single event

could you try to reproduce with 8.5 and 8.7bc1?

thanks

[TristanMa]

@aspacca I can confirm that the AWS integration for RDS works as intended for the AWS version 1.28 running on an 8.4 cluster.

Let me try it out on an 8.7 build

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!