Aws rds Integration support (#741) (#841)

* update the vpc flow log readme with a dashboard screenshot and update the sourceUrl to point to the correct place * add support for observability on aws rds logs - using the pre-canned dashboards * update rds integration * add dashboard.jpeg screenshot * append minor changes * append minor changes * fix sq-ip field * fix sq-ip field * update dashboard image * rearrange dashboard * update according pr comments * update schema mapping files including cloud mapping * update vpc flow log containers log name * update index template to avoid name collisions * update according to name collision component strategy * remove none relevant fixes --------- (cherry picked from commit 2aedbde) Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-project · Aug 8, 2023 · 353fe8a · 353fe8a
1 parent 07f69f1
commit 353fe8a
Show file tree

Hide file tree

Showing 13 changed files with 1,209 additions and 2 deletions.
diff --git a/opensearch_dashboards.json b/opensearch_dashboards.json
@@ -18,4 +18,4 @@
     "urlForwarding",
     "visualizations"
   ]
-}
+}
diff --git a/server/adaptors/integrations/__data__/repository/aws_rds/assets/README.md b/server/adaptors/integrations/__data__/repository/aws_rds/assets/README.md
@@ -0,0 +1,36 @@
+# AWS RDS Integration Assets
+
+API: http://osd:5601/api/saved_objects/_import?overwrite=true
+
+- [Assets](aws_rds-1.0.0.ndjson)
+
+## Asset List
+The next table details the assets
+
+| Name                                        |     Type      |                                                Description                                                 |
+|---------------------------------------------|:-------------:|:----------------------------------------------------------------------------------------------------------:|
+| `ss4o_logs_rds-*-*`                         | index-pattern |                                             The Index Pattern                                              |
+| `AWS RDS Log Event Overview`                |   dashboard   |                                    The pre-canned dashboard for AWS RDS                                    |
+| `[AWS RDS] Filters`                         | visualization |                      [Controls] Interactive controls for easy dashboard manipulation                       |
+| `[AWS RDS] Slowquery`                       | visualization |  [Metric] Shows the latest slow queries that took more time than specified in the slow query parameters.   |
+| `[AWS RDS] Slow Query History`              | visualization |         [Vertical Bar] Presents a timeline representation of slow queries over a specified period.         |
+| `[AWS RDS] Average Slow Query Time History` | visualization |              [Pie] : Provides a historical trend of average execution times for slow queries.              |
+| `[AWS RDS] Total Slow Queries`              | visualization |              [Metric] Depicts the total count of slow queries within a specified time frame.               |
+| `[AWS RDS] Top Slow Query IP Table`         | visualization |                     [Line] Lists the IP addresses that initiated the most slow queries                     |
+| `[AWS RDS] Slow Query Scatter Plot`         | visualization | [Line] A scatter plot illustrating slow queries against two different parameters such as time and duration |
+| `[AWS RDS] Average Slow Query Duration`     | visualization |                   [Metric] Represents the average time taken by slow queries to execute                    |
+| `[AWS RDS] Slow Query Pie`                  | visualization |                         [Pie] A pie chart showing the distribution of slow queries                         |
+| `[AWS RDS] Slow Query Table Pie`            | visualization |                        [Table] A pie chart showing the distribution of slow queries                        |
+| `[AWS RDS] Top Slow Query`                  | visualization |                             [Table] Top 10 source showing the slowest queries                              |
+| `[AWS RDS] Lock`                            | visualization |              [Table] A visualization showing the number of active locks in your RDS instance               |
+| `[AWS RDS] Total Deadlock Queries`          | visualization |            [Table] Represents the total count of deadlock scenarios encountered in the database            |
+| `[AWS RDS] Deadlock History`                | visualization |                 [Table]      Provides a timeline showing occurrences of deadlock scenarios                 |
+| `[AWS RDS] Error Data`                      | visualization |              [Table]  Represents data related to various errors occurred in your RDS instance              |
+| `[AWS RDS] Audit Data`                      | visualization |             [Table] overview of audit logs, showing actions that have been tracked for review              |
+| `[AWS RDS] Total Error Logs`                | visualization |            [Line] Displays the total count of error logs recorded within a specific time frame             |
+| `[AWS RDS] Error History`                   | visualization |          [Line]  Provides a timeline representation of the errors occurred over a certain period.          |
+| `[AWS RDS] Audoit History`                  | visualization |      [Line]  Provides a timeline representation of the audited events occurred over a certain period.      |
+| `[AWS RDS] General Search`                  |    search     |                                     The pre-canned search for AWS RDS                                      |
+
+## Dashboard
+![](../static/dashboard.png)
diff --git a/server/adaptors/integrations/__data__/repository/aws_rds/assets/aws_rds-1.0.0.ndjson b/server/adaptors/integrations/__data__/repository/aws_rds/assets/aws_rds-1.0.0.ndjson
diff --git a/server/adaptors/integrations/__data__/repository/aws_rds/aws_rds-1.0.0.json b/server/adaptors/integrations/__data__/repository/aws_rds/aws_rds-1.0.0.json
@@ -0,0 +1,49 @@
+{
+  "name": "aws_rds",
+  "version": "1.0.0",
+  "displayName": "AWS RDS",
+  "description": "AWS RDS",
+  "license": "Apache-2.0",
+  "type": "logs_rds",
+  "author": "OpenSearch",
+  "sourceUrl": "https://github.com/opensearch-project/dashboards-observability/tree/main/server/adaptors/integrations/__data__/repository/aws_rds/info",
+  "statics": {
+    "logo": {
+      "annotation": "AWS RDS Logo",
+      "path": "aws-rds-icon.png"
+    },
+    "gallery": [
+      {
+        "annotation": "AWS RDS Dashboard",
+        "path": "dashboard.png"
+      }
+    ]
+  },
+  "components": [
+    {
+      "name": "aws_rds",
+      "version": "1.0.0"
+    },
+    {
+      "name": "cloud",
+      "version": "1.0.0"
+    },
+    {
+      "name": "logs_rds",
+      "version": "1.0.0"
+    },
+    {
+      "name": "aws_s3",
+      "version": "1.0.0"
+    }
+  ],
+  "assets": {
+    "savedObjects": {
+      "name": "aws_rds",
+      "version": "1.0.0"
+    }
+  },
+  "sampleData": {
+    "path": "sample.json"
+  }
+}
diff --git a/server/adaptors/integrations/__data__/repository/aws_rds/data/sample.json b/server/adaptors/integrations/__data__/repository/aws_rds/data/sample.json
@@ -0,0 +1,272 @@
+[
+  {
+    "@timestamp": "2023-07-17T08:14:05.000Z",
+    "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK",
+    "event": {
+      "result": "ACCEPT",
+      "name": "rds_log",
+      "domain": "aws.rds"
+    },
+    "attributes": {
+      "data_stream": {
+        "dataset": "aws.rds",
+        "namespace": "production",
+        "type": "logs"
+      }
+    },
+    "aws": {
+      "s3": {
+        "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2",
+        "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz"
+      },
+      "rds": {
+        "db-identifier": "db1",
+        "sq-user": "admin",
+        "sq-db-name": "testDB",
+        "sq-host-name": "host1",
+        "sq-ip": "192.0.2.0",
+        "sq-id": "sq123",
+        "sq-duration": 1.23,
+        "sq-lock-wait": 0.12,
+        "sq-rows-sent": 100,
+        "sq-rows-examined": 500,
+        "sq-table-name": "table1",
+        "sq-timestamp": "2023-07-22T11:22:33Z",
+        "sq-query": "SELECT * FROM table1",
+        "err-thread": "thread1",
+        "err-label": "error1",
+        "err-code": "err123",
+        "err-sub-system": "sub1",
+        "err-detail": "Error detail message",
+        "general-id": "gen123",
+        "general-action": "insert",
+        "general-query": "INSERT INTO table1 VALUES ('value1', 'value2')",
+        "audit-ip": "192.0.2.1",
+        "audit-user": "auditUser",
+        "audit-host-name": "host2",
+        "audit-connection-id": "conn123",
+        "audit-query-id": "query123",
+        "audit-operation": "SELECT",
+        "audit-db-name": "auditDB",
+        "audit-query": "SELECT * FROM auditDB",
+        "audit-retcode": "ret123",
+        "deadlock-thread-id-1": "thread2",
+        "deadlock-os-thread-handle-1": "osThread1",
+        "deadlock-query-id-1": "dq1",
+        "deadlock-ip-1": "192.0.2.2",
+        "deadlock-user-1": "user2",
+        "deadlock-action-1": "select",
+        "deadlock-query-1": "SELECT * FROM table2 WHERE column1 = 'value3'",
+        "deadlock-thread-id-2": "thread3",
+        "deadlock-os-thread-handle-2": "osThread2",
+        "deadlock-query-id-2": "dq2",
+        "deadlock-ip-2": "192.0.2.3",
+        "deadlock-user-2": "user3",
+        "deadlock-action-2": "update",
+        "deadlock-query-2": "UPDATE table2 SET column1 = 'value4' WHERE column1 = 'value3'",
+        "log-detail": "Log detail message"
+      }
+    },
+    "cloud": {
+      "provider": "aws",
+      "account": {
+        "id": "111111111111"
+      },
+      "region": "ap-southeast-2",
+      "resource_id": "vpc-0d4d4e82b7d743527",
+      "platform": "aws_rds"
+    },
+    "communication": {
+      "source": {
+        "address": "162.142.125.177",
+        "port": 38471,
+        "packets": 1,
+        "bytes": 44
+      },
+      "destination": {
+        "address": "10.0.0.200",
+        "port": 12313
+      }
+    }
+  },
+  {
+    "@timestamp": "2023-07-18T09:15:06.000Z",
+    "body": "3 111111111112 eni-0e250409d410e1291 162.142.125.178 10.0.0.201 38472 12314 6 2 45 1674898497 1674898508 ACCEPT OK",
+    "event": {
+      "result": "ACCEPT",
+      "name": "rds_log",
+      "domain": "aws.rds"
+    },
+    "attributes": {
+      "data_stream": {
+        "dataset": "aws.rds",
+        "namespace": "production",
+        "type": "logs"
+      }
+    },
+    "aws": {
+      "s3": {
+        "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c3",
+        "key": "AWSLogs/111111111112/vpcflowlogs/us-east-2/2023/01/28/111111111112_vpcflowlogs_us-east-2_fl-023c6afa025ee5a05_20230128T0940Z_3a9dfd9e.log.gz"
+      },
+      "rds": {
+        "db-identifier": "db2",
+        "sq-user": "admin2",
+        "sq-db-name": "testDB2",
+        "sq-host-name": "host2",
+        "sq-ip": "192.0.2.1",
+        "sq-id": "sq124",
+        "sq-duration": 1.24,
+        "sq-lock-wait": 0.13,
+        "sq-rows-sent": 101,
+        "sq-rows-examined": 501,
+        "sq-table-name": "table2",
+        "sq-timestamp": "2023-07-22T12:23:34Z",
+        "sq-query": "SELECT * FROM table2",
+        "err-thread": "thread2",
+        "err-label": "error2",
+        "err-code": "err124",
+        "err-sub-system": "sub2",
+        "err-detail": "Error detail message 2",
+        "general-id": "gen124",
+        "general-action": "update",
+        "general-query": "UPDATE table2 SET column1 = 'value3', column2 = 'value4'",
+        "audit-ip": "192.0.2.2",
+        "audit-user": "auditUser2",
+        "audit-host-name": "host3",
+        "audit-connection-id": "conn124",
+        "audit-query-id": "query124",
+        "audit-operation": "UPDATE",
+        "audit-db-name": "auditDB2",
+        "audit-query": "UPDATE auditDB2 SET column1 = 'value5'",
+        "audit-retcode": "ret124",
+        "deadlock-thread-id-1": "thread3",
+        "deadlock-os-thread-handle-1": "osThread2",
+        "deadlock-query-id-1": "dq2",
+        "deadlock-ip-1": "192.0.2.3",
+        "deadlock-user-1": "user3",
+        "deadlock-action-1": "update",
+        "deadlock-query-1": "UPDATE table3 SET column1 = 'value6' WHERE column1 = 'value5'",
+        "deadlock-thread-id-2": "thread4",
+        "deadlock-os-thread-handle-2": "osThread3",
+        "deadlock-query-id-2": "dq3",
+        "deadlock-ip-2": "192.0.2.4",
+        "deadlock-user-2": "user4",
+        "deadlock-action-2": "insert",
+        "deadlock-query-2": "INSERT INTO table4 VALUES ('value7', 'value8')",
+        "log-detail": "Log detail message 2"
+      }
+    },
+    "cloud": {
+      "provider": "aws",
+      "account": {
+        "id": "111111111112"
+      },
+      "region": "ap-southeast-2",
+      "resource_id": "vpc-0d4d4e82b7d743528",
+      "platform": "aws_rds"
+    },
+    "communication": {
+      "source": {
+        "address": "162.142.125.178",
+        "port": 38472,
+        "packets": 2,
+        "bytes": 45
+      },
+      "destination": {
+        "address": "10.0.0.201",
+        "port": 12314
+      }
+    }
+  },
+  {
+    "@timestamp": "2023-07-19T10:16:07.000Z",
+    "body": "4 111111111113 eni-0e250409d410e1292 162.142.125.179 10.0.0.202 38473 12315 6 3 46 1674898498 1674898509 ACCEPT OK",
+    "event": {
+      "result": "ACCEPT",
+      "name": "rds_log",
+      "domain": "aws.rds"
+    },
+    "attributes": {
+      "data_stream": {
+        "dataset": "aws.rds",
+        "namespace": "production",
+        "type": "logs"
+      }
+    },
+    "aws": {
+      "s3": {
+        "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c4",
+        "key": "AWSLogs/111111111113/vpcflowlogs/us-east-2/2023/01/28/111111111113_vpcflowlogs_us-east-2_fl-023c6afa025ee5a06_20230128T0950Z_3a9dfd9f.log.gz"
+      },
+      "rds": {
+        "db-identifier": "db3",
+        "sq-user": "admin3",
+        "sq-db-name": "testDB3",
+        "sq-host-name": "host3",
+        "sq-ip": "192.0.2.2",
+        "sq-id": "sq125",
+        "sq-duration": 1.25,
+        "sq-lock-wait": 0.14,
+        "sq-rows-sent": 102,
+        "sq-rows-examined": 502,
+        "sq-table-name": "table3",
+        "sq-timestamp": "2023-07-22T13:24:35Z",
+        "sq-query": "SELECT * FROM table3",
+        "err-thread": "thread3",
+        "err-label": "error3",
+        "err-code": "err125",
+        "err-sub-system": "sub3",
+        "err-detail": "Error detail message 3",
+        "general-id": "gen125",
+        "general-action": "delete",
+        "general-query": "DELETE FROM table3 WHERE column1 = 'value9'",
+        "audit-ip": "192.0.2.3",
+        "audit-user": "auditUser3",
+        "audit-host-name": "host4",
+        "audit-connection-id": "conn125",
+        "audit-query-id": "query125",
+        "audit-operation": "DELETE",
+        "audit-db-name": "auditDB3",
+        "audit-query": "DELETE FROM auditDB3 WHERE column1 = 'value10'",
+        "audit-retcode": "ret125",
+        "deadlock-thread-id-1": "thread4",
+        "deadlock-os-thread-handle-1": "osThread3",
+        "deadlock-query-id-1": "dq3",
+        "deadlock-ip-1": "192.0.2.4",
+        "deadlock-user-1": "user4",
+        "deadlock-action-1": "delete",
+        "deadlock-query-1": "DELETE FROM table4 WHERE column1 = 'value11'",
+        "deadlock-thread-id-2": "thread5",
+        "deadlock-os-thread-handle-2": "osThread4",
+        "deadlock-query-id-2": "dq4",
+        "deadlock-ip-2": "192.0.2.5",
+        "deadlock-user-2": "user5",
+        "deadlock-action-2": "select",
+        "deadlock-query-2": "SELECT * FROM table5",
+        "log-detail": "Log detail message 3"
+      }
+    },
+    "cloud": {
+      "provider": "aws",
+      "account": {
+        "id": "111111111113"
+      },
+      "region": "ap-southeast-2",
+      "resource_id": "vpc-0d4d4e82b7d743529",
+      "platform": "aws_rds"
+    },
+    "communication": {
+      "source": {
+        "address": "162.142.125.179",
+        "port": 38473,
+        "packets": 3,
+        "bytes": 46
+      },
+      "destination": {
+        "address": "10.0.0.202",
+        "port": 12315
+      }
+    }
+  }
+]
diff --git a/server/adaptors/integrations/__data__/repository/aws_rds/info/README.md b/server/adaptors/integrations/__data__/repository/aws_rds/info/README.md
@@ -0,0 +1,26 @@
+# AWS RDS Integrations
+
+## What is AWS RDS?
+AWS RDS (Relational Database Service) is a managed service that makes it easy to set up, operate, and scale a relational database in the cloud.
+
+RDS helps you perform tasks such as:
+
+* Managing database instances
+* Scaling compute resources and storage capacity
+* Automating time-consuming administration tasks including hardware provisioning, database setup, patching, and backups
+
+RDS keeps your database up-to-date with the latest patches, and it also provides automatic backups and disaster recovery capabilities. You can make database instances available in multiple regions to enhance availability and reliability for your data.
+
+See additional details [here](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.html).
+
+## What is AWS RDS Integration?
+An integration is a collection of predefined assets which are combined in a meaningful manner.
+
+AWS RDS integration includes dashboards, visualizations, queries, and index mapping to help you manage and monitor your database services more effectively.
+
+### Dashboards
+The Dashboard uses the index alias `logs-aws-rds` for shortening the index name - be advised.
+
+![](../static/dashboard_rds1.png)
+
+This integration provides you with a comprehensive view of your RDS instances, enabling you to monitor performance and resources effectively, troubleshoot problems quickly, and make data-driven decisions.