[receiver/splunkenterprise] 35445 add kvstore telemetry (#35657)

#### Description
Add telemetry around KV Store status to the Splunk Enterprise Receiver.

#### Link to tracking issue
Fixes
[35445](#35445)

#### Testing
Unit tests were updated and receiver component was deployed to test
environments.

#### Documentation
Documentation was updated to reflect additional metrics

.chloggen/35445-add-kvstore-telemetry.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: 'enhancement'
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: splunkenterprisereceiver
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Add telemetry around the Splunk Enterprise kv-store.
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [35445]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: [user]

receiver/splunkenterprisereceiver/documentation.md

@@ -404,6 +404,50 @@ Gauge tracking average bytes per second throughput of indexer. *Note:** Must be
 | ---- | ----------- | ------ |
 | splunk.indexer.status | The status message reported for a specific object | Any Str |
 
+### splunk.kvstore.backup.status
+
+Backup and restore status of the KV store.
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {status} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| splunk.kvstore.status.value | The string value of the status returned when reporting on KV store using the introspection endpoint | Any Str |
+
+### splunk.kvstore.replication.status
+
+Replication status of the KV store.
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {status} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| splunk.kvstore.status.value | The string value of the status returned when reporting on KV store using the introspection endpoint | Any Str |
+
+### splunk.kvstore.status
+
+This is the overall status of the kvstore for the given deployment.
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {status} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| splunk.kvstore.storage.engine | The backend storage used by the KV store. | Any Str |
+| splunk.kvstore.external | Value denoting if the KV store is using an external service. | Any Str |
+| splunk.kvstore.status.value | The string value of the status returned when reporting on KV store using the introspection endpoint | Any Str |
+
 ### splunk.server.introspection.queues.current
 
 Gauge tracking current length of queue. *Note:** Must be pointed at specific indexer `endpoint` and gathers metrics from only that indexer.