Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update docs for Remote Segment and Remote Translog store stats surfaced via nodes/indices/cluster stats APIs #4995

Merged
merged 7 commits into from
Sep 13, 2023
Merged

Update docs for Remote Segment and Remote Translog store stats surfaced via nodes/indices/cluster stats APIs #4995

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8471d24
Update docs for remote_store.moving_average_window_size
Sep 8, 2023
afad4e0
Update docs for Remote Segment and Remote Translog store stats surfac…
Sep 8, 2023
3fb096d
Apply suggestions from code review
Naarcha-AWS Sep 13, 2023
3ddba4f
Update _api-reference/nodes-apis/nodes-stats.md
Naarcha-AWS Sep 13, 2023
de043b3
Update nodes-stats.md
Naarcha-AWS Sep 13, 2023
c7dcd37
Update cluster-stats.md
Naarcha-AWS Sep 13, 2023
fff4b08
Apply suggestions from code review
Naarcha-AWS Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _api-reference/cluster-api/cluster-settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ The following request field parameters are compatible with the cluster API.
| cluster.max_shards_per_node | Integer | Limits the total number of primary and replica shards for the cluster. The limit is calculated as follows: `cluster.max_shards_per_node` multiplied by the number of non-frozen data nodes. Shards for closed indexes do not count toward this limit. Default is `1000`. |
| cluster.persistent_tasks.allocation.enable | String | Enables or disables allocation for persistent tasks: <br /> <br /> `all` – Allows persistent tasks to be assigned to nodes. <br /> <br /> `none` – No allocations are allowed for persistent tasks. This does not affect persistent tasks already running. <br /> <br /> Default is `all`. |
| cluster.persistent_tasks.allocation.recheck_interval | Time unit | The cluster manager automatically checks whether or not persistent tasks need to be assigned when the cluster state changes in a significant way. There are other factors, such as memory usage, that will affect whether or not persistent tasks are assigned to nodes but do not otherwise cause the cluster state to change. This setting defines how often assignment checks are performed in response to these factors. Default is `30 seconds`, with a minimum of `10 seconds` being required. |
| remote_store.moving_average_window_size | Integer | The moving average window size used to calculate the rolling statistic values exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. Minimum enforced is `5`. |
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

#### Example request

Expand Down
23 changes: 23 additions & 0 deletions _api-reference/cluster-api/cluster-stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,29 @@ Parameter | Type | Description
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 1112,
"max_unsafe_auto_id_timestamp": 1644269449096,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"mappings": {
Expand Down
156 changes: 152 additions & 4 deletions _api-reference/index-apis/stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,14 +199,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -326,14 +363,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -457,14 +531,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -584,14 +695,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down
66 changes: 65 additions & 1 deletion _api-reference/nodes-apis/nodes-stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,14 +245,51 @@ Select the arrow to view the example response.
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 288,
"max_unsafe_auto_id_timestamp" : -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes" : { }
},
"translog" : {
"operations" : 12,
"size_in_bytes" : 1452,
"uncommitted_operations" : 12,
"uncommitted_size_in_bytes" : 1452,
"earliest_last_modified_age" : 164160
"earliest_last_modified_age" : 164160,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache" : {
"memory_size_in_bytes" : 1649,
Expand Down Expand Up @@ -792,13 +829,40 @@ segments.index_writer_memory_in_bytes | Integer | The total amount of memory use
segments.version_map_memory_in_bytes | Integer | The total amount of memory used by all version maps, in bytes.
segments.fixed_bit_set_memory_in_bytes | Integer | The total amount of memory used by fixed bit sets, in bytes. Fixed bit sets are used for nested objects and join fields.
segments.max_unsafe_auto_id_timestamp | Integer | The timestamp for the most recently retired indexing request, in milliseconds since the epoch.
segments.remote_store | Object | Statistics about remote segment store operations.
segments.remote_store.upload | Object | Statistics about Remote Segment Store upload operations.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_upload_size | Object | The amount of data in bytes uploaded to the remote segment store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_upload_size.started_bytes | Integer | The number of bytes for which to upload to the remote segment store after the upload has started.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_upload_size.succeeded_bytes | Integer | The number of bytes successfully uploaded to the remote segment store.
segments.remote_store.upload.total_upload_size.failed_bytes | Integer | The number of bytes that failed to upload to the remote segment store.
segments.remote_store.upload.refresh_size_lag | Object | The amount of lag during upload between the remote segment store and the local store.
segments.remote_store.upload.refresh_size_lag.total_bytes | Integer | The total bytes that lagged during the upload refresh between the remote segment store and the local store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.refresh_size_lag.max_bytes | Integer | The maximum lag in bytes during the upload refresh between the remote segment store and the local store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.max_refresh_time_lag_in_millis | Integer | The maximum duration in milliseconds the remote refresh is behind the local refresh.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_time_spent_in_millis | Integer | The total time in milliseconds spent on uploads to the remote segment store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download | Object | Statistics related to download operation to the remote segment store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download.total_download_size | Object | The amount of data downloaded from the remote segment store.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The total amount"?

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download.total_download_size.started_bytes | Integer | The number of bytes downloaded from the remote segment store after the download starts.
segments.remote_store.download.total_download_size.succeeded_bytes | Integer | The number of bytes successfully downloaded from the remote segment store.
segments.remote_store.download.total_download_size.failed_bytes | Integer | The number of bytes that failed to down from the remote segment store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download.total_time_spent_in_millis | Integer | The total duration in milliseconds spent on downloads from the remote segment store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.file_sizes | Integer | Statistics about the size of the segment files.
translog | Object | Statistics about transaction log operations for the node.
translog.operations | Integer | The number of translog operations.
translog.size_in_bytes | Integer | The size of the translog, in bytes.
translog.uncommitted_operations | Integer | The number of uncommitted translog operations.
translog.uncommitted_size_in_bytes | Integer | The size of uncommitted translog operations, in bytes.
translog.earliest_last_modified_age | Integer | The earliest last modified age for the translog.
translog.remote_store | Object | Statistics related to operations from the remote translog store.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Statistics related to remote translog store operations"?

translog.remote_store.upload | Object | Statistics about upload operation to the remote translog store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_uploads | Object | The number of syncs to the remote translog store.
translog.remote_store.upload.total_uploads.started | Integer | The number of upload syncs to the remote translog store that have started.
translog.remote_store.upload.total_uploads.failed | Integer | The number of failed upload syncs to the remote translog store.
translog.remote_store.upload.total_uploads.succeeded | Integer | The number of successful upload syncs to the remote translog store.
translog.remote_store.upload.total_upload_size | Object | The amount of data uploaded to the remote translog store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_upload_size.started_bytes | Integer | The number of bytes actively uploading to the remote translog store after the upload has started.
translog.remote_store.upload.total_upload_size.failed_bytes | Integer | The number of bytes that failed to upload to the remote translog store.
translog.remote_store.upload.total_upload_size.succeeded_bytes | Integer | The number of bytes successfully uploaded to the remote translog store.
request_cache | Object | Statistics about the request cache for the node.
request_cache.memory_size_in_bytes | Integer | The memory size used by the request cache, in bytes.
request_cache.evictions | Integer | The number of request cache evictions.
Expand Down
Loading