Adds Point in Time documentation #1753
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,270 @@ | ||||||
| --- | ||||||
| layout: default | ||||||
| title: Point in Time API | ||||||
| nav_order: 58 | ||||||
| has_children: false | ||||||
| parent: Point in Time | ||||||
| --- | ||||||
|
|
||||||
| # Point in Time API | ||||||
|
|
||||||
| Use the [Point in Time (PIT)]({{site.url}}{{site.baseurl}}/opensearch/point-in-time/) API to manage Points in Time. | ||||||
|
There was a problem hiding this comment. If we are defining Point in Time as PIT, we should define it on first appearance and then use the acronym thereafter. Please apply to all files. There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| #### Table of contents | ||||||
| - TOC | ||||||
| {:toc} | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Create a PIT | ||||||
|
There was a problem hiding this comment. Can we also add info about these cluster settings.
|
||||||
| Introduced 2.4 | ||||||
| {: .label .label-purple } | ||||||
|
|
||||||
| Creates a PIT. The `keep_alive` query parameter is required; it specifies how long to keep this PIT. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| ### Path and HTTP methods | ||||||
|
|
||||||
| ```json | ||||||
| POST /<target_indexes>/_search/point_in_time?keep_alive=1h&routing=&expand_wildcards=&preference= | ||||||
| ``` | ||||||
|
|
||||||
| ### Path parameters | ||||||
|
|
||||||
| Parameter | Data Type | Description | ||||||
| :--- | :--- | :--- | ||||||
| target_indexes | String | The name(s) of the target index(es) for the PIT. May contain a comma-separated list or a wildcard index pattern. | ||||||
|
|
||||||
| ### Query parameters | ||||||
|
|
||||||
| Parameter | Data Type | Description | ||||||
|
There was a problem hiding this comment. Should we have a column to specify which param is options or required @bharath-techie @dhruv16dhr ? There was a problem hiding this comment. Yeah this could be useful. There was a problem hiding this comment. We are adhering to our API style guide, which does not have a "required/optional" column. We list the required/optional in the description. But if you feel that it's needed, I can add the column. There was a problem hiding this comment. Lets go ahead with the API style guide as we have been following till now |
||||||
| :--- | :--- | :--- | ||||||
| keep_alive | Time | The amount of time to keep the PIT. Required. | ||||||
| preference | String | The node or the shard used to perform the search. Optional. Default is random. | ||||||
|
There was a problem hiding this comment. PIT do not support preference. There was a problem hiding this comment. This is for create PIT, which does support preference. please add it back @kolchfa-aws |
||||||
| routing | String | Specifies to route search requests to a specific shard. Optional. Default is the document's `_id`. | ||||||
| expand_wildcards | String | The type of index that can match the wildcard pattern. Supports comma-separated values. Valid values are the following:<br>- `all`: Match any index or data stream, including hidden ones. <br>- `open`: Match open, non-hidden indexes or non-hidden data streams. <br>- `closed`: Match closed, non-hidden indexes or non-hidden data streams. <br>- `hidden`: Match hidden indexes or data streams. Must be combined with `open`, `closed` or both `open` and `closed`.<br>- `none`: No wildcard patterns are accepted.<br> Optional. Default is `open`. | ||||||
| allow_partial_pit_creation | Boolean | Specifies whether to create a PIT with partial failures. Optional. Default is `false`. | ||||||
|
|
||||||
| #### Sample request | ||||||
|
|
||||||
| ```json | ||||||
| POST /my-index-1/_search/point_in_time?keep_alive=100m | ||||||
| ``` | ||||||
|
|
||||||
| #### Sample response | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "pit_id": "o463QQEPbXktaW5kZXgtMDAwMDAxFnNOWU43ckt3U3IyaFVpbGE1UWEtMncAFjFyeXBsRGJmVFM2RTB6eVg1aVVqQncAAAAAAAAAAAIWcDVrM3ZIX0pRNS1XejE5YXRPRFhzUQEWc05ZTjdyS3dTcjJoVWlsYTVRYS0ydwAA", | ||||||
| "_shards": { | ||||||
| "total": 1, | ||||||
| "successful": 1, | ||||||
| "skipped": 0, | ||||||
| "failed": 0 | ||||||
| }, | ||||||
| "creation_time": 1658146050064 | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### Response fields | ||||||
|
|
||||||
| Field | Data Type | Description | ||||||
| :--- | :--- | :--- | ||||||
| pit_id | [Base64 encoded binary]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/binary) | The PIT ID. | ||||||
| creation_time | long | The time the PIT was created in milliseconds since the epoch. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| ## Extend a PIT time | ||||||
|
|
||||||
| You can extend a PIT time by providing a `keep_alive` parameter in the `pit` object when you perform a search: | ||||||
|
|
||||||
| ```json | ||||||
| GET /_search | ||||||
| { | ||||||
| "size": 10000, | ||||||
| "query": { | ||||||
| "match" : { | ||||||
| "user.id" : "elkbee" | ||||||
| } | ||||||
| }, | ||||||
| "pit": { | ||||||
| "id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==", | ||||||
| "keep_alive": "100m" | ||||||
|
There was a problem hiding this comment. Can we give a comment that this is optional in search to give more stress? and provide it's default value as well? There was a problem hiding this comment. I will add the note below the request that the parameter is optional. What is the default value? There was a problem hiding this comment.
There was a problem hiding this comment. In search it should be completely optional. there is no default val as of now. |
||||||
| }, | ||||||
| "sort": [ | ||||||
| {"@timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos"}}, | ||||||
| {"_shard_doc": "desc"} | ||||||
| ], | ||||||
| "search_after": [ | ||||||
| "2021-05-20T05:30:04.832Z" | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| The `keep_alive` parameter in a search request is optional. It specifies the amount by which to extend the time to keep a PIT. | ||||||
| {: .note} | ||||||
|
|
||||||
| ## List all PITs | ||||||
|
There was a problem hiding this comment. Can we also add a snippet where user can extend keep alive using search API. |
||||||
| Introduced 2.4 | ||||||
| {: .label .label-purple } | ||||||
|
|
||||||
| Returns all PITs in the OpenSearch cluster. | ||||||
|
There was a problem hiding this comment. Can we give an example of list all PITS api here? There was a problem hiding this comment. The example is in the "sample request" section below. |
||||||
|
|
||||||
| ### Cross-cluster behavior | ||||||
|
There was a problem hiding this comment. @bharath-techie @dhruv16dhr Should we have the cross cluster behaviour in different section for PIT apis? Defining it like this can be confusing for a new user since we are giving sample example under There was a problem hiding this comment. I feel like as a user , they might miss reading upon cross cluster behavior if its down below. |
||||||
|
|
||||||
| The List All PITs API returns only local PITs or mixed PITs (PITs created in both local and remote clusters). It does not return fully remote PITs. | ||||||
|
|
||||||
| #### Sample request | ||||||
|
|
||||||
| ```json | ||||||
| GET /_search/point_in_time/_all | ||||||
| ``` | ||||||
|
|
||||||
| #### Sample response | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "pits": [ | ||||||
| { | ||||||
| "pit_id": "o463QQEPbXktaW5kZXgtMDAwMDAxFnNOWU43ckt3U3IyaFVpbGE1UWEtMncAFjFyeXBsRGJmVFM2RTB6eVg1aVVqQncAAAAAAAAAAAEWcDVrM3ZIX0pRNS1XejE5YXRPRFhzUQEWc05ZTjdyS3dTcjJoVWlsYTVRYS0ydwAA", | ||||||
|
There was a problem hiding this comment. should we mention in comments that this one is local PIT or remote pit? @bharath-techie There was a problem hiding this comment. Lets not do that for now, it might be confusing for user. |
||||||
| "creation_time": 1658146048666, | ||||||
| "keep_alive": 6000000 | ||||||
| }, | ||||||
| { | ||||||
| "pit_id": "o463QQEPbXktaW5kZXgtMDAwMDAxFnNOWU43ckt3U3IyaFVpbGE1UWEtMncAFjFyeXBsRGJmVFM2RTB6eVg1aVVqQncAAAAAAAAAAAIWcDVrM3ZIX0pRNS1XejE5YXRPRFhzUQEWc05ZTjdyS3dTcjJoVWlsYTVRYS0ydwAA", | ||||||
| "creation_time": 1658146050064, | ||||||
| "keep_alive": 6000000 | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### Response fields | ||||||
|
|
||||||
| Field | Data Type | Description | ||||||
| :--- | :--- | :--- | ||||||
| pits | Array of JSON objects | The list of all PITs. | ||||||
|
|
||||||
| Each PIT object contains the following fields: | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| Field | Data Type | Description | ||||||
| :--- | :--- | :--- | ||||||
| pit_id | [Base64 encoded binary]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/binary) | The PIT ID. | ||||||
| creation_time | long | The time the PIT was created in milliseconds since the epoch. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| keep_alive | long | The amount of time to keep the PIT in milliseconds. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| ## Delete PITs | ||||||
| Introduced 2.4 | ||||||
| {: .label .label-purple } | ||||||
|
|
||||||
| Deletes one, several, or all PITs. PITs are automatically deleted when the `keep_alive` time period elapses. However, to free resources, you can delete a PIT using the Delete PIT API. The Delete PIT API supports deleting a list of PITs by ID or deleting all PITs at once. | ||||||
|
There was a problem hiding this comment. Instead of "to free resources", would "to make resources available" work? There was a problem hiding this comment. It's not the same thing. Changed to "de-allocate". There was a problem hiding this comment. "deallocate" (no hyphen) |
||||||
|
|
||||||
| ### Cross-cluster behavior | ||||||
|
|
||||||
| The Delete PITs by ID API fully supports deleting cross-cluster PITs. | ||||||
|
|
||||||
| The Delete All PITs API deletes only local PITs or mixed PITs (PITs created in both local and remote clusters). It does not delete fully remote PITs. | ||||||
|
|
||||||
| #### Sample Request: Delete all PITs | ||||||
|
|
||||||
| ```json | ||||||
| DELETE /_search/point_in_time/_all | ||||||
| ``` | ||||||
|
|
||||||
| If you want to delete one or several PITs, specify their PIT IDs in the request body. | ||||||
|
|
||||||
| ### Request fields | ||||||
|
|
||||||
| Field | Data Type | Description | ||||||
| :--- | :--- | :--- | ||||||
| pit_id | [Base64 encoded binary]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/binary) or an array of binaries | The PIT IDs of the PITs to be deleted. Required. | ||||||
|
|
||||||
| #### Sample request: Delete PITs by ID | ||||||
|
|
||||||
| ```json | ||||||
| DELETE /_search/point_in_time | ||||||
|
|
||||||
| { | ||||||
| "pit_id": [ | ||||||
| "o463QQEPbXktaW5kZXgtMDAwMDAxFkhGN09fMVlPUkVPLXh6MUExZ1hpaEEAFjBGbmVEZHdGU1EtaFhhUFc4ZkR5cWcAAAAAAAAAAAEWaXBPNVJtZEhTZDZXTWFFR05waXdWZwEWSEY3T18xWU9SRU8teHoxQTFnWGloQQAA", | ||||||
| "o463QQEPbXktaW5kZXgtMDAwMDAxFkhGN09fMVlPUkVPLXh6MUExZ1hpaEEAFjBGbmVEZHdGU1EtaFhhUFc4ZkR5cWcAAAAAAAAAAAIWaXBPNVJtZEhTZDZXTWFFR05waXdWZwEWSEY3T18xWU9SRU8teHoxQTFnWGloQQAA" | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| #### Sample response | ||||||
|
|
||||||
| For each PIT, the response contains a JSON object with a PIT ID and a `successful` field that specifies if the deletion was successful. Partial failures are treated as failures. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "pits": [ | ||||||
| { | ||||||
| "successful": true, | ||||||
| "pit_id": "o463QQEPbXktaW5kZXgtMDAwMDAxFkhGN09fMVlPUkVPLXh6MUExZ1hpaEEAFjBGbmVEZHdGU1EtaFhhUFc4ZkR5cWcAAAAAAAAAAAEWaXBPNVJtZEhTZDZXTWFFR05waXdWZwEWSEY3T18xWU9SRU8teHoxQTFnWGloQQAA" | ||||||
| }, | ||||||
| { | ||||||
| "successful": false, | ||||||
| "pit_id": "o463QQEPbXktaW5kZXgtMDAwMDAxFkhGN09fMVlPUkVPLXh6MUExZ1hpaEEAFjBGbmVEZHdGU1EtaFhhUFc4ZkR5cWcAAAAAAAAAAAIWaXBPNVJtZEhTZDZXTWFFR05waXdWZwEWSEY3T18xWU9SRU8teHoxQTFnWGloQQAA" | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### Response fields | ||||||
|
|
||||||
| Field | Data Type | Description | | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Implemented, although this does not make a difference in rendering :) |
||||||
| :--- | :--- | :--- | ||||||
| successful | Boolean | Whether the delete operation was successful. | ||||||
| pit_id | [Base64 encoded binary]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/binary) | The PIT ID of the PIT to be deleted. | ||||||
|
|
||||||
| ## PIT Segments | ||||||
|
There was a problem hiding this comment. should "Segments" be capitalized in this section? There was a problem hiding this comment. That's the API name so should be capitalized. There was a problem hiding this comment. I had the same question. There are previous headings, for example, "Delete all PITs", where we did not capitalize them as the name of the API. If the headings are intended to reflect the names of the APIs, let's capitalize them consistently, and would it make sense to include "API" in the headings for maximum clarity? There was a problem hiding this comment. Good point. Since there is no "official" name, I will make these sentence case and assume they refer to the action. |
||||||
| Introduced 2.4 | ||||||
| {: .label .label-purple } | ||||||
|
|
||||||
| Similarly to the [CAT Segments API]({{site.url}}{{site.baseurl}}/api-reference/cat/cat-segments), the PIT Segments API provides low-level information about the disk utilization of a PIT by describing its Lucene segments. The PIT Segments API supports listing segment information of a specific PIT by ID or of all PITs at once. | ||||||
|
|
||||||
| #### Sample request: PIT Segments of all PITs | ||||||
|
|
||||||
| ```json | ||||||
| GET /_cat/pit_segments/_all | ||||||
| ``` | ||||||
|
|
||||||
| If you want to list segments for one or several PITs, specify their PIT IDs in the request body. | ||||||
|
|
||||||
| ### Request fields | ||||||
|
|
||||||
| Field | Data Type | Description | ||||||
| :--- | :--- | :--- | ||||||
| pit_id | [Base64 encoded binary]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/binary) or an array of binaries | The PIT IDs of the PITs whose segments are to be listed. Required. | ||||||
|
|
||||||
| #### Sample request: PIT Segments of PITs by ID | ||||||
|
|
||||||
| ```json | ||||||
| GET /_cat/pit_segments | ||||||
|
|
||||||
| { | ||||||
| "pit_id": [ | ||||||
| "o463QQEPbXktaW5kZXgtMDAwMDAxFkhGN09fMVlPUkVPLXh6MUExZ1hpaEEAFjBGbmVEZHdGU1EtaFhhUFc4ZkR5cWcAAAAAAAAAAAEWaXBPNVJtZEhTZDZXTWFFR05waXdWZwEWSEY3T18xWU9SRU8teHoxQTFnWGloQQAA", | ||||||
| "o463QQEPbXktaW5kZXgtMDAwMDAxFkhGN09fMVlPUkVPLXh6MUExZ1hpaEEAFjBGbmVEZHdGU1EtaFhhUFc4ZkR5cWcAAAAAAAAAAAIWaXBPNVJtZEhTZDZXTWFFR05waXdWZwEWSEY3T18xWU9SRU8teHoxQTFnWGloQQAA" | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| #### Sample response | ||||||
|
|
||||||
| ```json | ||||||
| index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound | ||||||
| index1 0 r 10.212.36.190 _0 0 4 0 3.8kb 1364 false true 8.8.2 true | ||||||
| index1 1 p 10.212.36.190 _0 0 3 0 3.7kb 1364 false true 8.8.2 true | ||||||
| index1 2 r 10.212.74.139 _0 0 2 0 3.6kb 1364 false true 8.8.2 true | ||||||
| ``` | ||||||
|
|
||||||
| ## PIT Settings | ||||||
|
There was a problem hiding this comment. If "Settings" isn't a proper name, lowercase it. |
||||||
|
|
||||||
| You can specify the following settings for a PIT. | ||||||
|
|
||||||
| Setting | Description | Default | ||||||
| :--- | :--- | :--- | ||||||
| point_in_time.max_keep_alive | A cluster level setting that specifies the maximum value for the `keep_alive` parameter. | 24h | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| search.max_open_pit_context | A node level setting that specifies the maximum number of open PIT contexts for the node. | 300 | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"The amount of time"?