Skip to content

Commit

Permalink
Add a note about zstd incompatibility with k-NN and security analytics (
Browse files Browse the repository at this point in the history
#5014)

* Add a note about zstd incompatibility with  k-NN and security analytics plugins

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Capitalization

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Update _im-plugin/index-codecs.md

Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Update _im-plugin/index-codecs.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

---------

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
  • Loading branch information
2 people authored and vagimeli committed Dec 20, 2023
1 parent b47fd43 commit 1cd32ef
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions _im-plugin/index-codecs.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ It may be challenging to change the codec setting of an existing index (see [Cha
* `zstd` (OpenSearch 2.9 and later) -- This codec provides significant compression comparable to the `best_compression` codec with reasonable CPU usage and improved indexing and search performance compared to the `default` codec.
* `zstd_no_dict` (OpenSearch 2.9 and later) -- This codec is similar to `zstd` but excludes the dictionary compression feature. It provides faster indexing and search operations compared to `zstd` at the expense of a slightly larger index size.

As of OpenSearch 2.10, the `zstd` and `zstd_no_dict` compression codecs cannot be used for [k-NN]({{site.url}}{{site.baseurl}}/search-plugins/knn/index/) or [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) indexes.
{: .warning}

For the `zstd` and `zstd_no_dict` codecs, you can optionally specify a compression level in the `index.codec.compression_level` setting. This setting takes integers in the [1, 6] range. A higher compression level results in a higher compression ratio (smaller storage size) with a tradeoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies).

When an index segment is created, it uses the current index codec for compression. If you update the index codec, any segment created after the update will use the new compression algorithm. For specific operation considerations, see [Index codec considerations for index operations](#index-codec-considerations-for-index-operations).
Expand Down

0 comments on commit 1cd32ef

Please sign in to comment.