Make index.codec setting case-insensitive.

[mulugetam]

Description

OpenSearch currently supports default, best_compression, lucene_default, and ZSTD compression codecs for index.codec. The setting uses a case-sensitive comparison. If the user supplies BEST_COMPRESSION, for example, OpenSearch uses the default (LZ4) codec and not best_compression (deflate).

This PR addresses the issue by allowing a case-insensitive match for the setting. 

Issues Resolved

index.codec: BEST_COMPRESSION falls back to default (lz4) and not best_compression (deflate).

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14059/
• CommitID: 9047466
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14060/
• CommitID: 3349026
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14063/
• CommitID: 1d70a79
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dblock]

If the user supplies BEST_COMPRESSION, for example, OpenSearch uses the default (LZ4) codec and not best_compression (deflate).

I would expect an error, is this not the case? What if I specify foobar as a coded? Do we fallback silently?

This is an interface change that breaks bcw isn't it? If user was supplying LUCENE_DEFAULT before, were they getting lz4 quietly and would now be getting lucene_default?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14161/
• CommitID: 7cd6041
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14160/
• CommitID: 6bc10af
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[mulugetam]

If the user supplies BEST_COMPRESSION, for example, OpenSearch uses the default (LZ4) codec and not best_compression (deflate).

I would expect an error, is this not the case? What if I specify foobar as a coded? Do we fallback silently?

This is an interface change that breaks bcw isn't it? If user was supplying LUCENE_DEFAULT before, were they getting lz4 quietly and would now be getting lucene_default?

My bad, apparently the silent fallback has to do with OpenSearch-Benchmark (was using it for tests) and not OpenSearch. A direct REST call to OpenSearch with BEST_COMPRESSION throws an exception as expected.

I think the question now is whether we should allow for a case-insensitive match. I think we should. The current compression codecs are all in lower case except ZSTD: default, lucene_default, best_compression, ZSTD. I think we should either allow a case-insensitive matching or I should change ZSTD to zstd.

[mulugetam]

OpenSearch-Benchmark bug report: opensearch-project/opensearch-benchmark#268

@dblock @reta

[andrross]

I think we should either allow a case-insensitive matching or I should change ZSTD to zstd.

I vote for changing ZSTD to zstd for consistency, and keeping the case-sensitive behavior. I think we can make the backward-incompatible change as ZSTD support hasn't been released yet.

[reta]

OpenSearch-Benchmark bug report: opensearch-project/opensearch-benchmark#268

@dblock @reta

Thanks @mulugetam

I think the question now is whether we should allow for a case-insensitive match.

I would genuinely be surprised if ZSTD and zstd would be treated differently as codec name, may be we could use the case insensitive data structures for that:

        this.codecs = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);
        this.codecs.putAll(codecs.map());

[Poojita-Raj]

If we're allowing for users to set it to lucene_default and/or one of the listed available codecs (Lucene80, Lucene70, etc), this needs to be documented on the static settings for index creation for index.codec setting.

Ref: https://opensearch.org/docs/2.4/api-reference/index-apis/create-index/#static-index-settings

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14179/
• CommitID: 67ae16e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14190/
• CommitID: c1bc899
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[mulugetam]

Looks like @dblock @andrross @Poojita-Raj and for a case-sensitive index.codec. @reta, may I close this PR? (I will create a new one that changes ZSTD to zstd)?

[reta]

Looks like @dblock @andrross @Poojita-Raj and for a case-sensitive index.codec. @reta, may I close this PR? (I will create a new one that changes ZSTD to zstd)?

Sure, thanks @mulugetam !

[mulugetam]

Closing this PR and creating #7231 that changes the codec name from ZSTD to zstd.

@dblock @reta