Make Custom Result Index Name an Alias

[kaituo]

Description

Previously, we used configured custom result index names as the index name, creating the index only when a detector was created. This approach caused several problems for index management:

1. Index State Management (ISM) Incompatibility: ISM requires an alias for rolling over an index, but we only had the index name.
2. Index Recognition Issue: Even if we managed to roll over an index, AD could not recognize it because the index would not be recreated unless the detector was recreated.
3. Result History and Top Anomaly Results: The result history on the dashboard and top anomaly results were reading from a single index instead of an index pattern. Thus, after rolling over an index, the result history and top anomaly results would be lost.

This PR addresses these issues:

1. Custom Result Index Creation: We now create a custom result index with the name <custom-name-history-{now/d}-1> and use the alias custom-name to point to it.
2. Index Recreation: We recreate an index when failing to find a result index, updating the configured alias to point to the new index. This ensures continuity when an index is rolled over and new indices do not exist.
3. Query Index Pattern: The top anomaly result API now queries an index pattern instead of a single index. The result history on the dashboard follows the same approach. The dashboard code will be posted in a separate PR.

Additionally, this PR updates the custom result index mapping when the mapping is outdated, similar to existing logic on the default result index.

Testing Done:

• Successfully updated the mapping of the custom result index when the mapping is outdated.
• Verified that the frontend can still see old and new results after a rollover.
• Confirmed that the backend can still write to new indices after a rollover.
• Ensured all existing tests pass.

More tests will be added in the following PRs.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[jackiehanyang]

Synced offline about this pr. Thanks for the change!

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/anomaly-detection/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/anomaly-detection/backport-2.x
# Create a new branch
git switch --create backport/backport-1225-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4d400e3c926528ec884b54749e035c7855fb104c
# Push it to GitHub
git push --set-upstream origin backport/backport-1225-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/anomaly-detection/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1225-to-2.x.