Verify Usage of Opensearch Core System and Hidden Indices

[Rishikesh1159]

Description/Concept of System Indices and Hidden Indices on Opensearch core:

System Index - An index must extend system index plugin for a index to be called as system index.
Example: Security Plugin correctly extends and uses system Indices, more info here

Hidden index - An index must have hidden SETTING_INDEX_HIDDEN = "index.hidden" set on the index setting to call it as hidden. It doesn't matter if it starts with "." or not.
Example: Asynchronous-Search plugin correctly sets the index setting value here

Misconception:

Many plugins still misunderstand the actual definition of system and hidden indices. Usual misconception is that any index starting with . like .indexName is a system or hidden index, but this is incorrect. Any user can create an index with .indexName which is neither a system or hidden index. So users can mistakenly still create index starting with "." there is nothing stopping them from doing it.

To avoid this misconception all plugins should adopt/on-board with concept of system and hidden indices defined in opensearch core.

Goal:

The main ask of this issue is to make sure all plugins having/using system and hidden indices must on-board/adopt with concept defined in opensearch core.

Any plugin already on-board with opensearch core defined concept of system and hidden indices can ignore this issue and close the issue as completed on the plugin repo.

Additional info:

The following info provided below is not necessary for system/hidden indices, but might be useful info for plugins using system indices :

If your system indices need additonal security features/benefits provided by security plugin, follow the steps provided here. But to make sure these are additonal features provided by security plugin and it is completely decoupled from concept of system indices. It is upto the plugin owners to decide if they need these additional security benefits.

Open questions

In case of any questions or issues, please post it in core issue

[AMoo-Miki]

While plugins should be using the SavedObjects API and not create their own indices, OSD should look into how they define their metadata-store.

[Rishikesh1159]

@kavilla @AMoo-Miki do we have any update on this issue? If not can you guys please prioritize this one.

[kavilla]

@Rishikesh1159 thanks for following up.

I think it depends on any plans or breaking changes related to system indices. As of right now, OpenSearch Dashboards treats dot prefixes differently from other indices. For example here.

But it is a "system index" of OpenSearch Dashboards but to OpenSearch Core it is just another index. So it seems like unless we let OpenSearch Core manage our system index (ie we create a plugin within OpenSearch), then we have to make it a hidden index. We can still mark it as a hidden index and still filter, but it doesn't quite accurately convey the purpose of the index within OpenSearch Dashboards. It doesn't store data besides what is used to make OpenSearch Dashboards functional.

Is there another option here?

[Arpit-Bandejiya]

I was going through the OS core codebase and found that we already have a plugin module where we define the OSD indices as system indices here:

https://github.com/opensearch-project/OpenSearch/blob/afa67df3ba58d5642274877eea55b640bdc2820a/modules/opensearch-dashboards/src/main/java/org/opensearch/dashboards/OpenSearchDashboardsModulePlugin.java#L80-L96C2

Will it be possible to verify this list and update it with all the indices created by OSD? I think this should solve this issue.

Thoughts @Rishikesh1159 @AMoo-Miki @kavilla on it?

[AMoo-Miki]

@Arpit-Bandejiya, part of the problem i guess is that on OSD's side, the index names are dynamic (config-driven). That is the reason I believe it should be OSD's job to mark the indexes it uses as system; that piece of code in core shouldn't exist for OSD and OSD plugins at all.