[Question] How to Support System Indexes for Extensions?

[peternied]

How to Support System Indexes for Extensions

This document is structured to decide how to support system index feature for extensions.

• How to Support System Indexes for Extensions
• Background
• Problem
• Option 1a: Extensions do not use system indexes, use new index
• Option 1b: Extensions do not use system indexes, use local storage
• Option 2a: Extension service account, with Admin access
• Option 2b: Extension service account, scoped to specific system index [Recommended]
• Appendix
  • storeContext() allows elevation of privileges
  • Anomaly detection system index plan
  • Details for supporting permissions for system indices
    • OpenSearch
  • Security Plugin
    • Supporting more granular access to system indexes
  • Dynamic Configuration
• Implementation Details
  • New permission for read/write access on system indexes
  • Updates to SecurityIndexAccessEvaluator
  • Documentation
  • Release Version

Background

System indexes are used in OpenSearch to declare indexes that cannot be altered by standard users or applications interacting with the platform.

OpenSearch plugins utilize system indexes to store information that persists through the cluster, such as the security configuration for the Security Plugin or .replication-metadata-store indexes for cross cluster replication. These indexes contain data specific to the plugin and should not be altered by any other plugin or application.

To protect these indexes, administrative control is required. When plugins receive a user request that would cause an update to the system index, they elevate privileges using storeContext() during the operation, [see storeContext() allows elevation of privileges below]. Once complete, the context object is closed to return to the user's permission level.

Problem

Extensions do not have the ability to elevate privileges, which means they cannot access system indexes. To overcome this limitation, extensions will require a way to access and store protected metadata.

Open Question: Is extension metadata required to be protected as if it was a system index? .kibana is NOT a system index. Following up on opensearch-project/OpenSearch#6589

Yes - this is a high value feature for plugin developers.

Option 1a: Extensions do not use system indexes, use new index

This approach requires extensions to create and manage new indexes to store their metadata.

We will 1) selecting a new index for the extension, 2) modifying the plugin to detect the extension and migrate its data 3)then using the new index for storing the extension's data. This process needs to be repeated for each plugin/extension pair.

Pros:

• No changes on core OpenSearch or Security Plugin are needed.
• Extension data can be protected with normal index permissions.

Cons:

• Extension data is not as well protected as before.
• Plugins to implement their own migration.

Option 1b: Extensions do not use system indexes, use local storage

This approaches requires extensions to create and manage a new data store to store their metadata.

We will 1) modify the plugin on detection of extension to send it’s existing metadata, 2) the extension would register an action for receiving metadata migration via request, 3) extension would implement mapping from existing metadata storage to new local storage, 4) extension would migrate all calling patterns adopt local storage. Repeat this process for every plugin/extension pair.

Pros:

• Extension data is isolated
• No changes on core OpenSearch systems are needed

Cons:

• Plugins use OpenSearch index functionality, require considerable work to get parity
• Data would not be easily migratable back to OpenSearch cluster

Option 2a: Extension service account, with Admin access

This approaches requires extensions to have a way to interact with the OpenSearch cluster, such as a service account. This account would have admin access to the cluster. Managing how these interaction performed is covered in Asynchronous Operations for Extensions Decision Doc

We will 1) create a service account on extension registration associated with that extension, 2) this account has admin permissions, 3) have the extension store this service account information, 4) have extensions use the service account to read/write the metadata store.

Pros:

• No migration of existing plugin metadata
• Aligned with other work, service account interaction is required for other features

Cons:

• Extension have admin access to OpenSearch cluster.

Option 2b: Extension service account, scoped to specific system index [Recommended]

This approaches requires extensions to have a way to interact with the OpenSearch cluster, such as a service account. This account would have restricted access to some system indexes. Managing how these interaction performed is covered in Asynchronous Operations for Extensions Decision Doc

We will 1) create a service account on extension registration associated with that extension, 2) create a role mapping to the extension’s metadata indexes and the extension service account 3) have the extension store this service account information, 4) have extensions use the service account to read/write the metadata store.

Pros:

• No migration of existing plugin metadata.
• Extension metadata is isolated from other extensions.
• Aligned with other work, service account interaction is required for other features

Cons:

• Requires changes in security plugin, core, and extensions sdk components

Appendix

storeContext() allows elevation of privileges

When users make requests into a cluster the Security Plugin is involved by extracting authentication information from the request and saving those details into the thread context which are available to transport actions. If additional actions are performed while handling that request the permissions are persisted.

When plugins receives a request on a transport action the user information is already present, actions to the cluster will be check as if the user made additional requests. However; for scenarios where plugins want to read/write metadata they need to stop making requests as the user.

try (StoredContext context = threadContext.stashContext()) {
   // Executes an action without the current user context
   client.execute(action, request);
}

Anomaly detection system index plan

Extensions team has previously considered renaming the plugin’s index names to no longer use the ‘.’ prefix. Ultimately they choose not to make any changes to the index name conventions due to the migration process. No consideration as to access policy changes.
From https://github.com/opensearch-project/opensearch-sdk-java/issues/241

Details for supporting permissions for system indices

OpenSearch

System indexes actions are run on their own threadpool that does not get throttled as standard indexes are. There are other behaviors when prioritizing tasks [link], cluster stats reporting and other behaviors [enumeration link&type=code)].

System indices are defined by implementing the SystemIndexPlugin, which returns a list of indices, example. These are scanned and loaded when the OpenSearch service starts the node [link]. This workflow ensures overlap does not occur between these patterns and only allows indexes to be created that follow these conventions, an index starting with a period character that is not listed is not allowed.

Security Plugin

Security plugin provides extra protection on system indices by denying writes to any account that isn’t an admin account. This is done within the SecurityFilter where non-admin follows reach a call to PrivilegesEvaluator which has a list of evaluators including SecurityIndexAccessEvaluator. This checks the indices associated with the request and if a system index is found the request is marked presponse.allowed = false.

public class SecurityIndexAccessEvaluator {
    public PrivilegesEvaluatorResponse evaluate(
        final ActionRequest request,
        final Task task,
        final String action,
        final Resolved requestedResolved,
        final PrivilegesEvaluatorResponse presponse)  { 
            ...
        }
}

Supporting more granular access to system indexes

Note: There is a feature called ‘protected indices’ which is not documented. While similar to the SecurityIndexAccessEvaluator, ProtectedIndexAccessEvaluator has a limited support for granularity only allowing a list of roles that map to a list of indexes, as the relationship isn’t a mapping between roles and indexes it cannot be used to support this feature.

SecurityIndexAccessEvaluator will need to be modified to allow for user context to be passed to in its evaulate method. Existing actions follow the convention of cluster: or restapi:, to support these scenarios I would recommend a new action type such as system:admin/system_index. So long as the user has the associated index permission assigned to the role the request would be allowed. Example role configuration below.

### Roles.yml
ad_system_index:
  index_permissions:
    - index_patterns:
        - '.opendistro-anomaly-results*'
        - '.opendistro-anomaly-detector*'
        - '.opendistro-anomaly-checkpoints'
      allowed_actions:
        - 'system:admin/system_index'
        - '*'

By building off of existing index permissions only specific indexes can be allowed access for individual roles. This can be built out and tested independently of any extensions related functionality.

Dynamic Configuration

High level proposal in the recommended option [link] can be implemented by modifying static files on disk. As there are many files are associated with permissions on bootstrap this could be hard to maintain. Existing clusters will need a cookbook that can be followed to enable minimal functionality. Some considerations are as follows:

• Account creation, role creation, and role mapping can be performed by an API calls to the security plugin
• Extension systems are not dynamic [META] Add ability to add/update/remove an extension without restarting OpenSearch opensearch-sdk-java#356
• System Index are only loaded once on node startup https://github.dev/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/node/Node.java#L667

Implementation Details

After reviewing with members of the security team and soliciting feedback from plugin teams, Option 2b: Extension service account, scoped to specific system index will be the chosen solution.

New permission for read/write access on system indexes

A new permissions should be created a starting point is system:admin/system_index, where system: is a new convention. The new permission name should be used where {NEW PERMISSION R/W SYSTEM INDEX} is seen in the rest of this document

Updates to SecurityIndexAccessEvaluator

SecurityIndexAccessEvaluator is responsible for accepting/rejecting modifications to the system index. This class will be modified so the evaluate(...) method includes the permissions from the current user has '{NEW PERMISSION R/W SYSTEM INDEX}' permissions for each index in the request. If allowed on all the system indexes in the request, then preemptively return, otherwise use the conventional evaluation flow.

Along with this should be a new integration test that verify:

• The new permissions applies to the indexes that match its index pattern, no other system indexes.
• The new permissions allows for read and write operations on the permitted index(es)
• (Should already exist, if not add) Make sure that '*' allowed actions doesn't automatically give '{NEW PERMISSION R/W SYSTEM INDEX}' access

Documentation

End user documentation
New documentation on this feature should be added for administrators to configure/use this functionality.

Developer documentation
The migration documentation for plugin -> extensions needs to be updated to include examples for this feature

Release Version

This new feature for allowing access to system indexes by users can be included with any update to the security plugin, it should be released on any 2.x branch.

[peternied]

Open Question: Is extension metadata required to be protected as if it was a system index? .kibana is NOT a system index. Following up on opensearch-project/OpenSearch#6589

Yes - this is a high value feature for plugin developers. This points all indications that Option 2b: Extension service account, scoped to specific system index [Recommended] is moved forward with. I'll update the document with a section on detailed design elements.

[peternied]

We have locked in on Option 2b: Extension service account, scoped to specific system index [Recommended]

[saratvemulapalli]

Option 1a/b both don't solve the problem of migration of existing plugins to extensions.
Ideally extensions should not be treated with special features.
I like where we ended up with 2b, I would take the step further to implement generic system level permissions out of which system index is one API.