[Feature Proposal] Client Side Encryption in OpenSearch

[vikasvb90]

Client Side Encryption in OpenSearch

Overview

Currently, OpenSearch does not provide support for client side encryption. Due to this, features like Remote Storage, back up and restore of snapshots, etc. require an additional layer of encryption to be built to secure and persist the data in remote store.
This is a request for comments to discuss the proposal for building a plugin which can provide client side encryption support but not limited to remote stores.

Proposal

We propose to build an encryption plugin which can provide capabilities to support encryption/decryption. This plugin can be used for encrypting data during remote transfer but should be independent of any remote store context and therefore, any other future requirement such as local encryption should work without any change in the plugin. Plugin should offer the following capabilities :

1. Encryption or decryption of a complete file.
2. Encryption or decryption of a portion of a file. This will be useful in cases where processing of a file happens via multiple threads such as parallel multi-part remote transfers.
3. Estimation of the size of encrypted content given the size of decrypted content or vice-versa.

We also believe that this should follow SoC model and should be independent of type of content like segments, translogs, Lucene Directories, etc. and should independently function.

[vikasvb90]

Tagging @shwetathareja @itiyamas @elfisher @muralikpbhat @reta @mch2 @dreamer-89 @andrross @Bukhtawar @sachinpkale for feedback. Feel free to tag others or add feedbacks.

[reta]

@vikasvb90 we have been discussed the idea of having such a plugin few times, definitely would be useful. Aiven has such a plugin [1] which we could contribute back to OpenSearch project.

As a side note, the description of the feature is somewhat misleading, I think we are not looking into encryption at rest (on a storage level, AFAIK most storage providers offer that out of the box) but indeed client side encryption.

[1] https://github.com/aiven/encrypted-repository-opensearch

[reta]

I believe this is a duplicate of #5800

[vikasvb90]

Aiven has such a plugin [1] which we could contribute back to OpenSearch project.

@reta Aiven offers some of the basic encryption/decryption capabilities by adding wrappers around open source encryption/decryption sdk but we are looking to extend further in this plugin by providing capabilities like block level/partial encryption or decryption, estimation of encrypted or decrypted content, etc. which might be useful in some remote transfer scenarios. We can discuss the design details further in the design doc.
Also, this plugin need not be dependent on remote transfers and should be able to cater to local encryption use cases if required.

I believe this is a duplicate of #5800

Linked issue has specific mentions of Aiven and proposes to use that as a solution. Proposal here though is open for feedback.

[reta]

Hm ... the description is not giving enough details on what you are looking at, would be helpful to update it and consolidate the related work like another one for example #3469

[vikasvb90]

@reta I have updated the description to list some minimal use cases this plugin should offer. To emphasize again, we are leaving out the low level details from the doc and will be looking forward to discuss those in the design doc so that we can get ideas around what more or different this plugin should do.