add puml

Signed-off-by: Su <szhongna@amazon.com>

src/plugins/data_source/docs/high_level_design.md

@@ -19,7 +19,7 @@ From very high level, we propose to introduce `data-source` as a new OpenSearch
 
 ## Scope
 
-We are targeting to release the multiple data source support in OpenSearch 3.0 preview as an experimental feature, and make it GA over a few minor version throughout 3.x versions.
+We are targeting to release the multiple data source support in OpenSearch 2.x preview as an experimental feature, and make it GA over a few minor version throughout 2.x versions.
 
 ### Preview Scope
 
@@ -35,7 +35,7 @@ We are targeting to release the multiple data source support in OpenSearch 3.0 p
 
 ### GA Scope
 
-- Support all Elasticsearch 7.10 DSL/API compatible data sources, including customer self managed Elasticsearch 7.10, OpenSearch 3.x clusters, AWS managed OpenSearch and Elasticsearch 7.10 domains. OpenSearch Serverless collections.
+- Support all Elasticsearch 7.10 DSL/API compatible data sources, including customer self managed Elasticsearch 7.10, OpenSearch 2.x clusters, AWS managed OpenSearch and Elasticsearch 7.10 domains. OpenSearch Serverless collections.
   - Support Basic auth, AWS SigV4 signing with Data sources
 - OpenSearch Dashboards plugins such as Alerting/AD etc. can work with each data source depending on the data source capability
 - Observability visualizations are out of scope
@@ -138,7 +138,7 @@ Based on existing OpenSearch and OpenSearch Dashboards security implementations,
 
 Credentials is part of the data source object, and will be saved in OpenSearch Dashboards metadata index. OpenSearch Dashboards will use that credentials to authenticate with the data source when executing queries. This credentials will need to be encrypted regardless OpenSearch Dashboards has access control or not.
 
-We will use a symmetric key to encrypt the credentials before saving data source into OpenSearch Dashboards metadata index, and use the same key to decrypt it when OpenSearch Dashboards needs to authenticate with corresponding data source. For open source release, we will allow admins to configure the encryption key in the `opensearch_dashboards.yml` file. And we will also provide the option to integrate with a key store, such as AWS KMS, to use keys with the key store.
+We will use a symmetric key to encrypt the credentials before saving data source into OpenSearch Dashboards metadata index, and use the same key to decrypt it when OpenSearch Dashboards needs to authenticate with corresponding data source. For open source release, we will allow admins to configure the encryption key in the `opensearch_dashboards.yml` file.
 
 For more about credential encryption/decryption strategy, refer to [#1756](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1756)
 

src/plugins/data_source/docs/resources/client_management_architecture.puml

@@ -0,0 +1,126 @@
+@startuml
+hide stereotype
+skinparam nodesep 6
+skinparam ranksep 10
+
+skinparam component {
+backgroundColor<<new>> Green
+backgroundColor<<modify>> Orange
+backgroundColor<<existing>> LightYellow
+backgroundColor<<external>> LightGrey
+backgroundColor<<AOS>> Khaki
+backgroundColor<<Data>> Grey
+backgroundColor<<CS>> LightGrey
+}
+skinparam rectangle {
+backgroundColor<<new>> Green
+backgroundColor<<modify>> Orange
+backgroundColor<<existing>> LightYellow
+backgroundColor<<external>> LightGrey
+backgroundColor<<AOS>> Khaki
+backgroundColor<<Data>> Grey
+backgroundColor<<CS>> LightGrey
+}
+skinparam node {
+backgroundColor<<new>> Green
+backgroundColor<<modify>> Orange
+backgroundColor<<existing>> LightYellow
+backgroundColor<<external>> LightGrey
+backgroundColor<<AOS>> Khaki
+backgroundColor<<Data>> Grey
+backgroundColor<<CS>> LightGrey
+}
+
+title ** OSD Multi Data Source Client Management Architecture **
+
+node "Dashboards" as cluster {
+rectangle "Legend" {
+    rectangle "New" as new <<new>>
+    rectangle "Modified" as modify <<modify>>
+    rectangle "Existing" as existing <<existing>>
+    rectangle "External" as external <<external>>
+    new -[hidden]right- modify
+    modify -[hidden]right- existing
+    existing -[hidden]right- external    
+}
+
+  rectangle "Other Plugins" <<existing>> {
+    rectangle "visualization" as viz <<existing>> {
+
+    }
+    rectangle "Alerting or other" as a <<modify>> {
+
+    }
+  }
+  rectangle "Data Plugin" <<Existing>> {
+    rectangle "Search Module" as sm <<existing>> {
+      rectangle "Search Source" <<modify>> as source {
+
+      }
+      rectangle "Search Strategy" as strategy <<modify>> {
+
+      }
+
+    }
+    interface "DataPluginStart" as dps
+  }
+
+  rectangle "Core" <<existing>> as core {
+       rectangle "OpenSearch Data Service" as ods <<new>> {
+        component "DataSource Client" as dc <<new>>
+        interface "ServiceStart" as odsss
+      }
+      rectangle "opensearch service" as os_service <<modify>> {
+
+        component "internal/scoped client" as ic <<existing>>
+        interface "ServiceStart" as osss
+
+      }
+      interface "CoreStart" as core_start
+      rectangle "saved object service" as sos <<existing>> {
+        interface "ServiceStart" as soss
+      }
+
+      interface "CoreRouteHandlerContext" as cc <<modify>>
+
+
+  }
+
+  rectangle "Credential Manager" <<new>> as cm {
+
+  }
+
+  ods --> cm: decrypt user credential
+  dc --> es: query
+  ' os_service -[hidden]- cc
+  source -> strategy: call
+  strategy --> cc: get datasource client 
+  viz --> dps: speical viz types
+  viz --> source
+  dps --> sm
+  sos --> os: get saved objects
+  ' d --> sos: get datasource
+  core_start --> cc
+  core_start <.. a: get client
+  a ..> dps
+  a ...> cc: get client
+  core_start <-- osss: register
+  osss <-- soss: depends
+  soss <--- odsss: depends
+  ' soss <--> core_start
+  ' d ---> es: query
+  ic --> os: query
+  odsss --> core_start: register
+
+
+
+
+  rectangle "Default OpenSearch" <<existing>> as os {
+  }
+  rectangle "Datasource(OpenSearch)" <<external>> as es {
+
+  }
+}
+
+
+@enduml

src/plugins/data_source/docs/resources/client_management_dataflow.puml

@@ -0,0 +1,73 @@
+@startuml
+autoactivate on
+
+title ** Multiple Datasource Visualization call sequence **
+
+box "OSD Browser"
+participant "visualization" as viz
+' participant "Timeline/Vega/TSVB" as viz_s
+participant "expression" as e
+end box
+
+
+box "OSD Server-Data Plugin" #LightBlue
+participant "SearchSource\n(High Level API)" as s
+participant "Search Strategy\n(Low Level API)" as ss
+end box
+
+box "OSD Server-Core" #LightBlue
+participant "OpenSearch Service" as os 
+participant "OpenSearch Data Service" as ods #LightGreen
+participant "Saved Object Service" as sos
+
+end box
+
+box "OpenSearch"
+database "OSD metadata" as oi
+database "data index" as default_di
+end box
+
+box "DataSource[OS]"
+database "data index" as datasource_di
+end box
+
+sos --> os: depends on
+ods --> sos: depends on
+
+viz -> e: execute expression pipeline
+e -> s: create SearchSource 
+s -> ss: call .search()
+alt viz_type = Timelion/TSVB/Vega
+viz -> ss: call .search()
+end
+
+ss -> os: get client
+alt if (datasource == true)
+ss -> ods: get datasource client
+alt if exists in datasource client pool
+ods -> ods: find client
+end
+
+ods -> sos: call saved_obj_client
+sos -> oi: get datasource metadata
+oi --> sos:
+sos --> ods: datasource metadata
+ods --> ods: create datasource client \n and add to pool
+ods --> ss: return client
+
+end
+os --> ss: return client
+alt if (client is datasource Client)
+ss -> datasource_di: query
+datasource_di --> ss: data 
+end
+
+ss -> default_di: query
+default_di --> ss: data 
+
+ss --> s: data
+s --> e: data
+e --> viz: render data
+
+skinparam BoxPadding 15
+@enduml

src/plugins/data_source/docs/resources/cm_flow.puml

@@ -0,0 +1,48 @@
+@startuml
+title Credential Management in Stack Management
+
+
+:Credential Owner: as CO
+
+(Stack Management Page) as (Page-Stack)
+(Credential Management Page) as (Page-CM)
+(Add new Credential Page) as (Page-ANC)
+(Credential Grid View) as (Component-CG)
+(Credential Edit Page) as (Page-CEP)
+(Delete Credential Button) as (Component-DelC)
+
+(Credential Id) as (Component-CId)
+(Credential Name) as (Component-CName)
+(Credential Type) as (Component-CType)
+(Credential Key | UserName) as (Component-CKey)
+(Credential Value | Password) as (Component-CValue)
+
+
+CO -> (Page-Stack)
+(Page-Stack) -> (Page-CM)
+(Page-CM) -> (Page-ANC) : Add
+(Page-ANC) -> (Page-CM) : Save
+(Page-CM) ...> (Component-CG): View
+(Page-CM) <..> (Component-DelC) : Delete
+
+(Component-CG) -> (Page-CEP): Edit
+
+(Page-CEP) .....> (Component-CId) : ReadOnly, Auto Generated
+(Page-CEP) .....> (Component-CName) : Edit
+(Page-CEP) .....> (Component-CType): Edit
+(Page-CEP) .....> (Component-CKey): Edit
+(Page-CEP) .....> (Component-CValue): Edit
+
+note top of CO
+  DataSource Owner who has access to manage 
+   all Credentials.
+end note
+
+note bottom of (Component-CType)
+  OpenSearch supported credential type (Not Dashboards)
+  1. User name/Password
+  2. AWS Credential(AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY)
+end note
+
+
+@enduml

src/plugins/data_source/docs/resources/dsm_flow.puml

@@ -0,0 +1,45 @@
+@startuml 
+title DataSource Management in Stack Management
+
+
+:DataSource Owner: as DSO
+:DataSource User: as DSU
+
+(Stack Management Page) as (Page-Stack)
+(DataSource Management Page) as (Page-DM)
+(Add new DataSource Page) as (Page-AND)
+(DataSource Grid View) as (Component-DSG)
+(DataSource Edit Page) as (Page-DSEP)
+(Delete DataSource Button) as (Component-DelDS)
+(Export DataSource Button) as (Component-ExDS)
+(Import DataSource Button) as (Component-ImDS)
+
+
+(DataSource Name) as (Component-DSName)
+(DataSource Type) as (Component-DSType)
+(DataSource Endpoint) as (Component-DSEndpoint)
+(DataSource Credential) as (Component-DSCredential)
+
+DSO -> (Page-Stack)
+(Page-Stack) -> (Page-DM)
+(Page-DM) -> (Page-AND) : Add
+(Page-AND) -> (Page-DM) : Save
+(Page-DM) ...> (Component-DSG): View
+(Page-DM) <..> (Component-DelDS) : Delete
+(Page-DM) <..> (Component-ExDS) : Export
+(Page-DM) <..> (Component-ImDS) : Import
+
+(Component-DSG) -> (Page-DSEP): Edit
+
+(Page-DSEP) .....> (Component-DSName) : Edit
+(Page-DSEP) .....> (Component-DSType): Edit
+(Page-DSEP) .....> (Component-DSEndpoint): Edit
+(Page-DSEP) .....> (Component-DSCredential): Select
+
+note top of DSO
+  DataSource Owner who has access to manage 
+   all DataSources.
+  When security enabled, user could only see
+   DataSources added by them
+end note
+@enduml

src/plugins/data_source/docs/user_stories.md

@@ -48,14 +48,6 @@ We will need to
 - add a new data source entry in the stack management Nav app, with a data source list table
 - a data source detail page, to show detailed information of a specific data source, such as URL, auth type, endpoint capabilities etc.
 
-### Credential management
-
-![img](./img/cm_flow.png)
-
-Credential is used to establish an authenticated connection to other data source. Typical credentials are User/Password for basic auth. IAM auth for AWS specific authentication.
-
-Credential management is provided way for user to add/edit/remove the credential to be used in datasource management
-
 ### Index Pattern
 
 - Index pattern creation flow: With the data sources, users will need to specify which data source to use when creating a new index pattern.
@@ -66,8 +58,8 @@ Credential management is provided way for user to add/edit/remove the credential
 
 ### Data source security
 
-For the initial launch with OpenSearch 3.0 preview, we do not plan to change security design of OpenSearch.
+For the initial launch with OpenSearch 2.4 preview, we do not plan to change security design of OpenSearch.
 
-Users need to provide endpoint URL, username and password when creating a data source. OpenSearch Dashboards service will encrypt the username and password when storing it into metadata store.
+When creating a data source, users need to provide endpoint URL, username and password(if using basic authentication). OpenSearch Dashboards service will encrypt the username and password when storing it into metadata store.
 
 Data source is a new type of OpenSearch Dashboards saved objects. In current OpenSearch security model, access control on data source document is the same as other saved objects documents. Basically data source docs will be accessible by any user who has access to the tenant.