[Workspace][Data Source] Support data source assignment in workspace #7101

raintygao · 2024-06-25T09:19:15Z

Description

Add data source assignment in workspace create/update page.
Add addToWorkspaces and deleteFromWorkspaces method in savedObject client.
Update savedObject update/bulkUpdate in savedObject client and repository to support update workspace field.

Screenshot

demo.mp4

Testing the changes

Use find API to find any data sources and what workspaces they belong to. http://localhost:6601/w/${OSD_ID}/api/saved_objects/_find?fields=id&fields=title&fields=workspaces&per_page=10000&type=data-source&workspaces=*
Enter workspace create/update page to create workspaces and assign data sources or update and assign data sources.
Enter workspace update page or use find API to find that data source have been assigned to specific workspace.
http://localhost:6601/w/${OSD_ID}/api/saved_objects/_find?fields=id&fields=title&fields=workspaces&per_page=10000&type=data-source&workspaces=${WORKSPACE_ID}

Changelog

feat: Support data source assignment in workspace.

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

codecov · 2024-06-25T09:33:03Z

Codecov Report

Attention: Patch coverage is 89.31298% with 14 lines in your changes missing coverage. Please review.

Project coverage is 67.56%. Comparing base (417246b) to head (5004aee).
Report is 351 commits behind head on main.

Files with missing lines	Patch %	Lines
src/plugins/workspace/server/routes/index.ts	22.22%	7 Missing ⚠️
...onents/workspace_form/select_data_source_panel.tsx	91.30%	0 Missing and 2 partials ⚠️
src/plugins/workspace/server/workspace_client.ts	91.30%	0 Missing and 2 partials ⚠️
...rver/saved_objects/service/saved_objects_client.ts	94.44%	0 Missing and 1 partial ⚠️
...components/workspace_creator/workspace_creator.tsx	80.00%	0 Missing and 1 partial ⚠️
...components/workspace_updater/workspace_updater.tsx	90.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7101      +/-   ##
==========================================
+ Coverage   67.50%   67.56%   +0.06%     
==========================================
  Files        3468     3469       +1     
  Lines       68402    68508     +106     
  Branches    11117    11141      +24     
==========================================
+ Hits        46174    46287     +113     
+ Misses      19532    19515      -17     
- Partials     2696     2706      +10

Flag	Coverage Δ
Linux_1	`33.17% <70.22%> (+0.12%)`	⬆️
Linux_2	`55.25% <96.15%> (+0.04%)`	⬆️
Linux_3	`45.31% <0.00%> (-0.04%)`	⬇️
Linux_4	`34.73% <7.69%> (-0.02%)`	⬇️
Windows_1	`33.19% <70.22%> (+0.12%)`	⬆️
Windows_2	`55.20% <96.15%> (+0.04%)`	⬆️
Windows_3	`45.32% <0.00%> (-0.03%)`	⬇️
Windows_4	`34.73% <7.69%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/plugins/workspace/public/components/workspace_form/select_data_source_panel.tsx

src/plugins/workspace/public/components/workspace_updater/workspace_updater.tsx

SuZhou-Joe · 2024-06-28T07:37:48Z

src/plugins/workspace/server/integration_tests/routes.test.ts

-          permissions: { invalid_type: { users: ['foo'] } },
+          settings: {
+            permissions: { invalid_type: { users: ['foo'] } },
+            dataSources: [],


Nit: Though workspace is an experimental feature, I still not recommend we do the breaking change in the API side. Maybe next time we can add a parameter.

SuZhou-Joe · 2024-06-28T07:43:34Z

src/plugins/workspace/server/workspace_client.ts

@@ -110,6 +115,12 @@ export class WorkspaceClient implements IWorkspaceClientImpl {
          permissions,
        }
      );
+      if (dataSources) {


Shall we assign dataSources before create the workspace?

Yes we can, also i'm thinking whether we should add a logic that call deleteFromWorkspaces if creation is failed, what do you think?

That would be better, for now I think adjusting the order should be good enough.

SuZhou-Joe · 2024-06-28T07:45:20Z

src/plugins/workspace/server/workspace_client.ts

@@ -192,6 +205,27 @@ export class WorkspaceClient implements IWorkspaceClientImpl {
          throw new Error(DUPLICATE_WORKSPACE_NAME_ERROR);
        }
      }
+
+      if (newDataSources) {


Can we write unit test to cover the diff check code below? It is kind of complex.

Sure, currently there is not any tests for workspace client, maybe we need to do some work to fulfill workspace client test mock.

src/core/server/saved_objects/service/saved_objects_client.ts

Hailong-am · 2024-07-01T07:52:00Z

src/core/server/saved_objects/service/saved_objects_client.ts

+    const object = await this.get<T>(type, id);
+    const existingWorkspaces = object.workspaces ?? [];
+    const mergedWorkspaces = existingWorkspaces.concat(workspaces);
+    const nonDuplicatedWorkspaces = Array.from(new Set(mergedWorkspaces));
+
+    return await this.update<T>(type, id, object.attributes, {
+      workspaces: nonDuplicatedWorkspaces,
+    });
+  };


Overall, it is not an atomic operation, it may have race condition on high concurrency case. I would recommend we do a optimistic lock here by adding version into SavedObjectsUpdateOptions

return await this.update<T>(type, id, object.attributes, { workspaces: nonDuplicatedWorkspaces, version: object.version, });

Hailong-am · 2024-07-01T08:20:49Z

src/plugins/workspace/server/utils.ts

+      perPage: 10000,
+      workspaces,
+    })
+    .then((response) => {


shall we handle exception here? if user don't have permission to list all data sources

In new design only OSD admin will call this function and there will be a new PR to return empty array if user doesn't have permission. So i think we don't need to handle exception here.

src/plugins/workspace/public/components/workspace_form/select_data_source_panel.tsx

src/plugins/workspace/common/types.ts

src/plugins/workspace/public/workspace_client.ts

src/plugins/workspace/server/workspace_client.ts

Signed-off-by: tygao <tygao@amazon.com>

…7101) * feat: add data source assign Signed-off-by: tygao <tygao@amazon.com> * update workspace update Signed-off-by: tygao <tygao@amazon.com> * update workspace update test Signed-off-by: tygao <tygao@amazon.com> * Changeset file for PR #7101 created/updated * update name Signed-off-by: tygao <tygao@amazon.com> * update repository test Signed-off-by: tygao <tygao@amazon.com> * update integration test Signed-off-by: tygao <tygao@amazon.com> * update integration test Signed-off-by: tygao <tygao@amazon.com> * add test cases for utils Signed-off-by: tygao <tygao@amazon.com> * update utils test Signed-off-by: tygao <tygao@amazon.com> * add tests for saved object client Signed-off-by: tygao <tygao@amazon.com> * update and add tests for workspace client Signed-off-by: tygao <tygao@amazon.com> * use overwrite in deleteFromWorkspaces and update find Signed-off-by: tygao <tygao@amazon.com> * update saved ojects client test Signed-off-by: tygao <tygao@amazon.com> * use Promise.all and updates comment and type Signed-off-by: tygao <tygao@amazon.com> * add rescriction for title field to make it optional Signed-off-by: tygao <tygao@amazon.com> * use string array to send selected data sources Signed-off-by: tygao <tygao@amazon.com> --------- Signed-off-by: tygao <tygao@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit 5a4aba9) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…7101) (#7173) * feat: add data source assign * update workspace update * update workspace update test * Changeset file for PR #7101 created/updated * update name * update repository test * update integration test * update integration test * add test cases for utils * update utils test * add tests for saved object client * update and add tests for workspace client * use overwrite in deleteFromWorkspaces and update find * update saved ojects client test * use Promise.all and updates comment and type * add rescriction for title field to make it optional * use string array to send selected data sources --------- (cherry picked from commit 5a4aba9) Signed-off-by: tygao <tygao@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

raintygao requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe, ruanyl, BionIT, xinruiba, zhyuanqi and mengweieric as code owners June 25, 2024 09:19

github-actions bot added the seasoned-contributor label Jun 25, 2024

opensearch-changeset-bot bot added a commit to raintygao/OpenSearch-Dashboards that referenced this pull request Jun 25, 2024

Changeset file for PR opensearch-project#7101 created/updated

38d1abb

SuZhou-Joe reviewed Jun 28, 2024

View reviewed changes

src/plugins/workspace/public/components/workspace_form/select_data_source_panel.tsx Show resolved Hide resolved

SuZhou-Joe reviewed Jun 28, 2024

View reviewed changes

src/plugins/workspace/public/components/workspace_updater/workspace_updater.tsx Show resolved Hide resolved

SuZhou-Joe reviewed Jun 28, 2024

View reviewed changes

Hailong-am reviewed Jul 1, 2024

View reviewed changes

raintygao requested a review from LDrago27 as a code owner July 2, 2024 06:41

ruanyl added the workspace label Jul 2, 2024

ruanyl reviewed Jul 3, 2024

View reviewed changes

raintygao and others added 15 commits July 3, 2024 18:36

feat: add data source assign

b9baf9d

Signed-off-by: tygao <tygao@amazon.com>

update workspace update

0f47199

Signed-off-by: tygao <tygao@amazon.com>

update workspace update test

c42efef

Signed-off-by: tygao <tygao@amazon.com>

Changeset file for PR opensearch-project#7101 created/updated

1a67e01

update name

93c8e52

Signed-off-by: tygao <tygao@amazon.com>

update repository test

bd553e8

Signed-off-by: tygao <tygao@amazon.com>

update integration test

d577505

Signed-off-by: tygao <tygao@amazon.com>

update integration test

722dad8

Signed-off-by: tygao <tygao@amazon.com>

add test cases for utils

04f8209

Signed-off-by: tygao <tygao@amazon.com>

update utils test

8c5ddc3

Signed-off-by: tygao <tygao@amazon.com>

add tests for saved object client

dc67ed5

Signed-off-by: tygao <tygao@amazon.com>

update and add tests for workspace client

c7bdfde

Signed-off-by: tygao <tygao@amazon.com>

use overwrite in deleteFromWorkspaces and update find

d5fc0ad

Signed-off-by: tygao <tygao@amazon.com>

update saved ojects client test

30de7b2

Signed-off-by: tygao <tygao@amazon.com>

use Promise.all and updates comment and type

43fe170

Signed-off-by: tygao <tygao@amazon.com>

raintygao force-pushed the assign branch from cad5559 to 43fe170 Compare July 3, 2024 10:39

ruanyl previously approved these changes Jul 4, 2024

View reviewed changes

add rescriction for title field to make it optional

b393236

Signed-off-by: tygao <tygao@amazon.com>

raintygao dismissed ruanyl’s stale review via b393236 July 4, 2024 06:18

use string array to send selected data sources

5004aee

Signed-off-by: tygao <tygao@amazon.com>

ruanyl approved these changes Jul 4, 2024

View reviewed changes

SuZhou-Joe approved these changes Jul 4, 2024

View reviewed changes

SuZhou-Joe added backport 2.x v2.16.0 labels Jul 4, 2024

SuZhou-Joe merged commit 5a4aba9 into opensearch-project:main Jul 4, 2024
71 checks passed

opensearch-trigger-bot bot mentioned this pull request Jul 4, 2024

[Backport 2.x] [Workspace][Data Source] Support data source assignment in workspace #7173

Merged

AMoo-Miki added the in-next label Sep 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Workspace][Data Source] Support data source assignment in workspace #7101

[Workspace][Data Source] Support data source assignment in workspace #7101

raintygao commented Jun 25, 2024 •

edited

Loading

codecov bot commented Jun 25, 2024 •

edited

Loading

SuZhou-Joe Jun 28, 2024

SuZhou-Joe Jun 28, 2024

raintygao Jun 28, 2024

SuZhou-Joe Jul 1, 2024

SuZhou-Joe Jun 28, 2024

raintygao Jun 28, 2024

Hailong-am Jul 1, 2024

Hailong-am Jul 1, 2024

raintygao Jul 2, 2024

[Workspace][Data Source] Support data source assignment in workspace #7101

[Workspace][Data Source] Support data source assignment in workspace #7101

Conversation

raintygao commented Jun 25, 2024 • edited Loading

Description

Screenshot

Testing the changes

Changelog

Check List

codecov bot commented Jun 25, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raintygao commented Jun 25, 2024 •

edited

Loading

codecov bot commented Jun 25, 2024 •

edited

Loading