Setup S3 connection with integrations #1057

Swiddis · 2023-09-29T23:12:50Z

Description

Followup to #1052, this PR adds actual S3 integration functionality.

Work in progress -- Main querying pipeline is good, but need to do some work for the UI.

Issues Resolved

N/A

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

This test is unneeded after 12c4bcf Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

…atalog

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Let it be known at on this day, with this commit, I have truly grokked why we don't use `any` in typescript. Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

...c/components/integrations/components/__tests__/__snapshots__/setup_integration.test.tsx.snap

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

public/components/integrations/components/setup_integration.tsx

kavithacm · 2023-10-04T17:15:51Z

public/components/integrations/components/setup_integration.tsx

+                  if (!result.ok) {
+                    console.error('Query failed', result.error);
+                    setLoading(false);
+                    setToast('Something went wrong.', 'danger');


can we change this toast to more appropriate wording?

What's more appropriate wording? We don't have any error information in the response that we can use

derek-ho · 2023-10-04T17:18:40Z

...c/components/integrations/components/__tests__/__snapshots__/setup_integration.test.tsx.snap

                          "name": "sample",
-                          "type": "logs",
-                          "version": "2.0.0",
+                          "type": "",


Why are you removing type here

Stopped filling a default value when the integration isn't loaded yet -- We only use the type when we go to integration adding so it's safe to keep empty until it's loaded.

public/components/integrations/components/setup_integration.tsx

derek-ho · 2023-10-04T17:20:56Z

public/components/integrations/components/setup_integration.tsx

+        },
+      })
+    ).queryId;
+    while (true) {


Is there better way than while true?

Not sure what kind of polling timeout to put on it, I've heard some queries can take up to 10 mins and don't have any reference for when I should just tell the user that it isn't working

derek-ho

Approving to unblock, left a few comments

derek-ho · 2023-10-04T17:21:27Z

@Swiddis can you fix DCO

public/components/integrations/components/setup_integration.tsx

joshuali925 · 2023-10-04T17:13:54Z

public/components/integrations/components/setup_integration.tsx

+    }
+  } catch (err: any) {
+    console.error(err);
+    return { ok: false, error: err };


i think this works but usually it's error, loading, data. If loading is true then display loading, else if error is not null then display error, otherwise data is useable. for example like https://swr.vercel.app/#overview

this way you don't need to return both ok and error, and loading and sleeping should be distinguished in case UI needs them

public/components/integrations/components/setup_integration.tsx

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis · 2023-10-04T18:10:40Z

Dealt with small-scale PR feedback -- More large-scale changes will need to happen later. Just waiting on tests before merging.

* Update mocha (#960) * added code for the change of design of the sidebar, fixed overriding … (#928) * Refactor add_integration_flyout to break out nested methods (#956) * [Log Explorer] Remove top level tabs (#970) * Explorer's timeheader & color change (#959) * Update caching (#965) * Move query_utils to /public (#983) * Remove Zeppelin Parser from Notebook (#985) * Explorer empty bucket (#990) * refactored classnames for sidebar (#933) * Refactor Integrations Backend to abstract IO (#947) * Create React component for Integrations Setup UI (#1009) * Create Data sources plugin with Manage Datasources Flow (#1035) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * adjust explorer chart color and spacing (#1051) * Configure S3 datasource flow (#1049) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * modified explorer data grid to follow discover look and feel (#1041) * Create prometheus datasource flow (#1054) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * added design changes for sidebar (#1061) * updated snapshots (#1062) * Update UI for integrations setup (#1052) * Data sources UI improvements (#1059) * adding redirection to datasources (#1063) * Move convertDateTime, refactor query_utils (#1064) * Setup S3 connection with integrations (#1057) * Use approved svg from UX (#1066) * Fix missing import 'moment' on query_utils. (#1067) --------- Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: sumukhswamy <sumukhhs@amazon.com> Signed-off-by: Eric Wei <menwe@amazon.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Signed-off-by: Peter Fitzgibbons <peter.fitzgibbons@gmail.com> Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Signed-off-by: Simeon Widdis <sawiddis@gmail.com> Co-authored-by: sumukhswamy <sumukhhs@amazon.com> Co-authored-by: Eric Wei <menwe@amazon.com> Co-authored-by: Paul Sebastian <paul_sebastian@live.com> Co-authored-by: Peter Fitzgibbons <peter.fitzgibbons@gmail.com> Co-authored-by: Paul Sebastian <paulstn@amazon.com> Co-authored-by: Shenoy Pratik <sgguruda@amazon.com> Co-authored-by: Simeon Widdis <sawiddis@amazon.com>

* Working on review comments Signed-off-by: ruchika-narang <ruchika_narang@persistent.com> * Replacing series and dimension with dynamic naming Signed-off-by: ruchika-narang <ruchika_narang@persistent.com> * Revert code for histogram and treemap after antlr merge because of crashes in app Signed-off-by: ruchika-narang <ruchika_narang@persistent.com> * Working on review comments, renaming metrics to series Signed-off-by: ruchika-narang <ruchika_narang@persistent.com> * Removing extra import after rebasing main Signed-off-by: ruchika-narang <ruchika_narang@persistent.com> * Working on review comments Signed-off-by: ruchika-narang <ruchika_narang@persistent.com> * Replacing series and dimension with dynamic naming Signed-off-by: ruchika-narang <ruchika_narang@persistent.com> * Fixed logs view visualizations issues. Signed-off-by: Saisanju Sreevalsakumar <saisanju_s@persistent.com> * Resolved review comments for log view fixes 1. Signed-off-by: Saisanju Sreevalsakumar <saisanju_s@persistent.com> Signed-off-by: ruchika-narang <ruchika_narang@persistent.com> Signed-off-by: Saisanju Sreevalsakumar <saisanju_s@persistent.com> Co-authored-by: ruchika-narang <ruchika_narang@persistent.com>

* Update mocha (opensearch-project#960) * added code for the change of design of the sidebar, fixed overriding … (opensearch-project#928) * Refactor add_integration_flyout to break out nested methods (opensearch-project#956) * [Log Explorer] Remove top level tabs (opensearch-project#970) * Explorer's timeheader & color change (opensearch-project#959) * Update caching (opensearch-project#965) * Move query_utils to /public (opensearch-project#983) * Remove Zeppelin Parser from Notebook (opensearch-project#985) * Explorer empty bucket (opensearch-project#990) * refactored classnames for sidebar (opensearch-project#933) * Refactor Integrations Backend to abstract IO (opensearch-project#947) * Create React component for Integrations Setup UI (opensearch-project#1009) * Create Data sources plugin with Manage Datasources Flow (opensearch-project#1035) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * adjust explorer chart color and spacing (opensearch-project#1051) * Configure S3 datasource flow (opensearch-project#1049) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * modified explorer data grid to follow discover look and feel (opensearch-project#1041) * Create prometheus datasource flow (opensearch-project#1054) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * added design changes for sidebar (opensearch-project#1061) * updated snapshots (opensearch-project#1062) * Update UI for integrations setup (opensearch-project#1052) * Data sources UI improvements (opensearch-project#1059) * adding redirection to datasources (opensearch-project#1063) * Move convertDateTime, refactor query_utils (opensearch-project#1064) * Setup S3 connection with integrations (opensearch-project#1057) * Use approved svg from UX (opensearch-project#1066) * Fix missing import 'moment' on query_utils. (opensearch-project#1067) --------- Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: sumukhswamy <sumukhhs@amazon.com> Signed-off-by: Eric Wei <menwe@amazon.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Signed-off-by: Peter Fitzgibbons <peter.fitzgibbons@gmail.com> Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Signed-off-by: Simeon Widdis <sawiddis@gmail.com> Co-authored-by: sumukhswamy <sumukhhs@amazon.com> Co-authored-by: Eric Wei <menwe@amazon.com> Co-authored-by: Paul Sebastian <paul_sebastian@live.com> Co-authored-by: Peter Fitzgibbons <peter.fitzgibbons@gmail.com> Co-authored-by: Paul Sebastian <paulstn@amazon.com> Co-authored-by: Shenoy Pratik <sgguruda@amazon.com> Co-authored-by: Simeon Widdis <sawiddis@amazon.com> (cherry picked from commit 171acde)

Swiddis added 30 commits August 17, 2023 15:41

Stub catalog reader interface

c5cea8d

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add basic catalog functionality to new catalog reader

77ad89b

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Refactor validation logic with a deeper interface

676591d

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Refactor validation logic with a deeper interface

12c4bcf

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Remove redundant test.

412480e

This test is unneeded after 12c4bcf Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add tests for new validators

6d2bad1

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Merge branch 'better-validation' into prototype-network-catalog

c3dd111

Make better failure mode for invalid objects

55d6490

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Merge branch 'main' into prototype-network-catalog

c62d28b

Generalize Result type

ba56fb8

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Convert backend to use catalog reader (unstable)

cb29208

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Repair tests for integrations class (unstable)

c9de36e

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Refactor repository for new integration interface

37c7980

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Fix outer repository and backend tests

b331090

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add tests for sample data

4b29e17

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Merge remote-tracking branch 'upstream/main' into prototype-network-c…

b35dfa8

…atalog

Add CatalogReader JavaDocs

1bef29e

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Repair integrations builder

a18cf47

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Remove extra commented test

047b14a

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Remove unnecessary log statement

e65fe8b

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Repair getSchemas behavior to return correct type

00ddb5b

Let it be known at on this day, with this commit, I have truly grokked why we don't use `any` in typescript. Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add tests for getSchemas

8af7f4a

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add tests for asset and sample data backend methods

a31963d

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Break flyout validation methods out of constructing method

feeaef5

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add tests for extracted flyout methods

b644d11

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Switch validation method to use ValidationResult

614613a

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Merge branch 'prototype-network-catalog' into feature/integrations-setup

d97b931

Merge branch 'refactor-integ-frontend' into feature/integrations-setup

c8b5eff

Merge branch 'main' into prototype-network-catalog

af78c27

Swap out flyout for hello-world setup page

ab30ed0

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis requested review from rupal-bq, mengweieric, vamsi-amazon, penghuo, seankao-az and anirudha as code owners October 3, 2023 22:05

derek-ho reviewed Oct 3, 2023

View reviewed changes

...c/components/integrations/components/__tests__/__snapshots__/setup_integration.test.tsx.snap Outdated Show resolved Hide resolved

Fix faulty test

d89c004

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

YANG-DB approved these changes Oct 4, 2023

View reviewed changes

kavithacm reviewed Oct 4, 2023

View reviewed changes

public/components/integrations/components/setup_integration.tsx Outdated Show resolved Hide resolved

kavithacm reviewed Oct 4, 2023

View reviewed changes

derek-ho reviewed Oct 4, 2023

View reviewed changes

kavithacm reviewed Oct 4, 2023

View reviewed changes

public/components/integrations/components/setup_integration.tsx Outdated Show resolved Hide resolved

derek-ho reviewed Oct 4, 2023

View reviewed changes

public/components/integrations/components/setup_integration.tsx Outdated Show resolved Hide resolved

derek-ho reviewed Oct 4, 2023

View reviewed changes

derek-ho approved these changes Oct 4, 2023

View reviewed changes

joshuali925 approved these changes Oct 4, 2023

View reviewed changes

Swiddis added 4 commits October 4, 2023 10:27

Remove debug logging

97c02b2

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Remove redundant copy complexity

aaeb16d

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add assets to default state

d1394a6

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Fix progress updaters

d70b890

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis merged commit eee56aa into opensearch-project:main Oct 4, 2023
8 of 9 checks passed

Swiddis deleted the setup-s3 branch October 4, 2023 18:28

derek-ho pushed a commit to derek-ho/dashboards-observability that referenced this pull request Oct 4, 2023

Setup S3 connection with integrations (opensearch-project#1057)

4008c02

mengweieric pushed a commit to mengweieric/dashboards-observability that referenced this pull request Oct 15, 2023

Setup S3 connection with integrations (opensearch-project#1057)

e5b07ed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup S3 connection with integrations #1057

Setup S3 connection with integrations #1057

Swiddis commented Sep 29, 2023

kavithacm Oct 4, 2023

derek-ho Oct 4, 2023

Swiddis Oct 4, 2023

derek-ho Oct 4, 2023

Swiddis Oct 4, 2023

derek-ho Oct 4, 2023

Swiddis Oct 4, 2023

derek-ho left a comment

derek-ho commented Oct 4, 2023

joshuali925 Oct 4, 2023 •

edited

Loading

Swiddis commented Oct 4, 2023

Setup S3 connection with integrations #1057

Setup S3 connection with integrations #1057

Conversation

Swiddis commented Sep 29, 2023

Description

Issues Resolved

Check List

kavithacm Oct 4, 2023

Choose a reason for hiding this comment

derek-ho Oct 4, 2023

Choose a reason for hiding this comment

Swiddis Oct 4, 2023

Choose a reason for hiding this comment

derek-ho Oct 4, 2023

Choose a reason for hiding this comment

Swiddis Oct 4, 2023

Choose a reason for hiding this comment

derek-ho Oct 4, 2023

Choose a reason for hiding this comment

Swiddis Oct 4, 2023

Choose a reason for hiding this comment

derek-ho left a comment

Choose a reason for hiding this comment

derek-ho commented Oct 4, 2023

joshuali925 Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

Swiddis commented Oct 4, 2023

joshuali925 Oct 4, 2023 •

edited

Loading