Add query enhancement cypress tests to Github CI #8703

abbyhu2000 · 2024-10-25T06:23:02Z

Description

Add query enhancement cypress tests workflow to github CI to test discover 2.0 functionalities:

sql, ppl queries
dataset selector
...

The tests can be added under folder cypress/integration/core-opensearch-dashboards/opensearch-dashboards/apps/query_enhancement in the functional repo, and will be run in ciGroup10 in the CI.

Issues Resolved

Screenshot

Testing the changes

Changelog

feat: Add query enhancement cypress tests to Github CI

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

github-actions · 2024-10-25T06:23:32Z

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

abbyhu2000 · 2024-10-25T06:24:45Z

package.json

@@ -85,8 +85,7 @@
    "release_note:generate": "scripts/use_node scripts/generate_release_note",
    "cypress:run-without-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=false",
    "cypress:run-with-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=true,openSearchUrl=https://localhost:9200,WAIT_FOR_LOADER_BUFFER_MS=500",
-    "osd:ciGroup10": "echo \"dashboard_sanity_test_spec.js\"",
-    "osd:ciGroup11": "echo \"apps/vis_builder/*.js\"",


removed "osd:ciGroup11": "echo \"apps/vis_builder/*.js\"", because vis_builder tests are already running in ciGroup1 as defined here https://github.com/opensearch-project/opensearch-dashboards-functional-test/blob/main/package.json#L28

codecov · 2024-10-25T06:43:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.81%. Comparing base (9a25d0d) to head (80b8eaf).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8703      +/-   ##
==========================================
- Coverage   60.84%   60.81%   -0.03%     
==========================================
  Files        3793     3793              
  Lines       90486    90486              
  Branches    14212    14212              
==========================================
- Hits        55057    55031      -26     
- Misses      31939    32009      +70     
+ Partials     3490     3446      -44

Flag	Coverage Δ
Linux_1	`29.07% <ø> (ø)`
Linux_2	`?`
Linux_3	`37.66% <ø> (-0.01%)`	⬇️
Linux_4	`29.84% <ø> (ø)`
Windows_1	`29.08% <ø> (ø)`
Windows_2	`56.35% <ø> (ø)`
Windows_3	`37.67% <ø> (ø)`
Windows_4	`29.84% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-10-25T08:30:23Z

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

abbyhu2000 · 2024-10-25T18:23:08Z

Cypress runner:
https://github.com/opensearch-project/OpenSearch-Dashboards/actions/runs/11515231239/job/32057142616

CiGroup10 is testing query enhancement tests that are added in this functional repo PR: opensearch-project/opensearch-dashboards-functional-test#1598

All passing except for CiGroup 6, i think currently ciGroup6 is failing on main, should not be relate to this PR.

ananzh · 2024-10-25T20:39:10Z

.github/workflows/cypress_workflow.yml


 jobs:
  cypress-tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
-          group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+        include:


Will this work and help organize?

matrix: config: [standard, query_enhanced, dashboard] group: [1, 2, 3, 4, 5, 6, 7, 8, 9] include: - config: query_enhanced group: 10 test_location: ftr - config: dashboard group: 11 test_location: source exclude: - config: query_enhanced group: [1,2,3,4,5,6,7,8,9] - config: dashboard group: [1,2,3,4,5,6,7,8,9]

+1, using matrix to create the right config could make it shorter. but i do see a benefit of making it this granular as it we have more control. as it leave spaces in the future where we migrate the tests from ftr to this repo

ananzh · 2024-10-25T20:43:29Z

QQ: so for writing tests in ciGroup10, do we need add some code to create a workspace? and test in the workspace?

abbyhu2000 · 2024-10-26T00:11:26Z

QQ: so for writing tests in ciGroup10, do we need add some code to create a workspace? and test in the workspace?

i

QQ: so for writing tests in ciGroup10, do we need add some code to create a workspace? and test in the workspace?

I think i asked @ashwin-pc during standup and he responded that right now we only covers the flows on discover pages.

Do we want to add test to include extra workflows like creating the workspace? or we only test what discover page covers.

kavilla · 2024-10-26T07:00:46Z

.github/workflows/cypress_workflow.yml

+            test_location: ftr
+          - group: 3
+            config: standard
+            test_location: ftr


we have a variable FTR_PATH, should we use this/change FTR_PATH to something that makes more sense to usage and add the other locations? or should we get rid of the FTR_PATH variable

kavilla · 2024-10-26T07:02:52Z

.github/workflows/cypress_workflow.yml

@@ -42,18 +40,65 @@ env:
  COMMENT_TAG: '[MANUAL CYPRESS TEST RUN RESULTS]'
  COMMENT_SUCCESS_MSG: ':white_check_mark: Cypress test run succeeded!'
  COMMENT_FAILURE_MSG: ':x: Cypress test run failed!'
+  LATEST_VERSION: '2.17.0'


maybe in an inline comment on why we are using this. i believe it was because of issues trying to use main and install plugins for github virtual env

one thing we could also do is actually use to make this dynamic is rely on git tags. every release infra will cut a tag for the release in all the repos post release and then we use that to make a github release. we could potentially just get the highest version tag from our repo and assume that is the highest release version of OpenSearch that we can download

kavilla · 2024-10-26T07:18:24Z

.github/workflows/cypress_workflow.yml

-      - name: Setup spec files for Dashboards tests
-        if: ${{ inputs.specs == '' && matrix.group > 9 }}
+      # Run tests based on configuration
+      - name: Run FT repo tests with no query enhancements


nit: historically we called this FTRepo.

and I think the specification with no query enhancements might be unnecessary. we should go with the actual one run the query enhancements to say with query enhancements incase we want to follow this format in the future and add different type of situations.

kavilla · 2024-10-26T07:22:21Z

.github/workflows/cypress_workflow.yml

+      # Run tests based on configuration
+      - name: Run FT repo tests with no query enhancements
+        if: matrix.test_location == 'ftr' && matrix.config == 'standard'
+        uses: cypress-io/github-action@v2


did we want to bump the cypress github action if available and works? might have improvements https://github.com/cypress-io/github-action/blob/master/CHANGELOG.md. but i see somethings in our file will need to change so maybe not worht it at this time

kavilla · 2024-10-26T07:23:07Z

.github/workflows/cypress_workflow.yml

+      - name: Extract OpenSearch
+        if: matrix.test_location == 'ftr' && matrix.config == 'query_enhanced'
+        run: |
+          tar -xzf opensearch-*.tar.gz


we should be able to untar this file and pass the param to call it a specific name so line 238 doesn't have to be a specific version

kavilla · 2024-10-26T07:26:56Z

.github/workflows/cypress_workflow.yml

+        if: matrix.test_location == 'ftr' && matrix.config == 'query_enhanced'
+        run: |
+          /bin/bash -c "./opensearch-2.17.0/opensearch-tar-install.sh &"
+          sleep 30


confused about this one.

wait-on should be actually a better resolution to this: https://github.com/cypress-io/github-action?tab=readme-ov-file#wait-on

are you seeing issues with that?

kavilla · 2024-10-26T07:34:52Z

.github/workflows/cypress_workflow.yml

    container:
      image: docker://opensearchstaging/ci-runner:ci-runner-rockylinux8-opensearch-dashboards-integtest-v2
      options: --user 1001
    env:
+      START_CMD: ${{ matrix.config == 'query_enhanced' &&


i really like this concept of config because it actually follows the format well with how OSD functional tests are structured!

only nit I have here is that this actually is a little bit hard to read. and any fixes or additional updates to this will require us to know to update both places.

so i think minimum we should consider the logic being where we append extra configurations if the matrix config is query enhanced instead of having two.

if that is not good enough then it might be time to start sourcing a .env config file that stores this information so that we can modify this stuff and make this file more approachable.

kavilla · 2024-10-26T07:36:45Z

.github/workflows/cypress_workflow.yml

+            config: standard
+            test_location: ftr
+          - group: 4
+            config: standard


nit: default has been used for this type of situation or undefined and we automatically apply the default config

kavilla · 2024-10-26T07:40:13Z

.github/workflows/cypress_workflow.yml

+          IFS="," read -a SPEC_ARRAY <<< $(yarn --silent osd:ciGroup${{ matrix.group }})
+          DASHBOARDS_SPEC=''
+          for i in "${SPEC_ARRAY[@]}"; do
+          DASHBOARDS_SPEC+="cypress/integration/core_opensearch_dashboards/${i},"


we desperately need to be able to tag cypress test so that when we write them inhouse the test can decide which workflow it will run in

kavilla

i really like the proposal. i mentioned this in the comments moving towards the config route is actually more in line with how OSD functional tests are so the system is better built for that. So nice call!

Some comments though related to future proving the solution

abbyhu2000 added 16 commits October 23, 2024 23:59

Add cypress ciGroup11 to github workflow for discover 2.0 tests

adb3dfc

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

modify matrix logic

04fb022

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

fix start command

0a61085

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

more test

5b93291

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

switch to another way to install sql plugin

5b37d74

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

install wget

4f29a0a

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

try 1

93f0123

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

add job scheduler

3ee8452

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

use opensearch bundle link

4f19024

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

get rid of verifying

1d1b334

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

use 2.17.0

0d48f93

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

Try another way to run opensearch

76cfb80

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

correct conditions

a39b324

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

dis-enable workspace

2688de8

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

use test location instead of matrix number

29e1c41

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

remove ciGroup12 in package.json

80b8eaf

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

abbyhu2000 requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe and ruanyl as code owners October 25, 2024 06:23

abbyhu2000 requested review from mengweieric, LDrago27, virajsanghvi, sejli, joshuali925 and huyaboo as code owners October 25, 2024 06:23

github-actions bot added the distinguished-contributor label Oct 25, 2024

github-actions bot added the failed changeset label Oct 25, 2024

abbyhu2000 commented Oct 25, 2024

View reviewed changes

abbyhu2000 mentioned this pull request Oct 25, 2024

Add GitHub workflow cypress #8684

Closed

7 tasks

ananzh reviewed Oct 25, 2024

View reviewed changes

kavilla reviewed Oct 26, 2024

View reviewed changes

kavilla added ci operational excellence backport 2.x labels Oct 26, 2024

kavilla reviewed Oct 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add query enhancement cypress tests to Github CI #8703

Add query enhancement cypress tests to Github CI #8703

abbyhu2000 commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024

abbyhu2000 Oct 25, 2024 •

edited

Loading

codecov bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024

abbyhu2000 commented Oct 25, 2024

ananzh Oct 25, 2024

kavilla Oct 26, 2024

ananzh commented Oct 25, 2024

abbyhu2000 commented Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024

kavilla left a comment

Add query enhancement cypress tests to Github CI #8703

Are you sure you want to change the base?

Add query enhancement cypress tests to Github CI #8703

Conversation

abbyhu2000 commented Oct 25, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Changelog

Check List

github-actions bot commented Oct 25, 2024

❌ Invalid Prefix For Manual Changeset Creation

abbyhu2000 Oct 25, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Oct 25, 2024 • edited Loading

Codecov Report

github-actions bot commented Oct 25, 2024

❌ Invalid Prefix For Manual Changeset Creation

abbyhu2000 commented Oct 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ananzh commented Oct 25, 2024

abbyhu2000 commented Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kavilla Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kavilla Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kavilla left a comment

Choose a reason for hiding this comment

abbyhu2000 commented Oct 25, 2024 •

edited

Loading

abbyhu2000 Oct 25, 2024 •

edited

Loading

codecov bot commented Oct 25, 2024 •

edited

Loading

abbyhu2000 commented Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024 •

edited

Loading