Try to fix failing recommendations.test.js and waf-blocking.test.js E2E tests #34796

pypt · 2023-12-28T15:07:23Z

"Jetpack post-connection e2e tests" have been failing since forever:

https://github.com/Automattic/jetpack/actions/workflows/e2e-tests.yml

e.g. here for #34739:

https://github.com/Automattic/jetpack/actions/runs/7348501314/job/20006837157?pr=34739

As per #34796 (review), I've updated XPath selectors that check for the plan in the rendered webpage.

@jboland88, @ArSn, you've last touched this code, could you maybe have a look? I'm not familiar with those tests myself.

Proposed changes:

See above.

Other information:

Have you written new tests for your changes, if applicable?
Have you checked the E2E test CI results, and verified that your changes do not break them?
Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

n/a

Does this pull request change what data or activity we track or use?

No.

Testing instructions:

Make sure the tests succeed.
Make sure what needs to get tested still gets tested.

github-actions · 2023-12-28T15:08:14Z

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

To test on WoA, go to the Plugins menu on a WordPress.com Simple site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin, and enable the modify/remove-withPlan-in-failing-e2e-tests branch.

To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack modify/remove-withPlan-in-failing-e2e-tests

Interested in more tips and information?

In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
Read more about our development workflow here: PCYsg-eg0-p2
Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

github-actions · 2023-12-28T15:08:31Z

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

✅ Include a description of your PR changes.
✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
✅ Add testing instructions.
✅ Specify whether this PR includes any changes to data or privacy.
✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.

Once your PR is ready for review, check one last time that all required checks appearing at the bottom of this PR are passing or skipped.
Then, add the "[Status] Needs Team Review" label and ask someone from your team review the code. Once reviewed, it can then be merged.
If you need an extra review from someone familiar with the codebase, you can update the labels from "[Status] Needs Team Review" to "[Status] Needs Review", and in that case Jetpack Approvers will do a final review of your PR.

Jetpack plugin:

The Jetpack plugin has different release cadences depending on the platform:

WordPress.com Simple releases happen daily.
WoA releases happen weekly.
Releases to self-hosted sites happen monthly. The next release is scheduled for January 9, 2024 (scheduled code freeze on January 8, 2024).

If you have any questions about the release process, please ask in the #jetpack-releases channel on Slack.

andrii-lysenko

Thanks for working on this. These failing tests definitely need to be fixed.

Generally, I don't have strong knowledge on tests, but it seems reasonable to include Plans here. The thing I noticed with these tests though, is that it can't confirm the plan:

It seems that the issue is actually appeared after this change: #34715 . We removed alt name, however, isPlan function actually relies on the plan here:
https://github.com/Automattic/jetpack/blob/trunk/tools/e2e-commons/pages/wp-admin/jetpack-my-plan.js#L13

Maybe we should update isPlan function to correctly check the plan not relying on alt. Something like //h2[contains(@class, "my-plan-card__title") and text()="Jetpack Free"]. What do you think?

Leave as a comment not to be blocking if approved by somebody else

This reverts commit 8e4d8f4.

This reverts commit bc40601.

#34796 (review)

pypt · 2024-01-03T14:29:24Z

Thanks @andrii-lysenko , reverted my hack and applied your suggestion in 06ad8b2.

The first attempt to run one of the tests still fails, but the retry (and thus the whole test) succeeds:

Running 2 tests using 1 worker

Warning: Plugin 'jetpack' is already active.

  ✘  1 specs/post-connection/recommendations.test.js:18:1 › Recommendations (Jetpack Assistant) (4ms)
Warning: Plugin 'jetpack' is already active.

  ✓  2 specs/post-connection/recommendations.test.js:18:1 › Recommendations (Jetpack Assistant) (retry #1) (21.7s)
Warning: Plugin 'jetpack' is already active.

  ✓  3 specs/post-connection/waf-blocking.test.js:27:2 › WAF Blocking › Block a simple request (929ms)


  1) specs/post-connection/recommendations.test.js:18:1 › Recommendations (Jetpack Assistant) ──────

    Error: 'Jetpack Start provisioning failed: -10520

       at ../../../../../tools/e2e-commons/helpers/partner-provisioning.js:38

      36 | 		logger.cli( 'Successful provisioning' );
      37 | 	} else {
    > 38 | 		throw new Error( `'Jetpack Start provisioning failed: ${ json.error }` );
         | 		      ^
      39 | 	}
      40 |
      41 | 	await execWpCommand(

        at provisionJetpackStartConnection (/home/runner/work/jetpack/jetpack/tools/e2e-commons/helpers/partner-provisioning.js:38:9)
        at connect (/home/runner/work/jetpack/jetpack/tools/e2e-commons/env/prerequisites.js:121:3)
        at ensureConnectedState (/home/runner/work/jetpack/jetpack/tools/e2e-commons/env/prerequisites.js:107:3)
        at buildPrerequisites (/home/runner/work/jetpack/jetpack/tools/e2e-commons/env/prerequisites.js:[85](https://github.com/Automattic/jetpack/actions/runs/7398323571/job/20127424825?pr=34796#step:12:86):5)
        at Object.build (/home/runner/work/jetpack/jetpack/tools/e2e-commons/env/prerequisites.js:63:4)
        at /home/runner/work/jetpack/jetpack/projects/plugins/jetpack/tests/e2e/specs/post-connection/recommendations.test.js:8:2

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    output/results/specs-post-connection-recommendations-Recommendations-Jetpack-Assistant-/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

  1 flaky
    specs/post-connection/recommendations.test.js:18:1 › Recommendations (Jetpack Assistant) ───────
  1 passed (4.8m)
Error:   1) specs/post-connection/recommendations.test.js:18:1 › Recommendations (Jetpack Assistant) ──────
    Error: 'Jetpack Start provisioning failed: -10520

       at ../../../../../tools/e2e-commons/helpers/partner-provisioning.js:38

      36 | 		logger.cli( 'Successful provisioning' );
      37 | 	} else {
    > 38 | 		throw new Error( `'Jetpack Start provisioning failed: ${ json.error }` );
         | 		      ^
      39 | 	}
      40 |
      41 | 	await execWpCommand(

        at provisionJetpackStartConnection (/home/runner/work/jetpack/jetpack/tools/e2e-commons/helpers/partner-provisioning.js:38:9)
        at connect (/home/runner/work/jetpack/jetpack/tools/e2e-commons/env/prerequisites.js:121:3)
        at ensureConnectedState (/home/runner/work/jetpack/jetpack/tools/e2e-commons/env/prerequisites.js:[107](https://github.com/Automattic/jetpack/actions/runs/7398323571/job/20127424825?pr=34796#step:12:108):3)
        at buildPrerequisites (/home/runner/work/jetpack/jetpack/tools/e2e-commons/env/prerequisites.js:85:5)
        at Object.build (/home/runner/work/jetpack/jetpack/tools/e2e-commons/env/prerequisites.js:63:4)
        at /home/runner/work/jetpack/jetpack/projects/plugins/jetpack/tests/e2e/specs/post-connection/recommendations.test.js:8:2
Notice:   1 flaky
    specs/post-connection/recommendations.test.js:18:1 › Recommendations (Jetpack Assistant) ───────
  1 passed (4.8m)

I guess we could leave it at that?

…2E tests (#34796) * Test for plans using <h2> value, not alt="" text Automattic/jetpack#34796 (review) * changelog Committed via a GitHub action: https://github.com/Automattic/jetpack/actions/runs/7411554190

pypt added 2 commits December 28, 2023 17:03

Try removing .isPlan(), see what happens

bc40601

changelog

e06e1fc

pypt added [Type] Bug When a feature is broken and / or not performing as intended [Status] In Progress E2E Tests labels Dec 28, 2023

pypt self-assigned this Dec 28, 2023

github-actions bot added [Plugin] Jetpack Issues about the Jetpack plugin. https://wordpress.org/plugins/jetpack/ [Tests] Includes Tests labels Dec 28, 2023

Remove unused imports

8e4d8f4

This comment was marked as outdated.

Sign in to view

pypt requested review from ArSn, jboland88 and a team December 28, 2023 15:48

pypt added [Status] Needs Review To request a review from fellow Jetpack developers. Label will be renamed soon. and removed [Status] In Progress labels Dec 28, 2023

pypt changed the title ~~[WIP] Try to fix failing E2E tests~~ Try to fix failing E2E tests Dec 28, 2023

pypt changed the title ~~Try to fix failing E2E tests~~ Try to fix failing recommendations.test.js and waf-blocking.test.js E2E tests Dec 28, 2023

github-actions bot added the [Status] In Progress label Dec 28, 2023

pypt marked this pull request as ready for review December 28, 2023 15:49

pypt mentioned this pull request Dec 28, 2023

Backup: Add namespace versioning to Helper_Script_Manager and other classes #34739

Merged

3 tasks

andrii-lysenko previously requested changes Dec 29, 2023

View reviewed changes

pypt added 4 commits January 3, 2024 15:43

Merge branch 'trunk' into modify/remove-withPlan-in-failing-e2e-tests

5395f6a

Revert "Remove unused imports"

6ba1128

This reverts commit 8e4d8f4.

Revert "Try removing .isPlan(), see what happens"

6f83ca4

This reverts commit bc40601.

Test for plans using <h2> value, not alt="" text

06ad8b2

#34796 (review)

pypt requested a review from a team January 3, 2024 14:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try to fix failing recommendations.test.js and waf-blocking.test.js E2E tests #34796

Try to fix failing recommendations.test.js and waf-blocking.test.js E2E tests #34796

pypt commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Dec 28, 2023

This comment was marked as outdated.

andrii-lysenko left a comment •

edited

Loading

pypt commented Jan 3, 2024

Try to fix failing recommendations.test.js and waf-blocking.test.js E2E tests #34796

Try to fix failing recommendations.test.js and waf-blocking.test.js E2E tests #34796

Conversation

pypt commented Dec 28, 2023 • edited Loading

Proposed changes:

Other information:

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

github-actions bot commented Dec 28, 2023 • edited Loading

github-actions bot commented Dec 28, 2023

This comment was marked as outdated.

andrii-lysenko left a comment • edited Loading

Choose a reason for hiding this comment

pypt commented Jan 3, 2024

pypt commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Dec 28, 2023 •

edited

Loading

andrii-lysenko left a comment •

edited

Loading