Visual testing - Adds Sanity Test 4 Suite #5501

geriux · 2023-02-23T12:03:49Z

Related PRs:

This PR adds the first steps for the Sanity Test Suite 4:

Spacer block

Spacer block - Spacer in horizontal layout works as expected

Buttons block

Render custom text and background color
Render gradient background color
Check if selection/caret color matches font color
Edit text styles
Link from the clipboard is presented as an option in the link picker

It also updates the takeScreenshot util to take into account if the device is in landscape mode, if you want to avoid including the keyboard in the screenshot, or if you want to pass a custom height value for the screenshot to be cropped.

After several test failures, I realized that including the keyboard will generate false positives, as the keyboard could look different in our local environments, like having other languages, and the emoji keyboard. I also encountered issues where the Keyboard would have a totally different appearance on CI.

Finally, it changes the comparisonMethod to compare the screenshots to ssim. You can find a nice explanation on the docummentation, as the pixel by pixel comparison wasn't working right.

Note: Since this PR updates how it takes the screenshots, I'm updating all visual snapshots.

To test CI checks should pass on this PR.

PR submission checklist:

I have considered adding unit tests where possible.
I have considered if this change warrants user-facing release notes more info and have added them to RELEASE-NOTES.txt if necessary.

peril-wordpress-mobile · 2023-03-01T14:19:44Z

Wanna run full suite of Android and iOS UI tests? Click here and 'Approve' CI job!

dcalhoun

Great work! 👏🏻

All of my comments are merely thoughts, not blockers by any means.

Also, I noticed the title selection changed in __device-tests__/image-snapshots/gutenberg-editor-group-visual-test-js-gutenberg-editor-visual-test-for-group-block-should-show-the-empty-placeholder-for-the-unselected-state-1-android.png. Was that intended?

__device-tests__/utils.js

__device-tests__/gutenberg-editor-sanity-test-4-visual.test.js

dcalhoun · 2023-03-03T17:43:53Z

__device-tests__/gutenberg-editor-sanity-test-4-visual.test.js

+			// Wait for the modal to open
+			await editorPage.driver.sleep( 3000 );


Not necessarily required right now — maybe it is premature — but I wonder if we should abstract the modal animation times so that it is (1) clear what is occurring and (2) could be modified across tests as necessary.

const MODAL_OPEN_DURATION = 3000; const MODAL_CLOSE_DURATION = 2000;

Good idea, maybe even having a waitForModalToOpen / waitForModalToClose utils. I can create a follow-up PR to introduce either that or just the constants.

While I still think naming the sleep durations may help communicate intent, I may "walk back" on my suggestion after spending time in the e2e tests yesterday. My hot take is that we have a lot of test helpers, to the quantity that I find it difficult to comprehend each of them and their intent. I share that to say I now have reservations in regards to expanding test helpers further.

As a tangential thought to consider outside the scope of this PR, I wonder if we would benefit from evaluating if there any helpers we might should remove and replace with composing other existing, lower-level helpers. My perception is that we have some helpers that are very specific and niche. Obviously, a subject to discuss in a different place. Just sharing my thoughts.

My hot take is that we have a lot of test helpers, to the quantity that I find it difficult to comprehend each of them and their intent.

I share the same feeling 😅

I wonder if we would benefit from evaluating if there any helpers we might should remove and replace with composing other existing, lower-level helpers. My perception is that we have some helpers that are very specific and niche. Obviously, a subject to discuss in a different place.

I agree, I think it's good you shared this, working on adding new tests always brings new perspectives and all the feedback will improve our current workflows and the way we do things.

geriux · 2023-03-07T11:28:19Z

Great work! 👏🏻

All of my comments are merely thoughts, not blockers by any means.

Thanks for reviewing!

Also, I noticed the title selection changed in __device-tests__/image-snapshots/gutenberg-editor-group-visual-test-js-gutenberg-editor-visual-test-for-group-block-should-show-the-empty-placeholder-for-the-unselected-state-1-android.png. Was that intended?

Yes! I actually realized we needed to update that screenshot when I changed the comparison method, now it matches the iOS screenshot.

dcalhoun

Happy to have this coverage. Thank you!

dcalhoun · 2023-03-07T16:01:37Z

__device-tests__/gutenberg-editor-sanity-test-4-visual.test.js

+			await editorPage.removeBlockAtPosition( blockNames.buttons );
+		} );
+
+		it( 'Link from the clipboard is presented as an option in the link picker', async () => {


My perception is that all tests in this PR besides this one are very visual and likely benefit from visual regression testing snapshots. However, it seems we could likely test the spirit of this link picker test with a (possibly faster or simpler) integration test.

This is more a question regarding our overarching approach to testing, not necessarily directly related to the work of this PR. Not looking to block this PR, but seeking your thoughts. WDYT?

For this case, since we are testing copying a link from the clipboard and expecting it to show in the link picker it is very attached to the device imagine something fails like a new OS permission is added and the link can't be pasted, and having a visual test of the whole workflow would allow us not to have to test this manually.

Fair point. Avoiding a clipboard API mock provides additional coverage (e.g. OS permissions). There a few approaches to testing the spirit of this test case we could consider:

Capture visual images within an e2e tests.

Assert element presence within an e2e tests.

Mock the clipboard API and assert element presence within an integration tests.

It seems you prefer the first two for this specific test case, which I think is reasonable.

Additionally, I do not know if there is a meaningful "cost" between the first two. The first may "cost" more in regards to test speed or repository size for stored images. While the test case description doesn't necessarily require to visual testing (we could just assert the element presence), it is additional coverage. I suppose it is a judgement call as to whether the visual coverage is worth it.

…entation

…eyboard in the screenshots as well as passing a custom height value

geriux · 2023-03-09T16:00:32Z

Happy to have this coverage. Thank you!

Thank you for the review! 🚀

geriux added the Testing Anything related to automated tests label Feb 23, 2023

Base automatically changed from add-visual-tests-support to trunk February 23, 2023 18:41

geriux force-pushed the add-more-visual-tests branch from 70ed290 to 699d17e Compare March 1, 2023 14:19

geriux changed the title ~~Visual testing - Adds Sanity Test 4 suite~~ Visual testing - Adds Sanity Test 4 Suite Mar 1, 2023

geriux marked this pull request as ready for review March 3, 2023 13:00

dcalhoun self-requested a review March 3, 2023 13:09

geriux mentioned this pull request Mar 3, 2023

[Mobile] - E2E tests - Add new helpers WordPress/gutenberg#48358

Merged

dcalhoun reviewed Mar 3, 2023

View reviewed changes

dcalhoun approved these changes Mar 7, 2023

View reviewed changes

Gerardo added 12 commits March 8, 2023 15:58

E2E Utils - Update takeScreenshot to take into account the device ori…

5e6fb00

…entation

Visual testing - Add Sanity test suite 4

8223c60

Update Buttons screenshot to update keyboard language

46f8cdc

Update Screenshot

fe6a316

Update comparisonMethod to ssim

22f8030

Visual testing - Update Group test

a97cca6

Visual test - Sanity test 4 - Increase threshold

333b36b

E2E Utils - Update takeScreenshot util to support not including the k…

1e0d4b0

…eyboard in the screenshots as well as passing a custom height value

Update visual snapshots

cf5646d

Rename selectTextFromTextInput to selectTextFromElement

90ea540

E2E - takeScreenshot - Rename customHeight to heightPercentage

35ae380

Update Gutenberg ref

11b7fb9

geriux force-pushed the add-more-visual-tests branch from 201bd5f to 11b7fb9 Compare March 8, 2023 15:02

Gerardo added 2 commits March 9, 2023 10:32

Update Gutenberg ref

5f96fd2

Merge branch 'trunk' into add-more-visual-tests

6e38e45

geriux merged commit f1cbfe7 into trunk Mar 9, 2023

geriux deleted the add-more-visual-tests branch March 9, 2023 16:07

geriux mentioned this pull request May 30, 2023

Update tests to remove some in favor of automated testing wordpress-mobile/test-cases#142

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Visual testing - Adds Sanity Test 4 Suite #5501

Visual testing - Adds Sanity Test 4 Suite #5501

geriux commented Feb 23, 2023 •

edited

Loading

peril-wordpress-mobile bot commented Mar 1, 2023 •

edited

Loading

dcalhoun left a comment

dcalhoun Mar 3, 2023

geriux Mar 7, 2023

dcalhoun Mar 7, 2023

geriux Mar 9, 2023

geriux commented Mar 7, 2023

dcalhoun left a comment

dcalhoun Mar 7, 2023

geriux Mar 9, 2023

dcalhoun Mar 9, 2023

geriux commented Mar 9, 2023

		// Wait for the modal to open
		await editorPage.driver.sleep( 3000 );

Visual testing - Adds Sanity Test 4 Suite #5501

Visual testing - Adds Sanity Test 4 Suite #5501

Conversation

geriux commented Feb 23, 2023 • edited Loading

peril-wordpress-mobile bot commented Mar 1, 2023 • edited Loading

dcalhoun left a comment

Choose a reason for hiding this comment

dcalhoun Mar 3, 2023

Choose a reason for hiding this comment

geriux Mar 7, 2023

Choose a reason for hiding this comment

dcalhoun Mar 7, 2023

Choose a reason for hiding this comment

geriux Mar 9, 2023

Choose a reason for hiding this comment

geriux commented Mar 7, 2023

dcalhoun left a comment

Choose a reason for hiding this comment

dcalhoun Mar 7, 2023

Choose a reason for hiding this comment

geriux Mar 9, 2023

Choose a reason for hiding this comment

dcalhoun Mar 9, 2023

Choose a reason for hiding this comment

geriux commented Mar 9, 2023

geriux commented Feb 23, 2023 •

edited

Loading

peril-wordpress-mobile bot commented Mar 1, 2023 •

edited

Loading