element.visible is returning false for visible elements

[JPrymek-Change]

What is your Scenario?

After upgrading TestCafe, we found an issue with assertions for the visibility of some elements failing, despite the elements being visible on the page.

I've reproduced the test on TestCafe versions 3.5.0, 3.6.0, 3.6.1, and 3.6.2 and found that the visibility checks were working up to v3.6.0, but started failing in 3.6.1. Results are consistent with native automation mode enabled and disabled.

What is the Current behavior?

In TestCafe versions up to 3.6.0, assertions on element.visible were passing when the elements were visible on the page.

What is the Expected behavior?

The value for element.visible should be consistent with previous versions of TestCafe.

What is the public URL of the test page? (attach your complete example)

https://www.change.org/browse

What is your TestCafe test code?

import { Selector } from 'testcafe';

fixture('Petition card visibility')
.page('https://www.change.org');

test('Browse page shows petition cards', async t => {
await t.navigateTo('/browse');

const petitionCards = Selector('[data-qa="petition-list-card"]');
await t.expect(petitionCards.count).gt(0);
const firstCard = petitionCards.nth(0);
await t.expect(firstCard.visible).ok();
});

Your complete configuration file

{
"browsers": ["chrome:emulation:device=iPhone X --headless=new"],
"src": "./tests/*",

"skipJsErrors": true,
"disablePageCaching": true,

"disableNativeAutomation": true,

"reporter": [
{
"name": "xunit",
"output": "artifacts/test-results/results.xml"
},
{
"name": "spec"
}
],

"screenshots": {
"fullPage": true,
"takeOnFails": true,
"pathPattern": "${TEST_ID}/${RUN_ID}_${FILE_INDEX}.png",
"path": "artifacts/screenshots",
"thumbnails": false
}
}

Your complete test report

Running tests in:

• Chrome 126.0.0.0 / Sonoma 14 (Emulating iPhone X)

Petition card visibility
✖ Browse page shows petition cards (screenshots: /Users/jprymek/Documents/ChangeOrg/debugging/testcafe-examples/artifacts/screenshots/test-1/_errors/1.png)

1. AssertionError: expected false to be truthy

   Browser: Chrome 126.0.0.0 / Sonoma 14 (Emulating iPhone X)
   Screenshot: /Users/jprymek/Documents/ChangeOrg/debugging/testcafe-examples/artifacts/screenshots/test-1/_errors/1.png

   7 |  await t.navigateTo('/browse');
   8 |
   9 |  const petitionCards = Selector('[data-qa="petition-list-card"]');
   

   10 | await t.expect(petitionCards.count).gt(0);
   11 | const firstCard = petitionCards.nth(0);

   12 | await t.expect(firstCard.visible).ok();
   13 |});

   at (/Users/jprymek/Documents/ChangeOrg/debugging/testcafe-examples/tests/petition_card_visibility.js:12:37)
   at asyncGeneratorStep (/Users/jprymek/Documents/ChangeOrg/debugging/testcafe-examples/tests/petition_card_visibility.js:1:37)
   at _next (/Users/jprymek/Documents/ChangeOrg/debugging/testcafe-examples/tests/petition_card_visibility.js:1:37)

1/1 failed (16s)

Screenshots

Steps to Reproduce

1. Run attached tests using TestCafe versions 3.5.0, 3.6.0, 3.6.1, and 3.6.2
2. Observe test results changing between versions.

TestCafe version

3.6.2

Node.js version

20.12.2

Command-line arguments

testcafe # all options set in testcaferc.json file

Browser name(s) and version(s)

Chrome 126.0.6478.128, Chrome 126.0.6478.182

Platform(s) and version(s)

MacOS 14.5, Ubuntu 22.04

Other

Tests have been reproduced on multiple OS's; Chrome versions (TestCafe only shows the major version in the report); NodeJS versions (18 & 20); and with Chrome in both desktop mode, and with mobile emulation (iPhone X). Results are consistent across these options. Upgrading/downgrading TestCafe causes test results to change.

[github-actions]

We appreciate you taking the time to share information about this issue. We reproduced the bug and added this ticket to our internal task queue. We'll update this thread once we have news.