E2E Tests failing with error "You are probably offline."

[thomasplevy]

I've been at this for a few days now... Our e2e tests are based off @wordpress/scripts and use functions from @wordpress/e2e-test-utils.

I've constructed my own version of both of these packages to include in my plugin, doing so has made it simpler for my team and I to make modifications as per our preferences for use across our own repos (we maintain about 25 different plugins both on WordPress.org and self-hosted).

I haven't upgraded our packages in a while so I've been doing so and in doing so the first test in the test suite started to fail with the error "You are probably offline"

I traced this back to, I believe, an error being thrown by apiFetch:

gutenberg/packages/api-fetch/src/index.js

Line 130 in 257ba1e

message: __( 'You are probably offline.' ),

This error presents itself only when run in a CI (I started on Travis and have switched to GitHub actions because I hoped it would either go away or at the very least be easier to debug). The most recent failure as of this moment: https://github.com/gocodebox/lifterlms/runs/3614087294?check_suite_focus=true#step:13:18

The test in question has been reduced, more or less, to the following:

describe( 'something', () => {
	it ( 'should not fetch a thing.', async () => {
		expect( 1 ).toBeTruthy();
	} );
} );

And when I run that the test suite fails with "You are probably offline."

After several days of debugging what I assumed was a problem with my code/plugin/test I've now found that this error is happening when the test does basically nothing. The expect() passes, by the way.

I've found a similar-ish issue here: google/site-kit-wp#2674 but the solution closing that issue doesn't do much.

I've attempted to trace every apiFetch request and none of the ones I can follow via this sort of method seem to throw an error.

I have not been able to fully decouple my code enough to get a reduced test case (I tried install gutenberg for development but npm install isn't working and I don't have the energy to debug why my local environment isn't letting me install gutenberg today) so my code is intertwined and very well may be at fault but I can't figure out where or how or why.

I'm not sure what else to do or where to look. I'm a few days into debugging the same error message and I'm still at a loss. Thanks <3

[talldan]

@thomasplevy You mention:

I've constructed my own version of both of these packages to include in my plugin, doing so has made it simpler for my team and I to make modifications as per our preferences for use across our own repos

Does this mean you depend on the scripts package or not?

It sounds like you have a heavily customised environment, so helping might be difficult.

I did recently update the puppeteer version that scripts uses (#33327), so it might be worth checking you're using the right version of puppeteer in your project. There was also a change to how offline mode and network emulation works (#33410) as part of that. If you're using an up-to-date version of scripts and an outdated version of puppeteer you may experience unexpected results.

[thomasplevy]

Hey @talldan Thanks for getting back to me and sorry for going dark after asking a question.

Does this mean you depend on the scripts package or not?

Yes, it does.

It sounds like you have a heavily customised environment, so helping might be difficult.

It is customized. Yes, I've created shared config files that save me the effort of redefining large sets of configurations against all the plugins I maintain. But I don't know... I suppose it doesn't matter much anyway.

---

I took forever to get back because I was using verions 17.x of @wordpress/scripts. Mostly because I didn't have the time to adjust build configurations to work with webpack 5. It actually wasn't that hard when I sat down and did it but my time is always spread thin.

--

After upgrading to 18.x I'm still having these problems however I had a weird hunch that this has to do with setup / bootstrapping. I can't reproduce any of this locally and debugging in CI environments is pretty difficult in this particular scenario... Anyway... I added jest.retryTimes( 2 ) and now it's all working perfectly. The very first test in the suite fails the first time, it reruns and succeeds, and then all subsequent tests pass.

I don't like this solution but it seems to work. If there was someway to figure out what fetch request was causing the You are probably offline. error I think that would be extremely helpful but I can't figure out a way that I can get at that information. And I'm not sure there's anything the @wordpress/api-fetch package should be concerned with here anyway...

Thanks for your time <3

[talldan]

@thomasplevy I did encounter this situation yesterday!

It turns out I was misusing the __experimentalRest function from the e2e-test-utils package. It was making an incorrect request and returning the "You are probably offline." message (I guess this is what the REST API endpoint returned as an error and __experimentalRest logs this message).

Maybe that offers some clues about what's happening for you.

I can think of two things

• Maybe there's some usage of __experimentalRest in a gutenberg package that's failing for you because your test environment modifies a particular REST endpoint
• Maybe you encountered the same issue I did with your own usage of __experimentalRest.

[thomasplevy]

interesting... I've reviewed this and I don't see any I'm not personally using it in any of my tests, nor are any of my tests using functions from e2e-tests-utils that use it (since it looks like nothing in the package actually uses it). But it does seem like this could be a potential source of the strange problem... My tests are not currently using any REST API functionality where the failure is happening. Perhaps there's something in the WP Core that is being negatively impacted by this that I'm struggling to track down.

At the moment I don't have the time to continue trying to debug this since the retry "fixes" it and it's so tiresome to try to debug something that's only happening in the CI so... you know?

I'll keep this in my when I undoubtedly return to it at some point in the future though.

Thanks again!