Unit tests produce the error "Some of your tests did a full page reload!" after Chrome update

[philmayfield]

Which @angular/* package(s) are the source of the bug?

core

Is this a regression?

No

Description

I don't believe is an Angular issue, but I'm creating this report for clarity and to increase visibility.

Since updating to Chrome ~128.x running unit tests produce an error in spite of all tests passing. To reproduce you can simply create a new project and run ng test.

Please provide a link to a minimal reproduction of the bug

No response

Please provide the exception or error you saw

Chrome Headless 128.0.0.0 (Mac OS 10.15.7) ERROR
  Some of your tests did a full page reload!
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 4129 of 4129 SUCCESS (18.521 secs / 17.638 secs)

Please provide the environment you discovered this bug in (run ng version)

Angular CLI: 17.3.8
Node: 18.20.4
Package Manager: npm 10.7.0
OS: darwin arm64

Angular: 17.3.11
... animations, common, compiler, compiler-cli, core, forms
... language-service, localize, platform-browser
... platform-browser-dynamic, router

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.1703.8
@angular-devkit/build-angular      17.3.8
@angular-devkit/core               17.3.8
@angular-devkit/schematics         17.3.8
@angular-devkit/schematics-cli     17.3.8
@angular/cdk                       17.3.10
@angular/cli                       17.3.8
@angular/flex-layout               15.0.0-beta.42
@angular/material                  17.3.10
@angular/material-moment-adapter   17.3.10
@schematics/angular                17.3.8
ng-packagr                         17.3.0
rxjs                               7.8.1
typescript                         5.3.3
zone.js                            0.14.7

Anything else?

I tested it out on new Angular 17 and 18 projects using node 18 and 20 and get the same results for all of them. I was able to verify that upon reverting to Chrome 127.x the error is not produced.

[alemp]

Same happened here after update Chrome to v128.0.0.0.

Angular CLI: 17.3.8
Node: 18.20.3
Package Manager: npm 10.7.0
OS: darwin arm64

Angular: 17.3.12
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1703.8
@angular-devkit/build-angular   17.3.8
@angular-devkit/core            17.3.8
@angular-devkit/schematics      17.3.8
@angular/cdk                    17.3.10
@angular/cli                    17.3.8
@angular/material               17.3.10
@schematics/angular             17.3.8
rxjs                            7.8.1
typescript                      5.4.5
zone.js                         0.14.10

[freon27]

I also have this issue. I'm only commenting to add that I ONLY see the problem in non-watch mode : ng test --watch=false

[OPuder]

The same thing happened here after updating Chrome to v128.0.0.0.

[philmayfield]

I ONLY see the problem in non-watch mode : ng test --watch=false

I noticed that too, but I think once you stop watching it'll spit out the error.

[rdamazio]

Also seeing this, and it's not a specific test that fails:

Angular CLI: 18.2.1
Node: 22.7.0
Package Manager: npm 10.8.2
OS: darwin x64

Angular: 18.2.1
... animations, cdk, cli, common, compiler, compiler-cli, core
... forms, material, platform-browser, platform-browser-dynamic
... platform-server, router, ssr

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1802.1
@angular-devkit/build-angular   18.2.1
@angular-devkit/core            18.2.1
@angular-devkit/schematics      18.2.1
@schematics/angular             18.2.1
rxjs                            7.8.1
typescript                      5.5.4
zone.js                         0.14.10

Test output:

> ng test --watch=false --browsers=ChromeHeadless

✔ Browser application bundle generation complete.
23 08 2024 15:49:17.645:INFO [karma-server]: Karma v6.4.4 server started at http://localhost:9876/
23 08 2024 15:49:17.647:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
23 08 2024 15:49:17.651:INFO [launcher]: Starting browser ChromeHeadless
23 08 2024 15:49:19.042:INFO [Chrome Headless 128.0.0.0 (Mac OS 10.15.7)]: Connected on socket 0lhJDWyRZAKUz0g-AAAB with id 93949148
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 0 of 36 SUCCESS (0 secs / 0 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 0 of 36 SUCCESS (0 secs / 0 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 0 of 36 SUCCESS (0 secs / 0 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 0 of 36 SUCCESS (0 secs / 0 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 0 of 36 SUCCESS (0 secs / 0 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 0 of 36 SUCCESS (0 secs / 0 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 0 of 36 SUCCESS (0 secs / 0 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 36 of 36 SUCCESS (0.536 secs / 0.5 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7) ERROR
  Some of your tests did a full page reload!
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 36 of 36 ERROR (0.536 secs / 0.5 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7) ERROR
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 36 of 36 ERROR (0.55 secs / 0.5 secs)

[andonary]

Similar issue here, does not appear on watch=true. I can't find what is the root cause of it.

Angular CLI: 18.0.4
Node: 22.2.0
Package Manager: npm 10.7.0
OS: darwin arm64

Angular: 
... 

Package                      Version
------------------------------------------------------
@angular-devkit/architect    0.1800.4
@angular-devkit/core         18.0.4
@angular-devkit/schematics   18.0.4
@schematics/angular          18.0.4

> ng test --no-watch --no-progress --browsers=ChromeHeadless

26 08 2024 13:56:56.078:INFO [karma-server]: Karma v6.4.3 server started at http://localhost:9876/
26 08 2024 13:56:56.079:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
26 08 2024 13:56:56.081:INFO [launcher]: Starting browser ChromeHeadless
26 08 2024 13:56:56.739:INFO [Chrome Headless 128.0.0.0 (Mac OS 10.15.7)]: Connected on socket SRHYT2w467iGirpgAAAB with id 23475697
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 10 of 73 SUCCESS (0 secs / 0.045 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 73 of 73 SUCCESS (0.191 secs / 0.173 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7) ERROR
  Some of your tests did a full page reload!
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 73 of 73 ERROR (0.191 secs / 0.173 secs)
Chrome Headless 128.0.0.0 (Mac OS 10.15.7) ERROR
Chrome Headless 128.0.0.0 (Mac OS 10.15.7): Executed 73 of 73 ERROR (0.195 secs / 0.173 secs)

[Blackbaud-SteveBrush]

Related? karma-runner/karma#3560 (comment)

I set clearContext: config.singleRun in my local karma.config and the error goes away...

[rachcampitos]

Related? karma-runner/karma#3560 (comment)

I set clearContext: config.singleRun in my local karma.config and the error goes away...

In my case I did the same but still getting the error

[dgp1130]

I'm chatting with some Karma folks to see if we can understand what's going on here. Unfortunately we're not able to reproduce the exact issue, so I have a few questions someone can maybe help out with:

1. Can anyone provide a minimal example? The OP suggests "To reproduce you can simply create a new project and run ng test.", though that's not sufficient for me. I'm wondering if anyone can confirm on their end that this is a sufficient reproduction, or if other requirements are necessary?
2. All the reports I see here list Chrome Headless. Is anyone able to reproduce with a "headfull" test?
3. Related, is anyone able to reproduce with --headless=new? I suspect using the ChromeHeadless browser likely uses the old version of --headless in Chrome and I'm wondering if the issue might be related to that. I think testing both of these require specifying a "custom browser" in your Karma config.
4. All the reports I see here are also darwin arm64. I happen to be on Linux, so I'm wondering if anyone can reproduce on a non-Mac to at least rule out that particular aspect as a key requirement.
5. Do tests actually fail because of this error (exit code != 0) or is this just a spurious warning which doesn't actually affect the result of the test?

[Stargator]

@dgp1130
I'll take a stab at answering some of these from what I see:

Chrome version: 128.0.6613.85
@angular/core: 17.3.10
@angular/cli: 17.3.8

1. Can anyone provide a minimal example? The OP suggests "To reproduce you can simply create a new project and run ng test.", though that's not sufficient for me. I'm wondering if anyone can confirm on their end that this is a sufficient reproduction, or if other requirements are necessary?

I cannot do this. I just got the error and I am rushing to deliver a product. Maybe if no one else does, I can follow-up later on Tues.

2. All the reports I see here list Chrome Headless. Is anyone able to reproduce with a "headfull" test?

I have ran ng test and I do not get the error. Then I run the tests in headless mode and I see the error.

3. Related, is anyone able to reproduce with --headless=new? I suspect using the ChromeHeadless browser likely uses the old version of --headless in Chrome and I'm wondering if the issue might be related to that. I think testing both of these require specifying a "custom browser" in your Karma config.

I added --headless=new to my karma.config.js and the error persists

4. All the reports I see here are also darwin arm64. I happen to be on Linux, so I'm wondering if anyone can reproduce on a non-Mac to at least rule out that particular aspect as a key requirement.

I am also on a darwin arm64, so I cannot help there :(

5. Do tests actually fail because of this error (exit code != 0) or is this just a spurious warning which doesn't actually affect the result of the test?

1. I ran echo $? after running the headless tests and it responded with 0. So maybe it wouldn't cause a failure.
2. I ran the tests in our pipeline, but it currently uses Chrome 124.0.6367.201 and it didn't report any error.

[andonary]

I have ng test on my pre-commit using husky. If it throws an code != 0 I wouldn't able to commit and I can.

I'll watch for a minimal example but same as Stargator I get this error on a enterprise product I cannot show the code.

[freon27]

It caused my pre-push hook to fail and I had to switch off the hooks to be able to push so for us it appears to be actually causing a non-zero exit.

[alan-agius4]

@dgp1130, I was able to replicate this issue exclusively on macOS, and it occurs in both headless and non-headless modes of Chrome version 128.0.0.0.

When --watch is set to a non-truthy value, I also observed the Some of your tests did a full page reload! error appearing when the browser closes after all tests have successfully run. However, in my case, the process did not terminate with a non-zero exit code.

[btd1337]

Related? karma-runner/karma#3560 (comment)

I set clearContext: config.singleRun in my local karma.config and the error goes away...

This works for me.

Environment:

Angular 17
macOS Sonoma

[JohanBeekers]

I encountered the same issue today on our CI environment. It was fine locally with chrome 128, headless or not. But I'm running Windows. Our CI pipeline runs Mac OS though.

I was able to fix it by adding the following beforeAll code. Basically just preventing any redirects.

It changed the CI output from the left (failing) to the right (succeeding)

[andonary]

I encountered the same issue today on our CI environment. It was fine locally with chrome 128, headless or not. But I'm running Windows. Our CI pipeline runs Mac OS though.

I was able to fix it by adding the following beforeAll code. Basically just preventing any redirects.

I put this on a single test file, and it fixed the error for some reason.

[dgp1130]

@alan-agius4 was able to reproduce the issue in a base Karma setup without Angular, so that at least rules out Angular as the root cause here: karma-runner/karma#3887

That said, setting clearContext: singleRun seems to be a reasonable workaround, so we released that as a default behavior for Angular in all actively supported versions (16.2.15, 17.3.9, 18.2.2). We expect that will resolve the symptoms of the issue for most developers. If you have a custom Karma config and that explicitly sets clearContext, you may need to remove that or manually set it to true when singleRun === true.

Beyond that, hopefully the Karma and/or Chrome teams can figure out what's going on under the hood here, as there's definitely a deeper root cause which is not understood by us right now.

[aparzi]

Hi @dgp1130,
I have same error in Angular 15 and Chrome 128.0.0 is there a way to see the coverage report?

[rcollette]

Using the clientContext: config.singleRun option did not work for me and removing that setting entirely assuming that 16.2.16 was going to fix it, both did not work for me.

The issue only appears for me when running in headless mode with ng test from a command line. When I run with WebStorm

/Users/Richard.Collette/.nvm/versions/node/v20.15.1/bin/node /../myProjectDir/node_modules/@angular/cli/bin/ng test client --karma-config /Users/Richard.Collette/Applications/WebStorm.app/Contents/plugins/karma/js_reporter/karma-intellij/lib/intellij.conf.js --source-map

I don't see the error.

[manueltarouca]

Configuring karma custom launcher arguments to "--headless=old" fixed it.

[Yicong-Huang]

Sharing my experience for others who may still have the same issue. On my side, I am using Chrome 130.0.0.0 with MacOS 10.15.7 darwin arm64, angular cli 16.12.15 and karma 6.4.4.

I believe it is a karma issue but they have deprecated the project so not much maintenance to be expected on their end, so posting here in case this is helpful.

• I've tried with Firefox and there is no issue. Chrome had the issue no matter if it is on headless or "headfull" mode. And the return error code is 1.
• I've launched a 0-test run and it still fails with the error. DEBUG log suggest that the error is during forcing killing the chrome and disconnecting it, the chrome page got reloaded.
• The previous fix on --headless=old, --headless=new, clearContext: true or overloading window.onbeforeunload did not work for me.
• The working fix for me is ng test --watch=false, which clears the error and returns 0.