Capture + Display all logs for everything that happens in Cypress

[brian-mann]

Problem

Many of our users express the desire to see console.log statements when running headlessly. Typically you'd be able to see these when you're working in Dev Tools (in headed GUI mode).

We can pretty easily display console.log statements by mixing them into stdout when running headlessly, but we don't think this is really a "true" solution. It's kind of a hack and it still doesn't really tell you "everything" that's happening. For instance, when a console.log happens, how would you know when it happened in relation to everything else that's going on in a test?

So, we think to truly solve this problem, we need a comprehensive way of displaying not just console.log statements but also capturing:

• Network Traffic
• Commands
• Retries
• Under the hood Cypress wizardry
• DOM snapshots on failure

By providing all of this, the entire picture of what happened during a test is revealed and now you actually have even more information at your disposal than even what the GUI headed version provides you.

Solution

The logs will be available in the Cypress Cloud.

Once we capture and parse these logs, it will unlock an enormous amount of power in Cypress. We'll also be able to analyze the logs and deliver things like:

• Analytics into your tests
• Comparing a failing test to a previously passing one
• Diffing the logs for changes in response bodies, etc
• Understanding why tests are slow
• Discovering command anti patterns (like explicit waits in your code)

The logs could also be interactive - for instance they should "sync" up to the video and enable you to playback everything that actually happened in a Cypress test.

Here is an example comp of what we're going for.

We're still working on communicating the start and end of logs, and also communicating that a log event acts as "a stream" but that events are connected to other events.

For instance when a HTTP request starts - that's a 'start' event. When the response comes back, thats a separate 'end' event. However these are connected and you likely need to be able to view the data for both events from either the start or end event.

Another example:

A command starts. It retries 65x times. It eventually finishes. The duration of the retry events, the reason its retrying, and the eventual delta between a start and an end needs to be communicated intelligently.

These log files end up being a massive amount of data, but we already have the infrastructure in place to capture, parse and deliver them. At this point we're just iterating on a UI that's intuitive and doesn't overwhelm the user.

We're going to be providing a new Logs tab, but also for individual test failures, we'll just be providing the logs for that one independent test.

Feedback

...is welcome ;-)

[brian-mann]

@bahmutov what do you think?

[bahmutov]

Perfecto! 😍

[ghost]

@brian-mann excellent. Really looking forward to getting this. We're spending lots of time trying to debug why our compiled JS is not working when served static via Cypress. We cannot replicate anywhere else, so one peek at the logs would like solve everything!

[danielnass]

Hi guys!

Amazing feature! Any news about that?

Thanks

[cristopher-rodrigues]

AMAZING!

[lukemadera]

@brian-mann I've been banging my head against a wall trying to upgrade to cypress 0.20.1. It's been days - things work fine in the GUI but fail from the console. And not being able to log anything is infuriating. After super old school commenting out code one line at a time until it passes again to isolate the line, I've found that it is cy.request that fails. But I don't know how to fix it yet, and I can't log anything so have no idea what the issue is. Adding all the features you noted in this ticket sounds fine, but we need a simple way to debug locally, in the command line itself. How can we get console.log (or cy.log) to show in stdout when running headlessly? We're blocked from using Cypress right now until we fix that because our CI is failing now due to this other error, which is supposedly fixed in 0.20 #517

[brian-mann]

Also at this very moment you could do this trick here: #300 (comment)

That will get all console.log's to show up in the command log, which will stringify and print their arguments. You could then watch the video / screenshot to see what the information is.

That's what I would do today when running in CI. Of course locally just show the browser and iterate on it there.

[robertvansteen]

This would be a really nice addition. Right now my tests are failing on the CI and it looks like it has something to do with cookies but because Cypress is basically a black box right now on CI I have no way of figuring out what's going on. Hope this feature will arrive soon, keep up the good work 👏🏼

[brian-mann]

Related to #699 and #700.

Those will land pretty soon and should help alleviate these problems.

[brandonb927]

Just want to throw my 2 cents in, I'm also having an issue with (what I can only imagine is cookie-related) the CI build vs local cypress open run that @rovansteen is having, and not having a way to see what network requests are made is making it challenging to debug.

So far I'm loving Cypress though, you folks are doing amazing work and I'm more than willing to fork over our company credit card for a paid plan in the future!

[lmiller1990]

A lot of work happened over the last little while to enable us to finally work on something like this. It's not a lot more practical to implement something like this due to various under-the-hood improvements to the App and Cloud platforms over the last year or two. I hope we can share some updates and progress soon.

In the meantime, some debugging-in-CI techniques I've found useful:

Is there really no way to show cy.log() entries in the official results dashboard?

The way I've done this historically is by using cy.log and cy.screenshot. You can do cy.screenshot(..., { capture: 'runner' }) to include the command log in the screenshot.

You can pass JSON.stringify(<value>, null, 4) and get some pretty elaborate logging going. Not really a fully featured solution, but it's helped me out a ton when debugging.

Screenshot docs: https://docs.cypress.io/api/commands/screenshot#Arguments

[Opalexiss]

I am looking for this as well, and none of the workarounds are usable for me.
My app outputs various information to the console while in debugging mode, which is enabled for Cypress.
In a test, I would like to do something simple such as click on a button, then make an assertion based on what was immediately logged to the developer console afterwards.

The only actual workaround for this is to stick a hidden div somewhere in the DOM that mirrors everything that is being output to the console. My apps communicate internal states and variables for tests, and these are not always accessible from the DOM itself. I would like to assert based on these console.log messages.

[lmiller1990]

You can spy/stub the console:

• https://stackoverflow.com/questions/53898085/check-if-an-error-has-been-written-to-the-console
• https://www.youtube.com/watch?v=zRAjO6rccfI

[M4RcB4]

This issue is still in the 'proposal' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

Current workarounds: use these plugins

• https://github.com/flotwig/cypress-log-to-output - Prints all browser console logs to the terminal while running Cypress tests. Currently, only Chrome is supported.
• https://github.com/bahmutov/cypress-failed-log - Saves the Cypress test command log as a JSON file if a test fails.
• https://github.com/archfz/cypress-terminal-report - Logs cypress commands, route request data and browser console errors and warnings to terminal when tests fail on CI.

@brian-mann @bahmutov @lmiller1990 Any official work being done on this feature? I will try the plugins mentioned here. I would also love to keep in step with future improvement. I think this is an immensely valuable feature.

[lmiller1990]

@M4RcB4 Yes, there is. I don't have an exact date, but it's an active and high priority project. Please stay tuned - more info will come as soon as it is ready for testing!

The goal is to capture all the things - console, network etc. As a long time user, I am quite excited - debugging with videos is good, but this will be better.

[boweiliu]

Hey, is the intention for the logs to be only available via cypress dashboard, or would they also be printed to the terminal/saved to a local .txt as well?

[ryanthemanuel]

The project that @lmiller1990 mentioned will be only available in the cypress dashboard.

[Opalexiss]

So, still no way to make assertions against console output.

[lmiller1990]

@Opalexiss you can definitely make assertions against the console methods: https://glebbahmutov.com/cypress-examples/recipes/check-console-logs.html

[chrisbreiding]

Closed by #27040

[karlhorky]

@chrisbreiding nice, thanks! Although, the PR description on that #27040 PR is a bit unclear - which PR in v13.0.0 actually addresses this issue #448?

I couldn't figure that out from either the PR description or the v13.0.0 changelog

[chrisbreiding]

@karlhorky I can understand how it might be a bit confusing. #27040 was a PR for a long-running branch that collected a number of PRs for the v13 release. That includes several PRs that address this issue. For the most part, however, those changes are not user-facing, so they're not called out in the v13 PR or the changelog. Most of the work for this is under-the-hood (capturing all the things!).

While the changelog published with this repo doesn't mention it, the changelog that will be published to the docs highlights the functionality this will enable for Cypress Cloud.

[jennifer-shehane]

Released in Cypress 13.0.0 as Test Replay. Please read our Blog for more details.

[karlhorky]

Nice, thanks for the additional background, looks exciting!

[tchaton]

@jennifer-shehane I am not sure Test Replay was the actual requested feature.

Personally, I want to be able to dump console.log, network tabs to a file, so we I can read it to understand what went wrong.

Best,
T.C

[comp615]

@jennifer-shehane what's the story for replay around non-cloud CI/CD setups?

I'm a bit worried this is a dark pattern change designed to remove useful functionality from that set of users, especially when viewed as coupled with the removal of videoUploadOnPasses, for instance. Happy to take the tin foil hat off, but I didn't see any discussion here or on other threads of allowing an "asset" to be captured in CI that would let people view the runs locally.

[lmiller1990]

@comp615 Was something removed? I don't work at Cypress, but I still use it a lot -- my understanding is the Test Replay is for Cloud users. You can still record videos to your local machine, review them after a failure, as part of the OSS offering.