Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jest-snapshot: Display change counts in annotation lines #8982

Merged

Conversation

pedrottimark
Copy link
Contributor

@pedrottimark pedrottimark commented Sep 25, 2019

Summary

The number of change lines helps to interpret differences when a snapshot assertion fails:

  • only delete lines (that is, in snapshot but not in received)
  • only insert lines
  • equal number of delete and insert lines
  • almost equal number of delete and insert lines
  • quite unequal number of delete and insert lines

Similar to benefit of summary report following a test run, except these precede the comparison.

In addition to the includeChangeCounts option from #8881 here are supporting changes:

  1. Call functions directly from jest-diff instead of indirectly via jest-matcher-utils as prerequisite for color options in a future pull request, and also swap where two snapshot transformations are called:

    • Move removeExtraLineBreaks from _toMatchSnapshot function to match method, because it is a private detail of snapshot formatting
    • Move and rename unescape function from match method to printDiffOrStringified function, because its use depends on how the report displays the snapshot (see 2)
  2. Adjust changes in Unescape serialized snapshots for diffing #2852 to unescape double quote marks and also backslashes but only when the report displays string differences, and if both of the following:

    • snapshot looks like default serialization of a string (otherwise an array or object can have strings which ought to be enclosed in double quote marks and have escapes)
    • received is string which has default serialization
  3. With change counts as a clue to interpret edge cases when the snapshot or received is empty string and its counterpart is multiline string, display as comparison lines instead of labeled lines; which reverses a previous decision:

Next steps

  1. What do y’all think about a follow-up pull request to display change counts in differences when toBe, toEqual, toStrictEqual, toHaveProperty, toMatchObject assertions fail?

  2. Change confusing colors in snapshot differences, but not exactly as requested in Diff colors should show green for insert, red for delete #5430

  3. Ignore indentation in serialization of received objects and by heuristic in snapshots to remove distracting differences, see Optionally hide whitespace changes in diff #8626

Test plan

package test file changes
e2e failures updated 2 snapshots
e2e toMatchSnapshot replaced 1 old toMatch assertion with 2 new
e2e watchModeUpdateSnapshot updated 1 snapshot
jest-matcher-utils printDiffOrStringify replaced 1 old test (which had obsolete negative assertion) with 3 new
jest-snapshot printDiffOrStringified updated 23 17 and added 4 snapshots; added 3 6 tests

See also pictures in 4 following comments: baseline is at left and improved is at right

@pedrottimark
Copy link
Contributor Author

pedrottimark commented Sep 25, 2019

In the following 5 pictures, the only difference is the change counts

The changes are insertions in the received value:

1 object insert

The changes are replacements:

2 markup

This comparison will improve in future pull request to ignore indentation:

3 object indentation

Some changed lines have substring differences:

4 diffStringsUnified

Unlike other assertions, snapshots do not “know” the expected type:

5 different type

Copy link
Contributor

@scotthovestadt scotthovestadt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks solid to me. Not 100% sold on the way it's displayed and want opinions.

+ Received 1 +

- bar
+ foo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this case, where it's a string that was 1 line and will again be 1 line, this information is pretty useless. Can we not display it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I will deal with comparison of two one-line strings as a special case for the concise labeled format, because it seems likely for toThrowErrorMatchingSnapshot assertions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe with a clearer labeling (other comment thread) it's actually fine and worth keeping for consistency?

Copy link
Contributor Author

@pedrottimark pedrottimark Sep 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tim, good point about prioritizing consistency higher than a more concise special case.

If most reports have differences when snapshot assertions fail, and if next pull request changes the colors to be less confusing, some people might learn to scan quickly which console output is from snapshots versus other assertions, and more intuitively choose which decision path.

For example, the e2e tests for this pull request had 3 snapshots to update and 1 toMatch assertion which I needed to rewrite as 4 assertions.

To test this theory: think fast, what do you need to do for each of the following:

Expected: "bar"
Received: "foo"
Snapshot: "bar"
Received: "foo"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I know what you mean, if there's lots of (correct) test failures I usually try to change non-snapshot tests first and then update snapshots once they're the only things still failing. Not sure how that's related to this comment thread but I'd definitely appreciate making it easier to tell them apart.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See pictures in #8982 (comment) of concise label format if neither string is multiline

@pedrottimark
Copy link
Contributor Author

pedrottimark commented Sep 25, 2019

For default serialization in concise format, improved does not unescape string differences:

See if you can interpret the meaning

For default string serialization, keep the concise label format:

16 expected empty

The improved picture at the right is possible with custom raw string serialization:

6 expected empty

Because the received is multiline string, display as comparison lines:

7 received empty

For default string serialization, keep the concise label format:

13 both with

The improved picture at the right is possible with custom raw string serialization:

8 one line

@pedrottimark
Copy link
Contributor Author

pedrottimark commented Sep 25, 2019

In the following 3 pictures of edge cases, improved does not unescape quoted strings:

baseline code never unescaped backslashes and improved does not in object serialization:

9 escaped backslash

baseline (above) always unescaped double quotes but improved (below) doesn’t in object serialization:

10 escape quote

do not display differences for a few received types like number

the expected snapshot string '1 \'\n2 "\n3 \\' has escaped characters:

11 received non-line-diffable

@pedrottimark
Copy link
Contributor Author

pedrottimark commented Sep 25, 2019

The last picture illustrates the edge case of change from quoted string in default serialization to unquoted string in raw serialization, like recent chores #8957 #8959 #8976

12 to unquoted

@codecov-io
Copy link

codecov-io commented Sep 25, 2019

Codecov Report

Merging #8982 into master will increase coverage by 0.15%.
The diff coverage is 95.74%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8982      +/-   ##
==========================================
+ Coverage    63.9%   64.05%   +0.15%     
==========================================
  Files         277      277              
  Lines       11652    11676      +24     
  Branches     2860     2863       +3     
==========================================
+ Hits         7446     7479      +33     
+ Misses       3576     3572       -4     
+ Partials      630      625       -5
Impacted Files Coverage Δ
packages/jest-snapshot/src/State.ts 0% <0%> (ø) ⬆️
packages/jest-snapshot/src/print.ts 100% <100%> (+29.62%) ⬆️
packages/jest-snapshot/src/utils.ts 95.18% <100%> (+0.18%) ⬆️
packages/jest-diff/src/printDiffs.ts 100% <100%> (ø) ⬆️
packages/jest-snapshot/src/index.ts 27.39% <50%> (-1.47%) ⬇️
packages/jest-matcher-utils/src/index.ts 83.22% <0%> (+0.64%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 630350a...96389e7. Read the comment docs.

@pedrottimark
Copy link
Contributor Author

pedrottimark commented Sep 27, 2019

If neither quoted string is multiline, then restore a previous improvement to display with labels:

Small change suggests either update after clean up message or fragile test (for example, v8 versus Gecko) so replace toThrowErrorMatchingSnapshot with toThrow(regexp) assertion:

13 both with

Either regression or change if received has no common substring with expected:

14 both without

Reasons why not to display as unescaped strings without double quote marks:

  • empty string looks confusing
  • content of string might look like serialization of another type, like number 17

Possible regression if received message is empty string:

15 received empty

Rare edge case for expected snapshot to be empty string:

16 expected empty

However, for custom raw string serialization, always display as diff comparison lines.

packages/jest-diff/src/printDiffs.ts Outdated Show resolved Hide resolved
packages/jest-diff/src/printDiffs.ts Show resolved Hide resolved
packages/jest-snapshot/src/index.ts Outdated Show resolved Hide resolved
packages/jest-snapshot/src/print.ts Show resolved Hide resolved
Copy link
Contributor

@jeysal jeysal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@scotthovestadt wanna take a look again to see if you like it in the current state?

@pedrottimark
Copy link
Contributor Author

And what do y’all think about a follow-up pull request to display change counts for other assertions?

@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 11, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants