Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show full result when failing to match output in tests #1112

Merged
merged 1 commit into from
Aug 15, 2022

Conversation

dirceu
Copy link
Contributor

@dirceu dirceu commented Aug 15, 2022

We seem to have a flaky test in main; the test failure only tells us that the output is empty, but it doesn't show anything about stderr or about the overall success.

This PR introduces and uses two new assertions that might make it easier to debug such test failures:

  • assert_stdout_includes
  • assert_stderr_includes

If I manually setup a failure on that flaky test, this is what we get now:

cli
  gem
    generate
      it must not include `rbi` definitions into `tapioca` RBI        FAIL (1.61s)
        ########## STDOUT ##########
        Requiring all gems to prepare for compiling...  Done
        
        Removing RBI files of gems that have been removed:
        
          Nothing to do.
        
          Compiled tapioca
              create  sorbet/rbi/gems/tapioca@0.9.2.rbi
        
        Checking generated RBI files...  Done
        
          Changed strictness of sorbet/rbi/gems/tapioca@0.9.2.rbi to `typed: false` (conflicting with DSL files)
        
        All operations performed in working directory.
        Please review changes and commit them.
        
        ########## STDERR ##########
        <empty>
        ########## STATUS: true ##########

@dirceu dirceu requested a review from a team as a code owner August 15, 2022 17:24
spec/spec_with_project.rb Outdated Show resolved Hide resolved
spec/spec_with_project.rb Show resolved Hide resolved
@dirceu dirceu force-pushed the show-full-result-when-output-doesnt-match branch from 0e5fc96 to 1e13084 Compare August 15, 2022 17:41
Copy link
Member

@paracycle paracycle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙏

This should make it easier to debug test failures.
@dirceu dirceu force-pushed the show-full-result-when-output-doesnt-match branch from 1e13084 to f447eb2 Compare August 15, 2022 18:06
@dirceu dirceu merged commit 16c8c74 into main Aug 15, 2022
@dirceu dirceu deleted the show-full-result-when-output-doesnt-match branch August 15, 2022 18:28
@shopify-shipit shopify-shipit bot temporarily deployed to production August 31, 2022 14:40 Inactive
@shopify-shipit shopify-shipit bot temporarily deployed to 0-10-stable September 14, 2022 02:35 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants