Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use $BATS_RUN_TMPDIR for $BATS_MOCK_TMPDIR #19

Merged
merged 1 commit into from
Feb 13, 2024

Conversation

KlfJoat
Copy link

@KlfJoat KlfJoat commented Jan 3, 2024

Update $BATS_MOCK_TMPDIR to $BATS_RUN_TMPDIR instead of $BATS_TMPDIR

Observed Behavior

Re-running Bats tests that call bats-mock exhibit unusual behavior when not all mocks are used and unstub is not called on all programs.

I created a bats test file that used bats-mock and (admittedly) I didn't write the test properly and I didn't unstub one program correctly. IMO, bats-mock should be resilient against my mistakes.

The problem exhibits just running a test like the one below, using only bats (and bats helper) code.

#!/usr/bin/env bats
load "$BATS_TEST_DIRNAME/bats_helpers/bats-support/load"
load "$BATS_TEST_DIRNAME/bats_helpers/bats-assert/load"
load "$BATS_TEST_DIRNAME/bats_helpers/bats-mock/stub"
@test "quux" {
  # Verify that foo is stubbed
  stub foo ': echo "STUBBED"'
  run foo
  assert_success
  assert_output "STUBBED"

  # Verify that bar is stubbed
  stub bar \
    ': echo "STUBBED"' \
    ': exit 1'
  run bar
  assert_success
  assert_output "STUBBED"

  # Test the exit function
  # run bar
  # assert_failure 1
  # refute_output

  unstub foo
  # unstub bar
}

Repeatedly running this test will alternate between passing and failing.

Why?

It appears that bats-mock uses $BATS_TMPDIR under the hood as the mock location. The problem is, per https://bats-core.readthedocs.io/en/stable/writing-tests.html#special-variables , that is just $TMPDIR or /tmp.

Since bats-mock does not clean up after itself, the ${program}-stub-plan and ${program}-stub-run files (that were not unstubbed by my bad test writing) persist across runs, leading to the alternating behavior.

On the other hand, $BATS_RUN_TMPDIR (added in bats v1.2.1) is cleaned up (or not if bats --no-tempdir-cleanup is used) after each run... And reuse of that directory is actively avoided by bats's invocation of mktemp when creating $BATS_RUN_TMPDIR.

Testing

I wanted to add a test for this issue to bats-mock, however because the results change between instances of bats itself, I'm not sure how to write a bats or other test for the issue.

I have tested the change and the test code above succeeds on every run. While that's not desired behavior (ideally bats-mock would throw an error if a program is stubbed and not unstubbed), it's at least consistent behavior which to my mind is preferable to inconsistency.

All of my other tests that use bats-mock pass with these changes, as do the bats-mock/tests.

Re-running incomplete tests with bats-mock exhibited inconsistent test
results.
Copy link

@pzeballos pzeballos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @KlfJoat ! we had it in our backlog to fix 🙌🏻

@pzeballos pzeballos merged commit 4554c14 into buildkite-plugins:master Feb 13, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants