goss add command to set default assertions for empty stdout/stderr

[gberche-orange]

Describe the feature:

As a goss user,
In order to detect non expected stderr/stdout messages in command output
I need the goss add command to set assertions when observed stdout/stderr are empty

For instance, if a binary is missing from the test environment and stderr includes "command not found", I'd need the stderr assert to fail

Describe the solution you'd like

A way to assert empty stdout/stderr similar to #184

And this syntax should be used by default in goss.yaml when the goss add command observes empty stdout/stderr

Describe alternatives you've considered

I tried using rexexp to check empty strings such as:

command:
  # the sleep command produces no stdout or stderr
  sleep 1s:
    exit-status: 0
    stdout: [] # this syntax seems to always succeed, regarless of whether output is sent to stdout
    stderr: [ "/^$/" ] # this syntax seems to always fail, whereas no output to stderr should pass test
    timeout: 10000

however goss isn't yet failing this assertion:

$ goss validate
.F

Failures/Skipped:

Command: sleep 1s: stderr: patterns not found: [/^$/]

Total Duration: 1.006s
Count: 2, Failed: 1, Skipped: 0

[aelsabbahy]

Thank you for filing this. There seem to be two parts:

1. A bug
2. A feature request

bug

The bug has to do with "/^$/", you're right that is odd behavior. I know why it's happening, and it's not intuitive for the end user. The workaround suggested in the linked issue will solve it, but it isn't evident:

command:
  # the sleep command produces no stdout or stderr
  sleep_test:
    exec: |
      >&2 echo "hi"
    exit-status: 0
    stderr: [ "!/./" ]
    timeout: 10000

See my comment here for more info: #145 (comment)

Essentially, the check is if any line is empty.. but there are no lines, so it fails to find an empty line. You can test this behavior with echo "" vs echo -n "", the former will pass [ "/^$/" ] the latter won't.

My memory is a bit hazy, but I feel like I explored making a special case for empty output, but it caused a breaking change somewhere.. but I could be wrong. I'll re-evaluate this with fresh eyes and see if it makes sense to special case empty string.

feature request

For the feature request, the upcoming goss v4 #814 provides a way to handle this easier:

command:
  sleep_test:
    exec: |
      >&2 echo hi
    exit-status: 0
    stderr: ""

$ goss v
.F

Failures/Skipped:

Command: sleep_test: stderr:
Expected
    "hi\n"
to equal
    ""

Total Duration: 0.001s
Count: 2, Failed: 1, Skipped: 0

It will also handle newlines correctly, for example:

$ cat goss.yaml 
command:
  sleep_test:
    exec: |
      >&2 echo ""
    exit-status: 0
    stderr: ""
$ goss v
.F

Failures/Skipped:

Command: sleep_test: stderr:
Expected
    "\n"
to equal
    ""

Total Duration: 0.002s
Count: 2, Failed: 1, Skipped: 0

The only way the above will pass is with echo -n ""

I'll have to reflect a bit about defaults when doing a goss add. It probably won't be in the initial v4 release.. but might be added in a future release.

feedback

Would love your thoughts on the above..

[gberche-orange]

Thanks a lot @aelsabbahy for your prompt and detailed answer !

Bug
The workaround suggested in the linked issue will solve it

Thanks for the workaround which properly detects empty strings, with the following known limitations

• the current stdout/stderr isn't displayed

command:
  "echo my_stdout > /dev/stdout; echo my_stderr > /dev/stderr":
    exit-status: 0
    stdout: [ "!/./" ]
    stderr: [ "!/./" ]
    timeout: 10000

$ goss validate
.FF

Failures/Skipped:

Command: echo stdout > /dev/stdout; echo stderr > /dev/stderr: stdout: patterns not found: [!/./]
Command: echo stdout > /dev/stdout; echo stderr > /dev/stderr: stderr: patterns not found: [!/./]

Total Duration: 0.002s
Count: 3, Failed: 2, Skipped: 0

feature request
For the feature request, the upcoming goss v4 #814 provides a way to handle this easier

This looks great to me, fully satisfying my needs !

I'll have to reflect a bit about defaults when doing a goss add. It probably won't be in the initial v4 release.. but might be added in a future release.

With the v4 behavior for "" matcher, the current default behavior (i.e. adding "" matchers in goss add command when observing empty strings) seems a good sensible default to me.

I'm looking forward to using goss v4 !

[aelsabbahy]

Forgot to mention, I released a release candidate v0.4.0-rc1 a few days (weeks?) ago.

Would love your feedback on it to make sure it correctly covers this need.

[aelsabbahy]

v0.4.2 is released with empty strings when there's no output.

Feel free to re-open an issue if this is still a problem. Thank you for reporting!