[SPARK-32106][SQL][FOLLOWUP] Fix flaky tests in transform.sql

[maropu]

What changes were proposed in this pull request?

This PR intends to fix flaky GitHub Actions (GA) tests below in transform.sql (this flakiness does not seem to happen in the Jenkins tests):

• https://github.com/apache/spark/runs/1592987501
• https://github.com/apache/spark/runs/1593196242
• https://github.com/apache/spark/runs/1595496305
• https://github.com/apache/spark/runs/1596309555

This is because the error message is different between test runs in GA (the error message seems to be truncated indeterministically) ,e.g.,

# https://github.com/apache/spark/runs/1592987501
Expected "...h status 127. Error:[ /bin/bash: some_non_existent_command: command not found]", but got "...h status 127. Error:[]" Result did not match for query #2

# https://github.com/apache/spark/runs/1593196242
Expected "...istent_command: comm[and not found]", but got "...istent_command: comm[]" Result did not match for query #2

The root cause of this indeterministic behaviour happening only in GA is not clear though, this test throws SparkException consistently even in GA. So, this PR proposes to make the test just check if it will be thrown when running it.

This PR comes from the @dongjoon-hyun comment: https://github.com/apache/spark/pull/29414/files#r547414513

Why are the changes needed?

Bugfix.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added tests.

[dongjoon-hyun]

Could you describe what was the root cause of the flakiness, @maropu ?

[AngersZhuuuu]

Seems some subprocess didn't write error message to stderrBuffer in checkFailureAndPropagate

[maropu]

Seems some subprocess didn't write error message to stderrBuffer in checkFailureAndPropagate

hm, that might be so because the error message seems to be truncated indeterministically. But, I've checked test logs in both GA/Jenkins, then it seems this flakiness happens only in GA tests. So, I'm not really sure about why...

[AngersZhuuuu]

Seems some subprocess didn't write error message to stderrBuffer in checkFailureAndPropagate

hm, that might be so because the error message seems to be truncated indeterministically. But, it seems this flakiness happens only in GA tests. So, I'm not really sure about why...

Not sure +1...

[maropu]

Could you describe what was the root cause of the flakiness, @maropu ?

Yea, thanks for the comment, @dongjoon-hyun and I've updated the description. I actually don't know what the root cause is now. So, the current approach of this PR just rewrote the test in a safer way. Any suggestion?

[dongjoon-hyun]

Then, could you add a few empty commits in this PR to verify the stableness, @maropu ?

[HyukjinKwon]

Looks good. +1 for @dongjoon-hyun's suggestion to trigger few more times with empty commits.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37849/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37849/

[SparkQA]

Test build #133253 has finished for PR 30896 at commit b29d6a1.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. (Pending GitHub Action sql - slow tests results)
Thank you for the investigation and this quick fix.

[SparkQA]

Test build #133245 has finished for PR 30896 at commit d205a13.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master.

[maropu]

Thanks, all!

[SparkQA]

Test build #133251 has finished for PR 30896 at commit b38541f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.