17.11 - error output is treated as error and fails tests

[birarroshan]

ℹ️ workarounds

Set ForwardStandardOutput to false in your runsettings:

<RunSettings>
  <RunConfiguration>
    <ForwardStandardOutput>false</ForwardStandardOutput>
  </RunConfiguration>
</RunSettings>

or

set VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING=1 environment variable.

---

Description

We are running Appium-WinAppdriver UI tests with ADO pipeline. After upgrading to 17.11.0, VStest task is reporting appium debug logs as error.

Steps to reproduce

1. Install 17.11.0 for VSTestPlatform - Latest Stable is 17.11.0

• task: VisualStudioTestPlatformInstaller@1
  inputs:
  packageFeedSelector: 'nugetOrg'
  versionSelector: 'latestStable'

2. Run test using
   • task: VSTest@2
     inputs:
     testSelector: 'testAssemblies'
     testAssemblyVer2: "Test.dll"
     searchFolder: $(Pipeline.Workspace)
     testFiltercriteria: "$(testFilterExpression)"
     vsTestVersion: 'toolsInstaller'

Expected behavior

Tests should run as usual without any error logs

Actual behavior

Appium DEBUG logs are being reported as Error.

Note - We have not enabled any debug logging for appium

Also, specFlow logs are reported as Warning.

Diagnostic logs

Could not collect Diagnostic logs. Will upload it one collected.

Environment

OS - Hosted Pool with windows 10 base image

Aditional Notes -

ADO job is passing, but all errors are reported under Error tab of pipeline run

[shapea]

I can confirm the same issue in vstest.console.exe as of version 17.11.0. As soon as the update was released, I had ADO build pipelines with no other changes start to fail with stderr output. Digging further into it, there seems to be a bug with some of the new output forwarding changes in the 17.11.0 release as noted here: https://github.com/microsoft/vstest/releases.

@nohwnd if bugs were ducks, this certainly walks, talks and quacks like a duck. From what I can tell, it appears that stdout output from tests gets erroneously forwarded to stderr instead. This causes benign test output to become errors. I can even verify this by manually calling vstest.console.exe with tests exhibiting this behaving and sending stderr output to nul (i.e. 2> nul).

I managed to sleuth out a workaround for this issue using the information here: #4998

For folks impacted by this, it seems you can update your test .runsettings to include this to disable the fowarding causing the issues:

<RunSettings>
  <RunConfiguration>
    <ForwardStandardOutput>false</ForwardStandardOutput>
  </RunConfiguration>
</RunSettings>

@birarroshan this may help unblock you, it did for me in ADO for the same reason. :)

[nohwnd]

The change was introduced here:

135b4f2#diff-d494c9e9058f3e044a79a007c308ad13da804fd3fb6d404462c09e3cd8003e50R59

This change then gets enabled to default in 17.11.

The ForwardStandardOutput option set to false is effective because it prevents forwarding the error stream as error messages.

it appears that stdout output from tests gets erroneously forwarded to stderr instead.

The appium logs go into error stream, not into standard output. All the streams get forwarded correctly, but you are right that this is not the desired behavior and the error stream should be sent as info messages instead, not as errors, because applications like to use error stream as output for logs, or info messages.

@birarroshan there is also VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING=1 env variable, if that is easier for you to set. as documented in #4998

[nohwnd]

Re-opening for awareness.

This bug affects VS17.11, TestPlatform 17.11, net8.0, and all newer versions.

Each product has it's own servicing calendar, will update this thread when fixes are released.

• release 17.11.1 to nuget
• insert 17.11.1 in VS
• insert 17.11.1 into net8
• insert 17.12.x into net9
• insert 17.12.x into net10

[shapea]

@nohwnd in my case it's not coming from Appium, but rather an internal-only MS thing.

Edit: It took a while to pull in symbols and sources to step deeper into their native pinvoke'd code, but it's doing the same sort of thing and writing non-error output to stderr as well.

[nohwnd]

@nohwnd in my case it's not coming from Appium, but rather an internal-only MS thing.

Edit: It took a while to pull in symbols and sources to step deeper into their native pinvoke'd code, but it's doing the same sort of thing and writing non-error output to stderr as well.

Thanks for confirming.