Fail on no test file for PublishTestResults@2

[derekendres]

Note

Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo

For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

If you have an issue or request for the Azure Pipelines service, use developer community instead:

https://developercommunity.visualstudio.com/spaces/21/index.html )

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Feature

Enter Task Name: PublishTestResults@2

list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

Environment

• Server - Azure Pipelines or TFS on-premises?

  • If using Azure Pipelines, provide the account name, team project name, build definition name/build number:

https://dev.azure.com/osieng/engineering/_build/results?buildId=103053&view=logs&j=846ec57a-2bea-5833-7752-007160b95b9a&t=d1056ef1-5a88-52db-be24-b861f9682b72

• Agent - Hosted or Private:

  • If using Hosted agent, provide agent queue name:

    linux:
    imageName: ubuntu-latest
    windows:
    imageName: windows-latest

Issue Description

When publishing a test result, if there is no test result file, this task gives a warning that is eaten by a success indication. It should flag as a warning that it didn't actually publish the result and ideally there should be an option to fail if the test file doesn't exist.

Task logs

Starting: Publish test results

Task : Publish Test Results
Description : Publish test results to Azure Pipelines
Version : 2.171.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-test-results

/usr/bin/dotnet --version
3.1.301
##[warning]No test result files matching **/test-*.xml were found.
Finishing: Publish test results

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

[Insert error from the logs here for a quick overview]

[csurfleet]

This would be incredibly useful, we're using bash scripts between each test task at the moment

[BarsikV]

My team and I would appreciate this feature very much, too!

This would be incredibly useful, we're using bash scripts between each test task at the moment

Would you mind sharing your script?

[bneumann]

This would be incredibly useful, we're using bash scripts between each test task at the moment

Yeah if I need to script my way around problems like that I really don't need these tasks at all. I prefer a fix for this :)

[navin22]

Adding @shailesh-sk

[xncbf]

Anybody working on this issue? This option is essential

[walliski]

Coming in yet half year later.

This would be really essential for the pipelines I am working on. In the current setup I want to do, I would not want to fail the build based on the exit status of the tests step, but continue and later use the published test results further down in the pipeline. This means that both this task, and the Powershell task that runs the tests is set to not fail the build on failure. To safeguard from the testrunner failing completely and not running any tests at all, I would need this task to fail in case it cannot find any test result files!

[jmhon08]

I would like to see this as well. @shailesh-sk the PR is waiting your review

[shailesh-sk]

@jmhon08 PR looks good to me, added upadhyayap@github.com to cherry pick this change.

[ghost]

A bit late to the party, but I'm looking forward to seeing this added as well.

[jmhon08]

@upadhyayap any idea when this will be merged?

[walliski]

@shailesh-sk @upadhyayap

Did we get anywhere with #14868 ?

[mathiash98]

This is an essential feature... It should at least make a warning if not fail completely at the "Publish Test Results task"

[LynxJ]

Would really like to see this feature implemented

[ColeSiegelTR]

Also looking for this, seems fundamental

[mathiash98]

We ended up using follow bash script tasks which first check if test result file exists, then check if number of results are more than 0:

We rely on running tests inside a docker compose as we also test with database. But if you purely rely on the VSTest task you can use the argument minimumExpectedTests https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/vstest?view=azure-devops#arguments to fail if there are no tests run.

- bash: |
    if [ $(test_results_host_folder)**/*.trx ] && [ $(grep -E "<UnitTestResult" $(test_results_host_folder)**/*.trx -c) -gt 0 ]; then
      echo "##vso[task.setVariable variable=TESTRESULTFOUND]true"
    fi
  displayName: Check if test results file & results >= 1
- script: |
      echo No test result found
      exit 1
  displayName: No test result found
  condition: ne(variables.TESTRESULTFOUND, 'true')

[ColeSiegelTR]

been another 4 months any plans to fix this?

[b055man]

Coming in yet half year later.

This would be really essential for the pipelines I am working on. In the current setup I want to do, I would not want to fail the build based on the exit status of the tests step, but continue and later use the published test results further down in the pipeline. This means that both this task, and the Powershell task that runs the tests is set to not fail the build on failure. To safeguard from the testrunner failing completely and not running any tests at all, I would need this task to fail in case it cannot find any test result files!

The problem is the PublishTestResults succeeds not only if there are no test results but also if it fails to parse the test results, e.g. if a failed test case produces output that cannot be parsed.
This's just happened in our environment when some test cases were failing but this was not captured by the pipeline.

Yes, you can make a workaround to address this (like check and store result of the test command and fail the pipeline later etc.) but that's only if you know that the PublishTest task is unreliable beforehand.

To me this sounds like a bug, not an enhancement!

[danon]

I think it would make more sense if the job would fail by default, and maybe add an option to explicitly silence it.

[willieowens]

Starting: Publish test result to ADO
==============================================================================
Task         : Publish Test Results
Description  : Publish test results to Azure Pipelines
Version      : 2.203.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-test-results
==============================================================================
/usr/bin/dotnet --version
6.0.302
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
No Result Found to Publish '/home/vsts/work/1/s/cypress-results/test-output-57f4d8778c4987bda6a1790eaa703782.xml'.
Async Command Start: Publish test results
Publishing test results to test run '376211378'.
TestResults To Publish 108, Test run id:376211378
Test results publishing 108, remaining: 0. Test run id: 376211378
Published Test Run : https://msasg.visualstudio.com/XXXXXXXX/_TestManagement/Runs?runId=376211378&_a=runCharts
##[warning]Intermittent failure attempting to call the restapis System.Threading.Tasks.Task`1[Microsoft.TeamFoundation.TestManagement.WebApi.TestRunStatistic] <GetTestRunSummaryAsync>b__0(). Retry attempt 0. Exception: Run summary is not available, It is recommended to try again after 1 sec. 
##[warning]Intermittent failure attempting to call the restapis System.Threading.Tasks.Task`1[Microsoft.TeamFoundation.TestManagement.WebApi.TestRunStatistic] <GetTestRunSummaryAsync>b__0(). Retry attempt 1. Exception: Run summary is not available, It is recommended to try again after 1 sec. 
##[warning]Intermittent failure attempting to call the restapis System.Threading.Tasks.Task`1[Microsoft.TeamFoundation.TestManagement.WebApi.TestRunStatistic] <GetTestRunSummaryAsync>b__0(). Retry attempt 2. Exception: Run summary is not available, It is recommended to try again after 1 sec. 
TestRunDataPublisher.DoesRunsContainsFailures: Exception occured using test run summary api System.AggregateException: One or more errors occurred. (Run summary is not available, It is recommended to try again after 1 sec.)
 ---> Microsoft.TeamFoundation.TestManagement.WebApi.TestObjectNotFoundException: Run summary is not available, It is recommended to try again after 1 sec.
   at Microsoft.TeamFoundation.TestClient.PublishTestResults.RetryHelper.Retry[T](Func`1 action, Func`2 timeDelayInterval, Func`2 shouldRetryOnException)
   at Microsoft.TeamFoundation.TestClient.PublishTestResults.TestRunPublisher.GetTestRunSummaryAsync(String projectName, Int32 testRunId, Boolean allowRetry, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Microsoft.VisualStudio.Services.Agent.Worker.TestResults.TestRunDataPublisherHelper.DoesRunsContainsFailures(IList`1 testRuns, String projectName)
##[warning]TestRunDataPublisher.DoesRunsContainsFailures: Validating if test run is failed or succeeded based on summary where calulation of summary is not in completed state.
TestRunPublisher.IsTestRunFailed: Getting test run summary with run id: 376211378 using stats api
TestRunPublisher.IsTestRunFailed: checking if test run is failed using run summary input
Async Command End: Publish test results
Finishing: Publish test result to ADO

Somewhat different, but we just had a live site because although many of our tests failed, this task had some internal error reporting/fetching the test results. Not good... We need to correct the error handling in this task.

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[walliski]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[AurimasNav]

I'd love to see this fail by default if no test files are found.

[jamesstojan1]

This would be really helpful. The default assumption when publishing non existent files should be to error out.

[coffeebe4code]

nearly 3 years this has been opened?

Everyone going to be here for its 3rd birthday in July?

[dts12263]

I contacted Microsoft support through my work and was suggested to simply use the bash workaround.....I also share this frustration. So it does not seem like there are any plans to fix this bug. It's astonishing Microsoft is willingly letting this bug persist and relying on workarounds suggested by other users. Then they left me with a survey asking how well they addressed my problem!! Hysterical

[abatishchev]

Bump.

[xncbf]

I got tired of waiting and found the solution myself.
The root cause I've encountered is that the failure signal appears to be ignored if there is another command on the next line that runs pytest.
I solved it by receiving the signal from the line immediately following the test and processing it directly, as follows.

pipenv run pytest --junitxml=junit/test-results.xml --cov=. --cov-report=html
EXIT_CODE=$?
if [ $EXIT_CODE -ne 0 ]; then
exit $EXIT_CODE
fi

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[ChristianStadelmann]

This issue is not stale, but I guess the bot is…

[KevinPartingtonTR]

Microsoft folks, if this is something you don't plan to fix, you should close this issue as not planned so we know. Thanks!

[dts12263]

[like] Dustin Sauriol reacted to your message:

…

________________________________ From: Kevin Partington ***@***.***> Sent: Monday, March 18, 2024 7:44:03 PM To: microsoft/azure-pipelines-tasks ***@***.***> Cc: Dustin Sauriol ***@***.***>; Comment ***@***.***> Subject: Re: [microsoft/azure-pipelines-tasks] Fail on no test file for ***@***.*** (#13275) [External Sender] Microsoft folks, if this is something you don't plan to fix, you should close this issue as not planned so we know. Thanks! — Reply to this email directly, view it on GitHub<#13275 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AU2YHMPMJCOSZW2XHNIVOELYY47YHAVCNFSM4OYRLHC2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBQGQ3TQMBWGIYA>. You are receiving this because you commented.Message ID: ***@***.***>

[riQQ]

This is available with version 2.228.0+ via setting the input failTaskOnMissingResultsFile to true. It was implemented by #18931 and is also documented on learn.microsoft.com.

- task: PublishTestResults@2
  inputs:
    failTaskOnMissingResultsFile: true # boolean. Fail if no result files are found. Default: false.