[solved] TestPlatform 17.7.0 fails to run in Azure DevOps pipeline

[jsdeshazer]

✅ This issue was mitigated by unlisting the version from nuget.org.

Description

Steps to reproduce

What steps can reproduce the defect?
My Azure build pipeline has the Visual Studio Test Platform Installer task and was always pulling the newest version.
The build started failing a little bit after 17.7.0 got updated. When trying to run the latest installer in the test assemblies task.
After changing the version to be specifically 17.6.3 it worked.

Expected behavior

Not to fail during the build.

Actual behavior

Failed to create a TestPlatformVersion object

[dr4zz]

Seeing the same issue

DtaExecutionHost version 19.210.32906.4.
##[error]TestExecution encountered unexpected failures Microsoft.VisualStudio.TestService.Common.TestPlatformVersionException: Failed to create a TestPlatformVersion object. Check inner exception for more details ---> System.FormatException: Input string was not in a correct format.
   at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at Microsoft.VisualStudio.TestService.Common.TestPlatformVersion..ctor(String versionString)
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.TestService.Common.TestPlatformVersion..ctor(String versionString)
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.TestPlatformVersionFinder.IdentifyTestPlatformVersion(String vsTestConsoleExeParentFolder, Boolean debugLogs)
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.InputValidator.PopulateEnvironmentVariables()
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.InputValidator.ValidateAndFormatInputs()
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.Program.Main(String[] args)

[frek818]

@jsdeshazer We also use Azure DevOps. How did you pin the version to 17.6. Are you using the VSTest task? The docs show the VSTest task only accepts major version for the vsTestVersion parameter..

[jimmydelinea]

We also just hit this issue. Had to change every test pipeline to use specific version 17.6.3.

[VladimirNilov]

For Azure DevOps, you can include the preview version, 17.8.0-preview-23371-04 doesn't have this problem

[jsdeshazer]

@frek818 Sorry it should be 17.6.3 in the Test Platform Version

[tonyzahnle]

@jsdeshazer We also use Azure DevOps. How did you pin the version to 17.6. Are you using the VSTest task? The docs show the VSTest task only accepts major version for the vsTestVersion parameter..

You can use the VisualStudioTestPlatformInstaller task to specify a version installed on the agent, then have the test task set vsTestVersion: 'toolsInstaller'

[mslukebo]

My pipelines are also failing due to the same System.FormatException mentioned above.

[nohwnd]

I just unlisted 17.7.0 from Nuget, please someone let me know if it helped.

[ghazlitt]

Looks like the same issue from June?

#4544

[mslukebo]

I just unlisted 17.7.0 from Nuget, please someone let me know if it helped.

Unlisting has fixed my pipeline. It now pulls 17.6.3

2023-08-04T19:30:53.7891386Z Starting VsTest platform tools installer task.
2023-08-04T19:30:53.7894104Z ==============================================================================
2023-08-04T19:30:53.7911850Z ##[debug]Selected package feed: nugetOrg
2023-08-04T19:30:53.7912337Z ##[debug]Going via nuget org download flow.
2023-08-04T19:30:53.7916362Z ##[debug]Using the package source https://api.nuget.org/v3/index.json to get the Microsoft.TestPlatform nuget package.
2023-08-04T19:30:53.7918797Z Looking for the latest stable version of the Microsoft.Testplatform.
2023-08-04T19:30:53.7943085Z [command]D:\a\_tasks\VisualStudioTestPlatformInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7111\1.215.0\nuget.exe list packageid:Microsoft.TestPlatform -NonInteractive -Source https://api.nuget.org/v3/index.json
2023-08-04T19:30:59.2613307Z Microsoft.TestPlatform 17.6.3

[nohwnd]

Yes, I was just looking for that issue. The version generator was changed in the meantime to output semver v1 versions that the logic in DTA Host can handle. Or so we thought. Apparently it does not work for final versions.

Honestly I don't know why this was released on Friday evening, and I am sorry that it broke you.

If someone can let me know that hiding the package helped I will got back to my dinner and do post-mortem on Monday 🙏

[nohwnd]

Thanks @mslukebo. I will have another look later if there are more issues reported. I did not want to hide too many packages.

[mslukebo]

Looks like the same issue from June?

#4544

This is not likely the same issue. The linked issue was caused by the prelease suffix at the end of the version string. This error is happening when using the latest stable release, which does not contain a suffix.

[nohwnd]

The callstack looks very similar if not the same, at least to my tired eyes. 😅

[mslukebo]

The callstack looks very similar of not the same, at least to my tired eyes. 😅

Same callstack, but the input is malformed in different ways. For the prelease bug, the version parsing is unhappy about the -preview suffix.

If I inspect the logs for the VSTest, I notice the following:

17.6.3 says
2023-08-04T19:31:27.0505746Z ##[debug]Found file version info for vstest.console.exe present in C:\hostedtoolcache\windows\VsTest\17.6.3\x64\tools\net462\Common7\IDE\Extensions\TestPlatform to be 17.6.3

17.7.0 says
2023-08-04T17:23:57.2744683Z ##[debug]Found file version info for vstest.console.exe present in C:\hostedtoolcache\windows\VsTest\17.7.0\x64\tools\net462\Common7\IDE\Extensions\TestPlatform to be 17.7.0+246832ceaa77ef978eebaeac5199bc56731747a1

So the issue is that 17.7.0's version number contains the git hash (+246832ceaa77ef978eebaeac5199bc56731747a1) as a suffix, causing version parsing to fail.

[ghazlitt]

The callstack looks very similar of not the same, at least to my tired eyes. 😅

Same callstack, but the input is malformed in different ways. For the prelease bug, the version parsing is unhappy about the -preview suffix.

If I inspect the logs for the VSTest, I notice the following:

17.6.3 says 2023-08-04T19:31:27.0505746Z ##[debug]Found file version info for vstest.console.exe present in C:\hostedtoolcache\windows\VsTest\17.6.3\x64\tools\net462\Common7\IDE\Extensions\TestPlatform to be 17.6.3

17.7.0 says 2023-08-04T17:23:57.2744683Z ##[debug]Found file version info for vstest.console.exe present in C:\hostedtoolcache\windows\VsTest\17.7.0\x64\tools\net462\Common7\IDE\Extensions\TestPlatform to be 17.7.0+246832ceaa77ef978eebaeac5199bc56731747a1

So the issue is that 17.7.0's version number contains the git hash (+246832ceaa77ef978eebaeac5199bc56731747a1) as a suffix, causing version parsing to fail.

The preview version mentioned in the older issue contains a git hash as well:
17.7.0-preview.23280.1**+94103c3f3198fe6f1405e1cb8fe486d6cd7132fd**

A previous post here indicated that 17.8.0-preview-23371-04 version worked without error; seems the "+" preceding the hash or the separators "." might be the issue. I haven't tested with 17.8 preview though

[nohwnd]

Thanks for the pointers.

The code in the original issue was unhappy about the extra . in the suffix because it checked for 3 dots in the whole version. Now it probably is unhappy that version is not just numbers.

I know where the code is so I can have a look on monday. I also saw in docs of Arcade (our build system) that + part should not be emitted when option for semver1 is used (which we are using). One way or another we will have to fix on our side, and hope for future fix in DTA host.

[nohwnd]

Had a look on the code, it splits the version to 3 parts, the last part is "0+aaaaabbbbbcccc" which is not convertible to int. So we first need to figure out how to remove that part from the ProductVersion on the vstest.console.exe.

We now build through Arcade build system, which caused us this issue and so we might need to await a fix from them, if there is not a clean way to force the version.

[Flinte]

For Azure DevOps, you can include the preview version, 17.8.0-preview-23371-04 doesn't have this problem

How Do you do that? Trying to get my yml pipeline using 17.8

[nohwnd]

You would use Test platform installer task to install a selected version of TP, and then the vstest task would be configured to pick it up:

steps:
- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Visual Studio Test Platform Installer'
  inputs:
    versionSelector: latestPreRelease  # <------------------

- task: VSTest@3
  displayName: 'VsTest - testAssemblies'
  inputs:
    vsTestVersion: toolsInstaller # <------------------

Docs for the installer task are here, you can also choose specific version. Or a different nuget feed than NuGet.org

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/visual-studio-test-platform-installer-v1?view=azure-pipelines

[nohwnd]

This was fixed in 17.8