Not all package versions are returned when running running choco list --all-versions --exact

[jborean93]

What You Are Seeing?

In previous Chocolatey releases if you had a package installed with multiple versions, doing choco.exe list --all-versions --exact <package> would show all the versions installed. Since 0.10.14 (and 15), adding --exact will only show the latest version present.

What is Expected?

All versions will be displayed with --all-versions --exact.

How Did You Get This To Happen? (Steps to Reproduce)

Use the following script to create the test packages

Function New-ChocolateyTestPackage {
    <#
    .SYNOPSIS
    Creates a Chocolatey test package.

    .DESCRIPTION
    Creates a nupkg that can be used to test out Chocolatey installs and the arguments that are passed into it.

    .PARAMETER Name
    The name of the package.

    .PARAMETER Version
    The version of the package.

    .PARAMETER OutPath
    The directory to create the package in.

    .PARAMETER ArtifactPath
    The directory which the package is configured to create it's install artifact in.

    .PARAMETER Id
    A unique identifier for the package, this defaults to a new GUID.

    .EXAMPLE
    New-ChocolateyTestPackage -Name test-package -Version '0.1.0' -OutPath 'C:\Chocolatey' -ArtifactPath 'C:\Chocolatey'
    &choco.exe source add --name test-source --source 'C:\Chocolatey'
    &choco.exe install test-package --source test-source

    .NOTES
    This is purely designed for testing Chocolatey, not affialited in any way.
    #>
    [CmdletBinding()]
    Param (
        [Parameter(Mandatory=$true)]
        [System.String]
        $Name,

        [Parameter(Mandatory=$true)]
        [System.String]
        $Version,

        [Parameter(Mandatory=$true)]
        [System.String]
        $OutPath,

        [Parameter(Mandatory=$true)]
        [System.String]
        $ArtifactPath,

        [System.String]
        $Id = ([System.Guid]::NewGuid().ToString())
    )

    $package_nuspec = @'
<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2015/06/nuspec.xsd">
    <metadata>
    <id>{0}</id>
    <version>{1}</version>
    <title>{0}</title>
    <authors>Jordan Borean</authors>
    <description>Test for allow multiple</description>
    </metadata>
    <files>
    <file src="tools\**" target="tools" />
    </files>
</package>
'@
    
    $choco_install = @'
$ErrorActionPreference = 'Stop'

$package_name = $env:ChocolateyPackageName
$package_version = $env:ChocolateyPackageVersion
$install_path = "{0}\$package_name-$package_version.txt"
$id = "{1}"  # used as a unique identifier for the package

if ($env:ChocolateyAllowEmptyChecksums) {{
    $allow_empty_checksums = $true
}} else {{
    $allow_empty_checksums = $false
}}
if ($env:ChocolateyIgnoreChecksums) {{
    $ignore_checksums = $true
}} else {{
    $ignore_checksums = $false
}}
if ($env:ChocolateyForce) {{
    $force = $true
}} else {{
    $force = $false
}}
if ($env:ChocolateyForceX86) {{
    $force_x86 = $true
}} else {{
    $force_x86 = $false
}}
$timeout = $env:chocolateyResponseTimeout

$package_info = @{{
    allow_empty_checksums = $allow_empty_checksums
    force = $force
    force_x86 = $force_x86
    id = $id
    ignore_checksums = $ignore_checksums
    install_args = $env:ChocolateyInstallArguments
    name = $package_name
    package_params = Get-PackageParameters
    proxy_url = $env:ChocolateyProxyLocation
    timeout = $timeout
    version = $package_version
}}
$package_json = ConvertTo-Json -InputObject $package_info

[System.IO.File]::WriteAllText($install_path, $package_json)
'@

    $choco_uninstall = @'
$ErrorActionPreference = 'Stop'

$package_name = $env:ChocolateyPackageName
$package_version = $env:ChocolateyPackageVersion
$install_path = "{0}\$package_name-$package_version.txt"

if (Test-Path -LiteralPath $install_path) {{
    Remove-Item -LiteralPath $install_path -Force > $null
}}
'@

    $temp_package_dir = Join-Path -Path $OutPath -ChildPath "$($Name)-$($Version)"
    New-Item -Path $temp_package_dir -ItemType Directory > $null
    try {
        $temp_package_tools_dir = Join-Path -Path $temp_package_dir -ChildPath 'tools'
        New-Item -Path $temp_package_tools_dir -ItemType Directory > $null

        $nuspec_text = $package_nuspec -f ($Name, $Version)
        $install_text = $choco_install -f ($ArtifactPath, $Id)
        $uninstall_text = $choco_uninstall -f ($ArtifactPath)
    
        $utf8 = New-Object -TypeName System.Text.UTF8Encoding -ArgumentList $false
        $utf8_bom = New-Object -TypeName System.Text.UTF8Encoding -ArgumentList $true
        $nuspec_path = Join-Path -Path $temp_package_dir -ChildPath "$($Name).nuspec"
        [System.IO.File]::WriteAllText($nuspec_path, $nuspec_text, $utf8)
        [System.IO.File]::WriteAllText(
            (Join-Path -Path $temp_package_tools_dir -ChildPath 'chocolateyinstall.ps1'),
            $install_text, $utf8_bom
        )
        [System.IO.File]::WriteAllText(
            (Join-Path -Path $temp_package_tools_dir -ChildPath 'chocolateyUninstall.ps1'),
            $uninstall_text, $utf8_bom
        )

        &choco.exe pack --out "`"$OutPath`"" --no-progress --limit-output "`"$nuspec_path`""
    } finally {
        Remove-Item -LiteralPath $temp_package_dir -Force -Recurse
    }
}

$root_path = 'C:\chocolatey_testing'
if (Test-Path -LiteralPath $root_path) {
    Remove-Item -LiteralPath $root_path -Force -Recurse
}
New-Item -Path $root_path -ItemType Directory > $null

@('0.0.1', '0.1.0') | ForEach-Object -Process {
    New-ChocolateyTestPackage -Name sxs-test -Version $_ -OutPath $root_path -ArtifactPath $root_path
}

Once created run the following commands to install them

choco.exe source add --name test-source --source C:\chocolatey_testing
choco.exe install sxs-test --source test-source --yes
choco.exe install sxs-test --source test-source --version 0.0.1 --allow-multiple-versions --yes

Now run the following to get the list output

choco.exe --version
choco.exe list --all-versions --local-only
choco.exe list --all-versions --local-only --exact sxs-test

# Test that the packages are definitely installed
Get-Content -Path 'C:\chocolatey_testing\sxs-test-0.0.1.txt' -Raw
Get-Content -Path 'C:\chocolatey_testing\sxs-test-0.1.0.txt' -Raw

Output Log

On Chocolatey 0.10.13 this is the output

PS C:\Windows\system32> choco.exe --version
0.10.13
PS C:\Windows\system32> choco.exe list --all-versions --local-only
Chocolatey v0.10.13
chocolatey 0.10.13
sxs-test 0.1.0
sxs-test 0.0.1
3 packages installed.
PS C:\Windows\system32> choco.exe list --all-versions --local-only --exact sxs-test
Chocolatey v0.10.13
sxs-test 0.1.0
sxs-test 0.0.1
2 packages installed.
PS C:\Windows\system32>
PS C:\Windows\system32> # Test that the packages are definitely installed
PS C:\Windows\system32> Get-Content -Path 'C:\chocolatey_testing\sxs-test-0.0.1.txt' -Raw
{
    "ignore_checksums":  false,
    "id":  "80096ebe-0251-49be-a787-20cd334f0912",
    "install_args":  null,
    "force_x86":  false,
    "proxy_url":  null,
    "allow_empty_checksums":  false,
    "name":  "sxs-test",
    "force":  false,
    "version":  "0.0.1",
    "package_params":  {

                       },
    "timeout":  "2700000"
}
PS C:\Windows\system32> Get-Content -Path 'C:\chocolatey_testing\sxs-test-0.1.0.txt' -Raw
{
    "ignore_checksums":  false,
    "id":  "25294b9f-7f23-43f1-91b2-9ed26e5d571a",
    "install_args":  null,
    "force_x86":  false,
    "proxy_url":  null,
    "allow_empty_checksums":  false,
    "name":  "sxs-test",
    "force":  false,
    "version":  "0.1.0",
    "package_params":  {

                       },
    "timeout":  "2700000"
}

On Chocolatey 0.10.14+ this is the output

PS C:\Windows\system32> choco.exe --version
0.10.14
PS C:\Windows\system32> choco.exe list --all-versions --local-only
Chocolatey v0.10.14
chocolatey 0.10.14
sxs-test 0.1.0
sxs-test 0.0.1
3 packages installed.
PS C:\Windows\system32> choco.exe list --all-versions --local-only --exact sxs-test
Chocolatey v0.10.14
sxs-test 0.1.0
1 packages installed.
PS C:\Windows\system32>
PS C:\Windows\system32> # Test that the packages are definitely installed
PS C:\Windows\system32> Get-Content -Path 'C:\chocolatey_testing\sxs-test-0.0.1.txt' -Raw
{
    "ignore_checksums":  false,
    "id":  "80096ebe-0251-49be-a787-20cd334f0912",
    "install_args":  null,
    "force_x86":  false,
    "proxy_url":  null,
    "allow_empty_checksums":  false,
    "name":  "sxs-test",
    "force":  false,
    "version":  "0.0.1",
    "package_params":  {

                       },
    "timeout":  "2700000"
}
PS C:\Windows\system32> Get-Content -Path 'C:\chocolatey_testing\sxs-test-0.1.0.txt' -Raw
{
    "ignore_checksums":  false,
    "id":  "25294b9f-7f23-43f1-91b2-9ed26e5d571a",
    "install_args":  null,
    "force_x86":  false,
    "proxy_url":  null,
    "allow_empty_checksums":  false,
    "name":  "sxs-test",
    "force":  false,
    "version":  "0.1.0",
    "package_params":  {

                       },
    "timeout":  "2700000"
}

You can see that on choco --local-only --all-versions shows both versions installed but as soon as --exact is used it only shows the latest version.

Full Log Output

PS C:\Windows\system32> choco.exe list --all-versions --local-only --exact sxs-test --verbose --debug
Chocolatey v0.10.14
Chocolatey is running on Windows v 10.0.17763.0
Attempting to delete file "C:/ProgramData/chocolatey/choco.exe.old".
Attempting to delete file "C:\ProgramData\chocolatey\choco.exe.old".
Command line: "C:\ProgramData\chocolatey\choco.exe" list --all-versions --local-only --exact sxs-test --verbose --debug
Received arguments: list --all-versions --local-only --exact sxs-test --verbose --debug
RemovePendingPackagesTask is now ready and waiting for PreRunMessage.
Sending message 'PreRunMessage' out if there are subscribers...
[Pending] Removing all pending packages that should not be considered installed...
Performing validation checks.
Global Configuration Validation Checks:
 - Package Exit Code / Exit On Reboot = Checked
System State Validation Checks:
 Reboot Requirement Checks:
 - Pending Computer Rename = Checked
 - Pending Component Based Servicing = Checked
 - Pending Windows Auto Update = Checked
 - Pending File Rename Operations = Checked
 - Pending Windows Package Installer = Checked
 - Pending Windows Package Installer SysWow64 = Checked
The source 'https://chocolatey.org/api/v2/;C:\chocolatey_testing' evaluated to a 'normal' source type

NOTE: Hiding sensitive configuration data! Please double and triple
 check to be sure no sensitive data is shown, especially if copying
 output to a gist for review.
Configuration: CommandName='list'|
CacheLocation='C:\Users\vagrant\AppData\Local\Temp\chocolatey'|
ContainsLegacyPackageInstalls='True'|
CommandExecutionTimeoutSeconds='2700'|WebRequestTimeoutSeconds='30'|
Sources='https://chocolatey.org/api/v2/;C:\chocolatey_testing'|
SourceType='normal'|Debug='True'|Verbose='True'|Trace='False'|
Force='False'|Noop='False'|HelpRequested='False'|
UnsuccessfulParsing='False'|RegularOutput='True'|QuietOutput='False'|
PromptForConfirmation='True'|AcceptLicense='False'|
AllowUnofficialBuild='False'|Input='sxs-test'|AllVersions='True'|
SkipPackageInstallProvider='False'|Prerelease='False'|ForceX86='False'|
OverrideArguments='False'|NotSilent='False'|
ApplyPackageParametersToDependencies='False'|
ApplyInstallArgumentsToDependencies='False'|IgnoreDependencies='False'|
AllowMultipleVersions='False'|AllowDowngrade='False'|
ForceDependencies='False'|Information.PlatformType='Windows'|
Information.PlatformVersion='10.0.17763.0'|
Information.PlatformName='Windows Server 2016'|
Information.ChocolateyVersion='0.10.14.0'|
Information.ChocolateyProductVersion='0.10.14'|
Information.FullName='choco, Version=0.10.14.0, Culture=neutral, PublicKeyToken=79d02ea9cad655eb'|

Information.Is64BitOperatingSystem='True'|
Information.Is64BitProcess='True'|Information.IsInteractive='True'|
Information.UserName='vagrant'|
Information.UserDomainName='WIN-QCFRT8C2NP1'|
Information.IsUserAdministrator='True'|
Information.IsUserSystemAccount='False'|
Information.IsUserRemoteDesktop='False'|
Information.IsUserRemote='True'|
Information.IsProcessElevated='True'|
Information.IsLicensedVersion='False'|Information.LicenseType='Foss'|
Information.CurrentDirectory='C:\Windows\system32'|
Features.AutoUninstaller='True'|Features.ChecksumFiles='True'|
Features.AllowEmptyChecksums='False'|
Features.AllowEmptyChecksumsSecure='True'|
Features.FailOnAutoUninstaller='False'|
Features.FailOnStandardError='False'|Features.UsePowerShellHost='True'|
Features.LogEnvironmentValues='False'|Features.LogWithoutColor='False'|
Features.VirusCheck='False'|
Features.FailOnInvalidOrMissingLicense='False'|
Features.IgnoreInvalidOptionsSwitches='True'|
Features.UsePackageExitCodes='True'|
Features.UseEnhancedExitCodes='False'|
Features.UseFipsCompliantChecksums='False'|
Features.ShowNonElevatedWarnings='True'|
Features.ShowDownloadProgress='True'|
Features.StopOnFirstPackageFailure='False'|
Features.UseRememberedArgumentsForUpgrades='False'|
Features.IgnoreUnfoundPackagesOnUpgradeOutdated='False'|
Features.SkipPackageUpgradesWhenNotInstalled='False'|
Features.RemovePackageInformationOnUninstall='False'|
Features.ExitOnRebootDetected='False'|
Features.LogValidationResultsOnWarnings='True'|
Features.UsePackageRepositoryOptimizations='True'|
Features.ScriptsCheckLastExitCode='False'|ListCommand.LocalOnly='True'|
ListCommand.IdOnly='False'|ListCommand.IncludeRegistryPrograms='False'|
ListCommand.PageSize='25'|ListCommand.Exact='True'|
ListCommand.ByIdOnly='False'|ListCommand.ByTagOnly='False'|
ListCommand.IdStartsWith='False'|ListCommand.OrderByPopularity='False'|
ListCommand.ApprovedOnly='False'|
ListCommand.DownloadCacheAvailable='False'|
ListCommand.NotBroken='False'|
ListCommand.IncludeVersionOverrides='False'|
UpgradeCommand.FailOnUnfound='False'|
UpgradeCommand.FailOnNotInstalled='False'|
UpgradeCommand.NotifyOnlyAvailableUpgrades='False'|
UpgradeCommand.ExcludePrerelease='False'|
NewCommand.AutomaticPackage='False'|
NewCommand.UseOriginalTemplate='False'|SourceCommand.Command='unknown'|
SourceCommand.Priority='0'|SourceCommand.BypassProxy='False'|
SourceCommand.AllowSelfService='False'|
SourceCommand.VisibleToAdminsOnly='False'|
FeatureCommand.Command='unknown'|ConfigCommand.Command='unknown'|
ApiKeyCommand.Remove='False'|PinCommand.Command='unknown'|
OutdatedCommand.IgnorePinned='False'|Proxy.BypassOnLocal='True'|
_ Chocolatey:ChocolateyListCommand - Normal Run Mode _
Searching for package information
Running list with the following filter = 'sxs-test'
--- Start of List ---
Using 'C:\ProgramData\chocolatey\lib'.
- Supports prereleases? 'True'.
- Is ServiceBased? 'False'.
Package 'sxs-test' found on source 'C:\ProgramData\chocolatey\lib'
sxs-test 0.1.0
 Title: sxs-test | Published: 6/4/2019
 Number of Downloads: n/a | Downloads for this version: n/a
 Package url
 Chocolatey Package Source: n/a
 Tags:
 Software Site: n/a
 Software License: n/a
 Description: Test for allow multiple

--- End of List ---
1 packages installed.
Sending message 'PostRunMessage' out if there are subscribers...
Exiting with 0

[ghost]

+1 to have this fixed.

[wilsonge]

This is affecting us on the https://github.com/joomla/joomla-cms project too

At a quick diff - I believe that 97a171f is the commit responsible for this? It's returning when exact is passed in so the all versions flag is skipped

[Hackwar]

+1 to fix this. 🙃

[ferventcoder]

@wilsonge thank you for that - we are aware of what commit caused this. I believe if you turn off repository optimizations it should return the way it did before. It's worth a quick check as there is a way to make it work the way it did previously.

[ferventcoder]

And apologies for all affected - we didn't realize there was a want to return multiple versions of something when you were searching in this way, so it made sense for us to go for the repository optimizations route on this one.

[wilsonge]

So in our use case we're running the slight abomination choco install ((choco search php --exact --all-versions -r | select-string -pattern $phpMinorVersion | sort { [version]($_ -split '\|' | select -last 1) } -Descending | Select-Object -first 1) -replace '[php|]','')

Which basically allows us for a given minor PHP to install the latest available release (it's kinda a hack around there not being alias' for a major/minor release of a language).

Turning off repository optimizations doesn't seem to work either - still only get the latest package back - debug build: https://ci.appveyor.com/project/release-joomla/joomla-cms/builds/26488368/job/ptejxkh6f7xj2i42

[marc1706]

@wilsonge the solution right now seems to be to just downgrade to a version that actually works with regards to choco search:
choco install chocolatey -y --version 0.10.13 --allow-downgrade

[wilsonge]

That works :) thanks for the pointer!

[ferventcoder]

@wilsonge @marc1706 I'd use upgrade, not install. It might work specifying version like that, but it's not something we expect to work as install should noop if something is already installed.

[filips123]

Is there any update? Is it possibnle to fix this without downgrading Choco?

[ferventcoder]

This has been fixed at 1f0b866 in stable and merged into master at b3a7db5.

Apologies on timing - we would have liked to have fixed this sooner. We'll commit to putting a release together addressing this (and possibly a couple of other issues) in the next few weeks.

[wilsonge]

@ferventcoder hello. any ideas on when this release is due. I know covid and everything has made life tricky - but it's pretty much been a year since you said a couple of weeks - so at least an idea on timescales would be useful!

[gep13]

@wilsonge this is slated to go out in the 0.10.16 release, which I am actively working on just now. Having said that though, we don't put any firm dates on when this will ship.

[AdmiringWorm]

@wilsonge just wanted to additionally note that the latest beta release of chocolatey includes the fix for the problem described in this issue if you are interested.

https://community.chocolatey.org/packages/chocolatey/0.10.16-beta-20200806