GC_Features._HeapExpansion_bestfit_1_bestfit_1_._HeapExpansion_bestfit_1_bestfit_1_sh failing on coreclr-ci

[tarekgh]

https://mc.dot.net/#/user/coreclr-pulls/pr~2Fdotnet~2Fcoreclr~2Frefs~2Fpull~2F22331~2Fmerge/test~2Ffunctional~2Fcli~2Fpri0/20190204.734/workItem/GC.Features/analysis/xunit/GC_Features._HeapExpansion_bestfit_1_bestfit_1_~2F_HeapExpansion_bestfit_1_bestfit_1_sh

 Xunit.Sdk.TrueException
Ubuntu.1404.Arm32.Open-arm-Checked-no_tiered_compilation
Get Repro environment
Unhandled Exception of Type Xunit.Sdk.TrueException
Message :

Assert failure(PID 16516 [0x00004084], Thread: 16516 [0x4084]): (chosen_power2 && (i == 0)) || (!chosen_power2) && (i < free_space_count)
    File: /__w/2/s/src/gc/gc.cpp Line: 9084
    Image: /ssd/dotnetbuild/work/a37e46c3-25da-44c7-8522-334270327137/Payload/corerun

/ssd/dotnetbuild/work/a37e46c3-25da-44c7-8522-334270327137/Work/1c02bafd-705e-4276-af6c-e31b0a86a880/Exec/HeapExpansion/bestfit_1/bestfit_1.sh: line 251: 16516 Aborted                 $_DebuggerFullPath \"$CORE_ROOT/corerun\" $ExePath $CLRTestExecutionArguments
/n/nReturn code:      1/nRaw output file:      /ssd/dotnetbuild/work/a37e46c3-25da-44c7-8522-334270327137/Work/1c02bafd-705e-4276-af6c-e31b0a86a880/Exec/Reports/GC.Features/HeapExpansion/bestfit_1/bestfit_1.output.txt/nRaw output:/nBEGIN EXECUTION
/ssd/dotnetbuild/work/a37e46c3-25da-44c7-8522-334270327137/Payload/corerun bestfit_1.exe 1 1000 50000
Using random seed: 2062910439
Took 6861 iterations to reach steady state
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test Harness Exitcode is : 1
/nTo run the test:/n> set CORE_ROOT=/ssd/dotnetbuild/work/a37e46c3-25da-44c7-8522-334270327137/Payload/n> /ssd/dotnetbuild/work/a37e46c3-25da-44c7-8522-334270327137/Work/1c02bafd-705e-4276-af6c-e31b0a86a880/Exec/HeapExpansion/bestfit_1/bestfit_1.sh/n
Expected: True
Actual:   False
Stack Trace :
   at GC_Features._HeapExpansion_bestfit_1_bestfit_1_._HeapExpansion_bestfit_1_bestfit_1_sh() in /__w/1/s/bin/tests/Linux.arm.Checked/TestWrappers/GC.Features/GC.Features.XUnitWrapper.cs:line 319

[echesakov]

Also happened in Jenkins CI
https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/arm_cross_checked_ubuntu_no_tiered_compilation_innerloop_flow_prtest/7238/

      GC_Features._HeapExpansion_bestfit_1_bestfit_1_._HeapExpansion_bestfit_1_bestfit_1_sh [FAIL]
        
        Assert failure(PID 9853 [0x0000267d], Thread: 9853 [0x267d]): (chosen_power2 && (i == 0)) || ((!chosen_power2) && (i < free_space_count))
            File: /mnt/j/workspace/dotnet_coreclr/master/arm_cross_checked_ubuntu_innerloop_prtest/src/gc/gc.cpp Line: 9084
            Image: /ssd/j/workspace/dotnet_coreclr/master/jitstress/arm_cross_checked_ubuntu_no_tiered_compilation_innerloop_tst_prtest/bin/tests/Linux.arm.Checked/Tests/Core_Root/corerun
        
        /ssd/j/workspace/dotnet_coreclr/master/jitstress/arm_cross_checked_ubuntu_no_tiered_compilation_innerloop_tst_prtest/bin/tests/Linux.arm.Checked/GC/Features/HeapExpansion/bestfit_1/bestfit_1.sh: line 259:  9853 Aborted                 $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"
        /n/nReturn code:      1/nRaw output file:      /ssd/j/workspace/dotnet_coreclr/master/jitstress/arm_cross_checked_ubuntu_no_tiered_compilation_innerloop_tst_prtest/bin/tests/Linux.arm.Checked/GC/Features/Reports/GC.Features/HeapExpansion/bestfit_1/bestfit_1.output.txt/nRaw output:/nBEGIN EXECUTION
        /ssd/j/workspace/dotnet_coreclr/master/jitstress/arm_cross_checked_ubuntu_no_tiered_compilation_innerloop_tst_prtest/bin/tests/Linux.arm.Checked/Tests/Core_Root/corerun bestfit_1.exe '1' '1000' '50000'
        Using random seed: 1599727688
        Took 7950 iterations to reach steady state
        Expected: 100
        Actual: 134
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        /nTo run the test:/n> set CORE_ROOT=/ssd/j/workspace/dotnet_coreclr/master/jitstress/arm_cross_checked_ubuntu_no_tiered_compilation_innerloop_tst_prtest/bin/tests/Linux.arm.Checked/Tests/Core_Root/n> /ssd/j/workspace/dotnet_coreclr/master/jitstress/arm_cross_checked_ubuntu_no_tiered_compilation_innerloop_tst_prtest/bin/tests/Linux.arm.Checked/GC/Features/HeapExpansion/bestfit_1/bestfit_1.sh/n
        Expected: True
        Actual:   False
        Stack Trace:

[BruceForstall]

@Maoni0 Have you seen this one? Does it need to be marked Milestone=3.0? or Future?

[Maoni0]

I have not seen this. how easy/hard is it to repro?

[BruceForstall]

@Maoni0 I don't believe it happens regularly in the CI

[echesakov]

The test has failed 8 times for the last week

Kusto query to see all the failures

TestResults
| where Method contains '_HeapExpansion_bestfit_1_bestfit_1_sh' and Result == 'Fail'
| join kind=inner (Jobs) on JobId 
| where Queued > now(-7d) 

[echesakov]

An example of the recent failure,
https://dev.azure.com/dnceng/public/_build/results?buildId=144306&view=ms.vss-test-web.build-test-results-tab

https://mc.dot.net/#/user/coreclr-ci/pr~2Fdotnet~2Fcoreclr~2Frefs~2Fpull~2F23751~2Fmerge/test~2Ffunctional~2Fcli~2F/20190405.78/workItem/GC.Features/analysis/xunit/GC_Features._HeapExpansion_bestfit_1_bestfit_1_~2F_HeapExpansion_bestfit_1_bestfit_1_sh