Crash in System.Net.WebSockets.Client - Illegal memory access

[karelz]

8/9:

• 13x hits in Runfo last 120 days (older crashes than PR build 1925920 on 8/5 are from different reasons)
• 18x hits in Kusto 8/5-8/9 (no crashes failures 6/9-8/4 at all)

Failures are on Linux only, with mix of architectures - arm64, amd64. Regression started on 8/5.

2 notable failures are on Rolling runs:

• 8/8 AM - 1929089
• 8/9 AM - 1931140

let crashes = (friendlyNamePrefix : string, includePR : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').WorkItems
    | where FriendlyName startswith friendlyNamePrefix
    //| where Queued > ago(7d)
    | where Status == "BadExit"
    | where ExitCode  == 139
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")) or (Branch startswith 'refs/heads/release/6.0'))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    | project-away PropertiesJson
    | order by Finished desc
};
crashes('System.Net.WebSockets.Client', true);

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

8/9:

• 13x hits in Runfo last 120 days (older crashes than PR build 1925920 on 8/5 are from different reasons)
• 18x hits in Kusto 8/5-8/9 (no crashes failures 6/9-8/4 at all)

Failures are on Linux only, with mix of architectures - arm64, amd64. Regression started on 8/5.

2 notable failures are on Rolling runs:

• 8/8 AM - 1929089
• 8/9 AM - 1931140

let crashes = (friendlyNamePrefix : string, includePR : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').WorkItems
    | where FriendlyName startswith friendlyNamePrefix
    //| where Queued > ago(7d)
    | where Status == "BadExit"
    | where ExitCode  == 139
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")) or (Branch startswith 'refs/heads/release/6.0'))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    | project-away PropertiesJson
    | order by Finished desc
};
crashes('System.Net.WebSockets.Client', true);

Author:	karelz
Assignees:	-
Labels:	area-System.Net, os-linux
Milestone:	7.0.0

[CarnaViire]

#73222 was merged on 8/5 so it might be related. The product change in the PR is minimal, but it added a bunch of tests.

cc @greenEkatherine

[jkotas]

This is likely the same root cause as #69125. See my comment at #69125 (comment) . You can confirm it by looking at the stacktrace of the crash using unmanaged debugger (lldb, gdb or windbg).

I would expect that any test that depend on sockets is going to hit #69125 with some frequency.

[wfurt]

I'm looking into it...

[hoyosjs]

Seen in https://dev.azure.com/dnceng/public/_build/results?buildId=1941374&view=results, but that stack is the same as #69125

[karelz]

Yes, we expect those 2 are dupes -- it was pending verification from @wfurt

[wfurt]

There is no failure in main since #73972 was merged.

[karelz]

Fixed in 8.0 (main) in PR #73972 and in 7.0 (RC1) in PR #74367.