Disable Xharness telemetry for wasm

[ilonatommy]

To prevent a trash log from getting into Wasm build log:

+ /usr/bin/python3 -u /datadisks/disk1/work/B43F097B/w/A1780928/u/xharness-event-processor.py
/usr/bin/python3: can't open file '/datadisks/disk1/work/B43F097B/w/A1780928/u/xharness-event-processor.py': [Errno 2] No such file or directory

This log is not the real error and might be confusing when examining the failure.

Example of a log with this problem.

Reason for the missing file explained: #66698 (comment).

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

To prevent some trash logs from getting into Wasm build log, such as:

+ /usr/bin/python3 -u /datadisks/disk1/work/B43F097B/w/A1780928/u/xharness-event-processor.py
/usr/bin/python3: can't open file '/datadisks/disk1/work/B43F097B/w/A1780928/u/xharness-event-processor.py': [Errno 2] No such file or directory

These logs are not the real error and might be confusing when examining the failure.

Example of a log with this problem.

Author:	ilonatommy
Assignees:	-
Labels:	area-Infrastructure-libraries
Milestone:	-

[radical]

What other messages will this disable? The better fix would be to remove the call to xharness-event-processor.py if it doesn't exist.

[premun]

@radical https://github.com/dotnet/arcade/blob/677ad5e2a0837322f132e57c8fa920825707d528/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/XHarnessRunner.targets#L80-L81

There is one pre and one post command that this disables. I'd rather leave the error in because if the telemetry should be on, we should see it being tried to send.

There is no other effect on your scenarios.

[radical]

@radical dotnet/arcade@677ad5e/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/XHarnessRunner.targets#L80-L81

There is one pre and one post command that this disables. I'd rather leave the error in because if the telemetry should be on, we should see it being tried to send.

There is no other effect on your scenarios.

Why does the file not exist? Does it work for non-wasm cases? This suggests that it would get copied, correct?

[premun]

This suggests that it would get copied, correct?

There is a piece or two missing in WASM flows actually.

1. Helix work item creation

The code path you link is called from CreateXHarnessAndroidWorkItems and CreateXHarnessAppleWorkItems. These are invoked when user defines XHarnessApkToTest and XHarnessAppBundleToTest MSBuild items (docs for these).

In those cases where you specify these items, the SDK takes over the creation of HelixWorkItem. For WASM flows, this is not implemented and you only request the InstallXHarnessCli functionality which pre-installs XHarness (common part for all platforms).

2. Collect diagnostic data in XHarness.WASM

The second piece that is missing is a little bit of code in the WASM XHarness part where we are not setting the telemetry data. Example for Apple is here: https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.CLI/Commands/Apple/AppleAppCommand.cs#L56-L58
Here we need to decide what to collect, such as Target can be the browser, OS can be the host operating system...

Once we have data to put into diagnostics.json and once we have xharness-event-reporter.py on the Helix agent, rest will start working (WASM telemetry appears in Kusto).

In case you want to think about the telemetry in the future, probably you should know that most of the data we store is in a property bag in Kusto. So even if the dimensions in IDiagnosticsData don't match what you would like to store, we can totally talk extend it and add custom fields for WASM flows. In Kusto we can parse this property bag and filter based on it too.

[radical]

Thank you for the detailed response, @premun! This looks interesting, and there is useful information that we can get through it, in the future :) For now, this PR sounds fine then 👍

[radical]

@ilonatommy Could you include a link to @premun's comment in the PR description?

[radical]

Good thinking, @ilonatommy!

[premun]

@radical in case you want to think about the telemetry in the future, probably you should know that most of the data we store is in a property bag in Kusto. So even if the dimensions in IDiagnosticsData don't match what you would like to store, we can totally talk extend it and add custom fields for WASM flows. In Kusto we can parse this property bag and filter based on it too.

Apart from these properties, we collect 2 metrics - exit code and duration of the XHarness command. I will update my comment with this piece too.