[wasm] Use source built emsdk packages for the runtime build #100266

steveisok · 2024-03-26T01:12:03Z

This change moves away from using upstream emsdk to our source built packages as part of the wasm build.

steveisok · 2024-03-26T01:47:39Z

@akoeplinger I hand edited the eng/Version.xml/props. Do we have to do anything w/ darc to help resolve what's failing?

akoeplinger · 2024-03-26T07:42:01Z

It was just a typo :)

I'm wondering though whether we need to add one of these source-build .Intermediate dependencies too?

akoeplinger · 2024-03-26T07:43:53Z

src/mono/mono.proj

+
+    <ItemGroup>
+      <EmsdkFiles   Condition="'%(PackageReference.Identity)' != 'runtime.$(_portableHostOS)-$(BuildArchitecture).Microsoft.NETCore.Runtime.Wasm.Node.Transport' and '%(PackageReference.Identity)' != 'Microsoft.NET.Runtime.Emscripten.3.1.34.Python.win-$(BuildArchitecture)'"
+                    Include="$(NuGetPackageRoot)\$([System.String]::Copy(%(PackageReference.Identity)).ToLowerInvariant())\%(PackageReference.Version)\tools\**" />


I think instead of NuGetPackageRoot you should be able to use GeneratePackagePath on the PackageReference

I tried that and it gets complicated as the package names are based on other properties, so the generated property name is for example PkgMicrosoft_NET_Runtime_Emscripten_3_1_34_Cache_$(_portableHostOS)-$(BuildArchitecture). It would work much better if the package path was added to the PackageReference items instead.

src/mono/browser/runtime/package-lock.json

lewing · 2024-03-27T03:55:15Z

I want to start using the packages asap but I don't think we want to use individual props for each item here when the package name needs to change whenever we do an emsdk bump.

steveisok · 2024-03-27T15:31:08Z

I want to start using the packages asap but I don't think we want to use individual props for each item here when the package name needs to change whenever we do an emsdk bump.

Change the package name out of emsdk then?

Works for the most part out of the box, but fails on the part where we npm install after the libraries build. The reason for the failure is our node transport package is incomplete. If you replace it with a legit node, the build works fully.

…k. Streamline package deps

… and adjust a bunch of paths

…can't see npm without it

akoeplinger · 2024-05-06T19:23:06Z

eng/pipelines/common/templates/pipeline-with-resources.yml

@@ -101,7 +101,7 @@ extends:
        image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8

      browser_wasm:
-        image: mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-webassembly-20230913040940-1edc1c6
+        image: mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-webassembly


is there anything else left in the -webassembly image that we need or can we switch to the plain mariner image?

Wasmtime and WASI SDK (which we ignore at the moment)?

isn't that for the wasi leg? I think this only deals with browser

I have opened #102069 to try that. I am not sure whether we still run some tests with v8.

eng/Versions.props

lewing

The versions.props suggestion needs to land but it looks like we are almost there

eng/Versions.props

eng/Version.Details.xml

Co-authored-by: Larry Ewing <lewing@microsoft.com>

akoeplinger · 2024-05-13T09:11:15Z

eng/Versions.props

+    <runtimewinx64MicrosoftNETCoreRuntimeWasmNodeTransportPackageVersion>9.0.0-alpha.1.24175.1</runtimewinx64MicrosoftNETCoreRuntimeWasmNodeTransportPackageVersion>
+    <EmsdkPackageVersion>$(MicrosoftNETRuntimeEmscriptenVersion)</EmsdkPackageVersion>
+    <NodePackageVersion>$(runtimewinx64MicrosoftNETCoreRuntimeWasmNodeTransportPackageVersion)</NodePackageVersion>
+    <EmsdkVersion>3.1.34</EmsdkVersion>


please move these up next to MicrosoftNETWorkloadEmscriptenCurrentManifest90100TransportVersion

akoeplinger · 2024-05-13T09:12:59Z

src/mono/mono.proj

-      <InstallCmd Condition="'$(HostOS)' == 'windows'">powershell -NonInteractive -command &quot;&amp; $(InstallCmd); Exit $LastExitCode &quot;</InstallCmd>
-      <ActivateCmd Condition="'$(HostOS)' == 'windows'">powershell -NonInteractive -command &quot;&amp; $(ActivateCmd); Exit $LastExitCode &quot;</ActivateCmd>
-      <PythonCmd Condition="'$(HostOS)' == 'windows'and '$(TargetsBrowser)' == 'true'">setlocal EnableDelayedExpansion &amp;&amp; call &quot;$([MSBuild]::NormalizePath('$(EMSDK_PATH)', 'emsdk_env.bat'))&quot; &amp;&amp; !EMSDK_PYTHON!</PythonCmd>
+      <_EmsdkPaths Condition="'$(HostOS)' != 'windows'">


please move the scripts into separate files, they shouldn't be inside mono.proj

akoeplinger · 2024-05-13T09:13:51Z

src/mono/mono.proj

+                    Include="$(NuGetPackageRoot)\$([System.String]::Copy(%(PackageReference.Identity)).ToLowerInvariant())\%(PackageReference.Version)\tools\**" />
+      <NodeFiles    Condition="'%(PackageReference.Identity)' == 'runtime.$(_portableHostOS)-$(BuildArchitecture).Microsoft.NETCore.Runtime.Wasm.Node.Transport'"
+                    Include="$(NuGetPackageRoot)\$([System.String]::Copy(%(PackageReference.Identity)).ToLowerInvariant())\%(PackageReference.Version)\tools\$(_portableHostOS)-$(BuildArchitecture)\**" />
+      <PythonFiles  Condition="'$(HostOS)' == 'windows' and '%(PackageReference.Identity)' == 'Microsoft.NET.Runtime.Emscripten.3.1.34.Python.win-$(BuildArchitecture)'"


this shouldn't hardcode 3.1.34 here

…100266) * Use the emsdk transport packages to build wasm instead of a cloned emsdk Works for the most part out of the box, but fails on the part where we npm install after the libraries build. The reason for the failure is our node transport package is incomplete. If you replace it with a legit node, the build works fully. * Pull in node transport package as opposed to the one packaged in emsdk. Streamline package deps * Make paths friendly for windows * Work in windows transport packages, copy python to its own directory, and adjust a bunch of paths * Bump to latest version of node that contains icu * Add windows deps and add DOTNET_EMSCRIPTEN_NODE_PATH because windows can't see npm without it * Bump emscripten packages to the latest * Fix typos in Version.Details.xml * Switch out to the plain mariner container to ensure no EMSDK already exists * Revert back to the browser-wasm docker image and bump to the latest * Container type-o * Fix condition where the node path isn't set properly * Provision even when building offsets * Fix goofy paths * Update new template to use the latest browser-wasm container * Don't put python.exe on the path for windows, but the folder instead * So that was a bad idea. May have to have emsdk_env.cmd have two entries * Fix offsets generation for browser * Fix emsdk's python path It is not versioned anymore as we use the content of our nugets * Fix EMSDK_PATH on non-windows platforms * Do not add link flags to compile flags To avoid: C:\helix\work\correlation\build\wasm-shared\WasmApp.Common.targets(825,5): error : emcc: warning: linker setting ignored during compilation: 'EXPORT_ES6' [-Wunused-command-line-argument] [C:\helix\work\workitem\e\publish\ProxyProjectForAOTOnHelix.proj] C:\helix\work\correlation\build\wasm-shared\WasmApp.Common.targets(825,5): error : emcc: warning: linker setting ignored during compilation: 'EXPORT_EXCEPTION_HANDLING_HELPERS' [-Wunused-command-line-argument] [C:\helix\work\workitem\e\publish\ProxyProjectForAOTOnHelix.proj] * Set the emsdk paths relative to the script location * Escape few characters and fix the added script * Fix .emscripten script * Use EM_CONFIG intead of __file__ * Fix emsdk_env.cmd script * Revert "Do not add link flags to compile flags" This reverts commit eb19ade. * Revert changes in package-lock.json * Feedback + cleaning * More cleaning * Try not to use the replace in the EMSDK_PATH * Better property names * Use package and emsdk version properties Co-authored-by: Larry Ewing <lewing@microsoft.com> * Use the updated properties * Fix emsdk version * Use a single version of emsdk and take node version from flow * Update eng/Version.Details.xml Co-authored-by: Larry Ewing <lewing@microsoft.com> --------- Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Co-authored-by: Radek Doulik <radek.doulik@gmail.com> Co-authored-by: Radek Doulik <radekdoulik@gmail.com> Co-authored-by: Larry Ewing <lewing@microsoft.com>

steveisok requested review from maraf, akoeplinger, lewing, pavelsavara and marek-safar as code owners March 26, 2024 01:12

dotnet-issue-labeler bot added the area-Build-mono label Mar 26, 2024

dotnet-policy-service bot assigned steveisok Mar 26, 2024

steveisok requested a review from kg March 26, 2024 01:46

akoeplinger reviewed Mar 26, 2024

View reviewed changes

maraf reviewed Mar 26, 2024

View reviewed changes

src/mono/browser/runtime/package-lock.json Outdated Show resolved Hide resolved

This was referenced Mar 27, 2024

[ci] Playwright test times out/fails because target closed #99961

Closed

Browser SignalRClientTests failing in CI #100307

Closed

steveisok force-pushed the test-build-wasm branch from 6c6c402 to a55848d Compare March 29, 2024 14:53

steveisok added 7 commits April 2, 2024 16:00

Pull in node transport package as opposed to the one packaged in emsd…

411fcb5

…k. Streamline package deps

Make paths friendly for windows

1964d80

Work in windows transport packages, copy python to its own directory,…

b8e1d17

… and adjust a bunch of paths

Bump to latest version of node that contains icu

35d78ba

Add windows deps and add DOTNET_EMSCRIPTEN_NODE_PATH because windows …

86b0487

…can't see npm without it

Bump emscripten packages to the latest

bad7dd2

Fix emsdk version

8a71663

This was referenced Apr 29, 2024

[wasm][firefox] crit: OpenQA.Selenium.WebDriverException: Failed to decode response from marionette #101617

Open

AbbreviatedMonthGenitiveNames_Get_ReturnsExpected_HybridGlobalization failing with string mismatch #101634

Closed

Merge branch 'main' into test-build-wasm

afe1b08

build-analysis bot mentioned this pull request Apr 30, 2024

Tracking issue for "WORKLOAD TIMED OUT" #90309

Closed

Merge branch 'main' into test-build-wasm

1d4b923

build-analysis bot mentioned this pull request Apr 30, 2024

System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721

Closed

Merge branch 'main' into test-build-wasm

b46a6a9

This was referenced May 1, 2024

[wasm] Assertion: should not be reached at D:/a/_work/1/s/src/mono/mono/sgen/sgen-scan-object.h:93 in System.Reflection.Tests #95805

Closed

[browser][MT] fail in System.Reflection.RuntimeMethodInfo.InternalInvoke #100831

Open

Merge branch 'main' into test-build-wasm

e976299

akoeplinger reviewed May 6, 2024

View reviewed changes

lewing reviewed May 7, 2024

View reviewed changes

eng/Versions.props Outdated Show resolved Hide resolved

eng/Version.Details.xml Outdated Show resolved Hide resolved

lewing added 2 commits May 7, 2024 11:37

Use a single version of emsdk and take node version from flow

48b0006

Merge branch 'main' into test-build-wasm

6513926

lewing approved these changes May 7, 2024

View reviewed changes

Merge branch 'main' into test-build-wasm

8672523

build-analysis bot mentioned this pull request May 8, 2024

arm32 fails in CI with "/lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.34' not found" #102030

Closed

radekdoulik and others added 2 commits May 9, 2024 11:15

Update eng/Version.Details.xml

7c1b7c3

Co-authored-by: Larry Ewing <lewing@microsoft.com>

Merge branch 'main' into test-build-wasm

543fe48

radekdoulik merged commit 5263848 into dotnet:main May 9, 2024
157 of 159 checks passed

MichalStrehovsky mentioned this pull request May 10, 2024

[main] Update dependencies from dotnet/runtime dotnet/sdk#40780

Merged

akoeplinger reviewed May 13, 2024

View reviewed changes

radekdoulik mentioned this pull request Jun 5, 2024

[wasm] Fix emsdk check #103027

Merged

github-actions bot locked and limited conversation to collaborators Jun 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm] Use source built emsdk packages for the runtime build #100266

[wasm] Use source built emsdk packages for the runtime build #100266

steveisok commented Mar 26, 2024

steveisok commented Mar 26, 2024

akoeplinger commented Mar 26, 2024

akoeplinger Mar 26, 2024 •

edited

Loading

radekdoulik Apr 23, 2024

lewing commented Mar 27, 2024

steveisok commented Mar 27, 2024

akoeplinger May 6, 2024

maraf May 7, 2024

akoeplinger May 7, 2024

radekdoulik May 10, 2024

lewing left a comment

akoeplinger May 13, 2024

akoeplinger May 13, 2024

akoeplinger May 13, 2024

[wasm] Use source built emsdk packages for the runtime build #100266

[wasm] Use source built emsdk packages for the runtime build #100266

Conversation

steveisok commented Mar 26, 2024

steveisok commented Mar 26, 2024

akoeplinger commented Mar 26, 2024

akoeplinger Mar 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewing commented Mar 27, 2024

steveisok commented Mar 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akoeplinger Mar 26, 2024 •

edited

Loading