Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasm] p/invoke and interop improvements #94446

Merged
merged 38 commits into from
Feb 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
6dd24f7
Checkpoint
kg Nov 3, 2023
86b5cdd
Repair merge damage
kg Nov 21, 2023
a355079
Merge branch 'main' into pinvoke-srm-1
lewing Nov 27, 2023
13157f3
Run M2NG in-process again
kg Nov 29, 2023
11c3d7a
Our current approach to SystemNative_SNPrintF is incorrect on some ar…
kg Nov 30, 2023
0d413b2
Support initonly fields in blittable structs in PInvokeTableGenerator…
kg Nov 30, 2023
a46ddd9
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Dec 1, 2023
51274ed
Work around our rampant use of warnaserror
kg Dec 2, 2023
bf9439d
Revert most csproj changes
kg Dec 2, 2023
6f49b91
Better solution for warnaserror (probably)
kg Dec 2, 2023
c25be75
Restore old EnumCalendarInfo since we're in-process again
kg Dec 2, 2023
65275b6
Revert fn ptr test changes
kg Dec 2, 2023
f47f8ea
Maybe fix blazor WBTs
kg Dec 4, 2023
835e278
Make LogAdapter Info high priority so it appears in failed build logs
kg Dec 5, 2023
d8ac1a9
Hack to make skiasharp build
kg Dec 5, 2023
f40b16a
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Dec 6, 2023
3039f55
Implement wasm abi struct scalar rules in interp again
kg Dec 6, 2023
e92b95d
Add test coverage for i64 pinvokes + add fixme comments
kg Dec 6, 2023
2964b73
Fix scalar int64 structs
kg Dec 8, 2023
ce56e66
Code cleanup
kg Dec 9, 2023
037df97
Formatting
kg Dec 9, 2023
0dda91a
Checkpoint (reverse out-of-process)
kg Dec 19, 2023
8f869e2
Checkpoint
kg Dec 19, 2023
36dc365
Merge branch 'main' into pinvoke-srm-1
radical Jan 2, 2024
77104b3
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Jan 5, 2024
9abbf32
Fix Vector64 AOT test failures (calling convention was incorrect when…
kg Jan 6, 2024
e9667c7
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Jan 8, 2024
8548fbf
Ensure we get MLC 8.0 in WasmAppBuilder
kg Jan 8, 2024
36dc34a
Update tests since function pointers work now
kg Jan 9, 2024
7466463
Attempt to resolve prebuilts/source build CI failure by adding (missi…
kg Jan 10, 2024
d15f07a
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Jan 10, 2024
19cd8b3
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Jan 18, 2024
c6a809a
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Jan 24, 2024
47478c5
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Jan 25, 2024
2a57464
Merge remote-tracking branch 'upstream/main' into pinvoke-srm-1
kg Feb 5, 2024
b1e64a0
Address PR feedback
kg Feb 5, 2024
1ba9222
Repair merge damage
kg Feb 5, 2024
5f75189
Merge branch 'main' into pinvoke-srm-1
lewing Feb 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions eng/Version.Details.xml
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,11 @@
<Sha>205ef031e0fe5152dede0bd9f99d0f6f9e7f1e45</Sha>
<SourceBuild RepoName="runtime" ManagedOnly="false" />
</Dependency>
<Dependency Name="System.Reflection.MetadataLoadContext" Version="8.0.0">
<Uri>https://github.com/dotnet/runtime</Uri>
<Sha>4dffd80c4d77c27e772a0be26e8036af77fbb26e</Sha>
<SourceBuild RepoName="runtime" ManagedOnly="false" />
</Dependency>
<Dependency Name="Microsoft.DotNet.ILCompiler" Version="9.0.0-alpha.1.24072.1">
<Uri>https://github.com/dotnet/runtime</Uri>
<Sha>205ef031e0fe5152dede0bd9f99d0f6f9e7f1e45</Sha>
Expand Down
6 changes: 3 additions & 3 deletions eng/Versions.props
Original file line number Diff line number Diff line change
Expand Up @@ -123,16 +123,16 @@
<MicrosoftWin32RegistryVersion>5.0.0</MicrosoftWin32RegistryVersion>
<StyleCopAnalyzersVersion>1.2.0-beta.507</StyleCopAnalyzersVersion>
<SystemBuffersVersion>4.5.1</SystemBuffersVersion>
<SystemCollectionsImmutableVersion>7.0.0</SystemCollectionsImmutableVersion>
<SystemCollectionsImmutableVersion>8.0.0</SystemCollectionsImmutableVersion>
<SystemComponentModelAnnotationsVersion>5.0.0</SystemComponentModelAnnotationsVersion>
<SystemDataSqlClientVersion>4.8.6</SystemDataSqlClientVersion>
<SystemDrawingCommonVersion>8.0.0</SystemDrawingCommonVersion>
<SystemIOFileSystemAccessControlVersion>5.0.0</SystemIOFileSystemAccessControlVersion>
<SystemMemoryVersion>4.5.5</SystemMemoryVersion>
<SystemReflectionMetadataVersion>9.0.0-alpha.1.24072.1</SystemReflectionMetadataVersion>
<!-- Keep toolset versions in sync with dotnet/msbuild and dotnet/sdk -->
<SystemReflectionMetadataToolsetVersion>7.0.0</SystemReflectionMetadataToolsetVersion>
<SystemReflectionMetadataLoadContextVersion>7.0.0</SystemReflectionMetadataLoadContextVersion>
<SystemReflectionMetadataToolsetVersion>8.0.0</SystemReflectionMetadataToolsetVersion>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the comment above, this toolset version should not be updated without coordination with msbuild and sdk repos.

@ViktorHofer, FYI, this is blocking the codeflow dotnet/sdk#38665 (while previous change made it to the next phases without problems dotnet/sdk#38553). We can either revert this or update https://github.com/dotnet/sdk/blob/b32df9afff77e8f1afbd417e7c70841867cb6cdc/src/Tasks/Microsoft.NET.Build.Tasks/Microsoft.NET.Build.Tasks.csproj#L75 to 8.0 (if we choose the latter, we can close #96809).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I coordinated with people from the msbuild team at multiple points while developing this along with the SBRP team, so I'm sorry that something fell through the cracks here. Is there a specific step we missed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see now from looking through the replies to Larry's comment on this issue that you told us there. For some reason, GitHub did not send me a notification for your comment, so I didn't see it :( I guess in the future blocking issues like that need to be PR comments instead of replies.

Copy link
Member

@lewing lewing Feb 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preference would definitely be to take the bump in the sdk. VS 17.9 is where these changes will ship and it has taken the update.

cc @ViktorHofer @marcpopMSFT

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NP, #96809 is tracking it, but it seems (almost) resolved by your PR. Once confirmed, we can just update sdk's Microsoft.NET.Build.Tasks.csproj#L74-L76 to 8.0 (now that all the related toolings are using 8.0; msbuild, HostModel and VS).

Copy link
Member

@ViktorHofer ViktorHofer Feb 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for this to work, sdk would need to upgrade its .NET Framework leg to a VS 17.9 Preview image (for the updated msbuild binding redirects that allow to unify to an 8.0.0.0 assembly version). I'm not sure if such an image is already available. Worst case, we would need to revert this dependency bump here. @lewing can you please make sure that this dependency flow issue gets resolved: dotnet/sdk#38665?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong but the sdk build should be fine, it'd just need the testing legs to use VS17.9 Preview right? That should already be the case: https://github.com/dotnet/sdk/blob/0962c1f89f5daf924a9fe876c80e80b0bde63b0d/.vsts-ci.yml#L108-L111

I'll try bumping the .csproj in the sdk PR and see what happens.

Copy link
Member

@ViktorHofer ViktorHofer Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. Binding redirect unification at run time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it worked :)

<SystemReflectionMetadataLoadContextVersion>8.0.0</SystemReflectionMetadataLoadContextVersion>
<SystemSecurityAccessControlVersion>6.0.0</SystemSecurityAccessControlVersion>
<SystemSecurityCryptographyCngVersion>5.0.0</SystemSecurityCryptographyCngVersion>
<SystemSecurityCryptographyOpenSslVersion>5.0.0</SystemSecurityCryptographyOpenSslVersion>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ internal static partial class Sys
/// success; if the return value is equal to the size then the result may have been truncated.
/// On failure, returns a negative value.
/// </returns>
[LibraryImport(Libraries.SystemNative, EntryPoint = "SystemNative_SNPrintF", StringMarshalling = StringMarshalling.Utf8, SetLastError = true)]
[LibraryImport(Libraries.SystemNative, EntryPoint = "SystemNative_SNPrintF_1S", StringMarshalling = StringMarshalling.Utf8, SetLastError = true)]
internal static unsafe partial int SNPrintF(byte* str, int size, string format, string arg1);

/// <summary>
Expand All @@ -47,7 +47,7 @@ internal static partial class Sys
/// success; if the return value is equal to the size then the result may have been truncated.
/// On failure, returns a negative value.
/// </returns>
[LibraryImport(Libraries.SystemNative, EntryPoint = "SystemNative_SNPrintF", StringMarshalling = StringMarshalling.Utf8, SetLastError = true)]
[LibraryImport(Libraries.SystemNative, EntryPoint = "SystemNative_SNPrintF_1I", StringMarshalling = StringMarshalling.Utf8, SetLastError = true)]
internal static unsafe partial int SNPrintF(byte* str, int size, string format, int arg1);
}
}
8 changes: 8 additions & 0 deletions src/mono/mono/mini/aot-runtime-wasm.c
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,14 @@ type_to_c (MonoType *t)
goto handle_enum;
}

// https://github.com/WebAssembly/tool-conventions/blob/main/BasicCABI.md#function-signatures
// Any struct or union that recursively (including through nested structs, unions, and arrays)
// contains just a single scalar value and is not specified to have greater than natural alignment.
// FIXME: Handle the scenario where there are fields of struct types that contain no members
MonoType *scalar_vtype;
if (mini_wasm_is_scalar_vtype (t, &scalar_vtype))
return type_to_c (scalar_vtype);

return 'I';
case MONO_TYPE_GENERICINST:
if (m_class_is_valuetype (t->data.klass))
Expand Down
36 changes: 25 additions & 11 deletions src/mono/mono/mini/interp/interp.c
Original file line number Diff line number Diff line change
Expand Up @@ -1345,10 +1345,22 @@ typedef enum {

typedef struct {
int ilen, flen;
PInvokeArgType ret_type;
MonoType *ret_mono_type;
PInvokeArgType ret_pinvoke_type;
PInvokeArgType *arg_types;
} BuildArgsFromSigInfo;

static MonoType *
filter_type_for_args_from_sig (MonoType *type) {
#if defined(HOST_WASM)
MonoType *etype;
if (MONO_TYPE_ISSTRUCT (type) && mini_wasm_is_scalar_vtype (type, &etype))
// FIXME: Does this need to be recursive?
return etype;
#endif
return type;
}

static BuildArgsFromSigInfo *
get_build_args_from_sig_info (MonoMemoryManager *mem_manager, MonoMethodSignature *sig)
{
Expand All @@ -1360,7 +1372,7 @@ get_build_args_from_sig_info (MonoMemoryManager *mem_manager, MonoMethodSignatur
g_assert (!sig->hasthis);

for (int i = 0; i < sig->param_count; i++) {
MonoType *type = sig->params [i];
MonoType *type = filter_type_for_args_from_sig (sig->params [i]);
guint32 ptype;

retry:
Expand Down Expand Up @@ -1442,7 +1454,9 @@ get_build_args_from_sig_info (MonoMemoryManager *mem_manager, MonoMethodSignatur
info->ilen = ilen;
info->flen = flen;

switch (sig->ret->type) {
info->ret_mono_type = filter_type_for_args_from_sig (sig->ret);

switch (info->ret_mono_type->type) {
case MONO_TYPE_BOOLEAN:
case MONO_TYPE_CHAR:
case MONO_TYPE_I1:
Expand All @@ -1463,17 +1477,17 @@ get_build_args_from_sig_info (MonoMemoryManager *mem_manager, MonoMethodSignatur
case MONO_TYPE_U8:
case MONO_TYPE_VALUETYPE:
case MONO_TYPE_GENERICINST:
info->ret_type = PINVOKE_ARG_INT;
info->ret_pinvoke_type = PINVOKE_ARG_INT;
break;
case MONO_TYPE_R4:
case MONO_TYPE_R8:
info->ret_type = PINVOKE_ARG_R8;
info->ret_pinvoke_type = PINVOKE_ARG_R8;
break;
case MONO_TYPE_VOID:
info->ret_type = PINVOKE_ARG_NONE;
info->ret_pinvoke_type = PINVOKE_ARG_NONE;
break;
default:
g_error ("build_args_from_sig: ret type not implemented yet: 0x%x\n", sig->ret->type);
g_error ("build_args_from_sig: ret type not implemented yet: 0x%x\n", info->ret_mono_type->type);
}

return info;
Expand Down Expand Up @@ -1563,7 +1577,7 @@ build_args_from_sig (InterpMethodArguments *margs, MonoMethodSignature *sig, Bui
}
}

switch (info->ret_type) {
switch (info->ret_pinvoke_type) {
case PINVOKE_ARG_INT:
margs->retval = (gpointer*)frame->retval;
margs->is_float_ret = 0;
Expand Down Expand Up @@ -1781,8 +1795,8 @@ ves_pinvoke_method (
g_free (ccontext.stack);
#else
// Only the vt address has been returned, we need to copy the entire content on interp stack
if (!context->has_resume_state && MONO_TYPE_ISSTRUCT (sig->ret))
stackval_from_data (sig->ret, frame.retval, (char*)frame.retval->data.p, sig->pinvoke && !sig->marshalling_disabled);
if (!context->has_resume_state && MONO_TYPE_ISSTRUCT (call_info->ret_mono_type))
stackval_from_data (call_info->ret_mono_type, frame.retval, (char*)frame.retval->data.p, sig->pinvoke && !sig->marshalling_disabled);

if (margs.iargs != margs.iargs_buf)
g_free (margs.iargs);
Expand Down Expand Up @@ -4252,7 +4266,7 @@ mono_interp_exec_method (InterpFrame *frame, ThreadContext *context, FrameClause
LOCAL_VAR (call_args_offset, gpointer) = unboxed;
}

jit_call:
jit_call:
{
InterpMethodCodeType code_type = cmethod->code_type;

Expand Down
25 changes: 17 additions & 8 deletions src/mono/mono/mini/mini-llvm.c
Original file line number Diff line number Diff line change
Expand Up @@ -4192,6 +4192,7 @@ emit_entry_bb (EmitContext *ctx, LLVMBuilderRef builder)
case LLVMArgVtypeAddr:
case LLVMArgVtypeByRef:
case LLVMArgAsFpArgs:
case LLVMArgWasmVtypeAsScalar:
{
MonoClass *klass = mono_class_from_mono_type_internal (ainfo->type);
if (mini_class_is_simd (ctx->cfg, klass)) {
Expand Down Expand Up @@ -4968,6 +4969,8 @@ process_call (EmitContext *ctx, MonoBasicBlock *bb, LLVMBuilderRef *builder_ref,
if (!addresses [call->inst.dreg])
addresses [call->inst.dreg] = build_alloca_address (ctx, sig->ret);
emit_store (builder, lcall, convert_full (ctx, addresses [call->inst.dreg]->value, pointer_type (LLVMTypeOf (lcall)), FALSE), is_volatile);
load_name = "wasm_vtype_as_scalar";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be WASM only?

should_promote_to_value = TRUE;
break;
}
default:
Expand Down Expand Up @@ -5421,6 +5424,7 @@ static LLVMValueRef
concatenate_vectors (EmitContext *ctx, LLVMValueRef xs, LLVMValueRef ys)
{
LLVMTypeRef t = LLVMTypeOf (xs);
g_assert (LLVMGetTypeKind (t) == LLVMVectorTypeKind);
unsigned int elems = LLVMGetVectorSize (t) * 2;
int mask [MAX_VECTOR_ELEMS] = { 0 };
for (guint i = 0; i < elems; ++i)
Expand Down Expand Up @@ -6175,8 +6179,13 @@ process_bb (EmitContext *ctx, MonoBasicBlock *bb)
}
break;
case LLVMArgWasmVtypeAsScalar:
g_assert (addresses [ins->sreg1]);
retval = LLVMBuildLoad2 (builder, ret_type, build_ptr_cast (builder, addresses [ins->sreg1]->value, pointer_type (ret_type)), "");
if (!addresses [ins->sreg1]) {
/* SIMD value */
g_assert (lhs);
retval = LLVMBuildBitCast (builder, lhs, ret_type, "");
} else {
retval = LLVMBuildLoad2 (builder, ret_type, build_ptr_cast (builder, addresses [ins->sreg1]->value, pointer_type (ret_type)), "");
}
break;
}
LLVMBuildRet (builder, retval);
Expand Down Expand Up @@ -6225,8 +6234,8 @@ process_bb (EmitContext *ctx, MonoBasicBlock *bb)

if (lhs) {
// Vector3: ret_type is Vector3, lhs is Vector3 represented as a Vector4 (three elements + zero). We need to extract only the first 3 elements from lhs.
int len = mono_class_value_size (klass, NULL) == 12 ? 3 : LLVMGetVectorSize (LLVMTypeOf (lhs));
int len = mono_class_value_size (klass, NULL) == 12 ? 3 : LLVMGetVectorSize (LLVMTypeOf (lhs));

for (int i = 0; i < len; i++) {
elem = LLVMBuildExtractElement (builder, lhs, const_int32 (i), "extract_elem");
retval = LLVMBuildInsertValue (builder, retval, elem, i, "insert_val_struct");
Expand Down Expand Up @@ -6841,7 +6850,7 @@ MONO_RESTORE_WARNING
// LLVM should fuse the individual Div and Rem instructions into one DIV/IDIV on x86
values [ins->dreg] = LLVMBuildTrunc (builder, LLVMBuildSDiv (builder, dividend, divisor, ""), part_type, "");
last_divrem = LLVMBuildTrunc (builder, LLVMBuildSRem (builder, dividend, divisor, ""), part_type, "");
break;
break;
}
case OP_X86_IDIVREMU:
case OP_X86_LDIVREMU: {
Expand All @@ -6856,7 +6865,7 @@ MONO_RESTORE_WARNING
LLVMValueRef divisor = LLVMBuildZExt (builder, convert (ctx, arg3, part_type), full_type, "");
values [ins->dreg] = LLVMBuildTrunc (builder, LLVMBuildUDiv (builder, dividend, divisor, ""), part_type, "");
last_divrem = LLVMBuildTrunc (builder, LLVMBuildURem (builder, dividend, divisor, ""), part_type, "");
break;
break;
}
case OP_X86_IDIVREM2:
case OP_X86_LDIVREM2: {
Expand Down Expand Up @@ -10671,7 +10680,7 @@ MONO_RESTORE_WARNING

// convert to 0/1
result = LLVMBuildICmp (builder, LLVMIntEQ, first_elem, LLVMConstAllOnes (LLVMInt64Type ()), "");

values [ins->dreg] = LLVMBuildZExt (builder, result, LLVMInt8Type (), "");
break;
}
Expand Down Expand Up @@ -12020,7 +12029,7 @@ MONO_RESTORE_WARNING
gboolean scalar = ins->opcode == OP_NEGATION_SCALAR;
gboolean is_float = (ins->inst_c1 == MONO_TYPE_R4 || ins->inst_c1 == MONO_TYPE_R8);

LLVMValueRef result = lhs;
LLVMValueRef result = lhs;
if (scalar)
result = scalar_from_vector (ctx, result);
if (is_float)
Expand Down
2 changes: 0 additions & 2 deletions src/mono/mono/mini/mini-wasm.c
Original file line number Diff line number Diff line change
Expand Up @@ -774,8 +774,6 @@ mini_wasm_is_scalar_vtype (MonoType *type, MonoType **etype)
return FALSE;
} else if (!((MONO_TYPE_IS_PRIMITIVE (t) || MONO_TYPE_IS_REFERENCE (t) || MONO_TYPE_IS_POINTER (t)))) {
return FALSE;
} else if (size == 8 && t->type != MONO_TYPE_R8) {
return FALSE;
} else {
if (etype)
*etype = t;
Expand Down
Loading
Loading