From f9a32c777461e76e1ece01c02c04a5e33f09ee1d Mon Sep 17 00:00:00 2001 From: Aaron Robinson Date: Wed, 18 Mar 2020 17:03:44 -0700 Subject: [PATCH 1/9] Initial proposal for Marshal Directives. --- docs/design/features/MarshalDirective.md | 206 +++++++++++++++++++++++ 1 file changed, 206 insertions(+) create mode 100644 docs/design/features/MarshalDirective.md diff --git a/docs/design/features/MarshalDirective.md b/docs/design/features/MarshalDirective.md new file mode 100644 index 0000000000000..94d07b24bdc41 --- /dev/null +++ b/docs/design/features/MarshalDirective.md @@ -0,0 +1,206 @@ +# User Defined Marshal Directive + +## Purpose + +The CLR possesses a rich built-in marshaling mechanism for interopability with native code that is handled at runtime. This system was written to free .NET developers from having to author complex and potentially ABI sensitive [type conversion code][typemarshal_link] from a managed to an unmanaged environment. The built-in system works with both [P/Invoke][pinvoke_link] (i.e. `DllImportAttribute`) and [COM interop](https://docs.microsoft.com/dotnet/standard/native-interop/cominterop). The generated portion is typically called an ["IL Stub"][il_stub_link] since the stub is generated by inserting IL instructions into a stream and then passing it off to the JIT for compilation. + +One consequence of this approach is that marshaling code is not immediately available post link for AOT scenarios (e.g. [`crossgen`](../../workflow/building/coreclr/crossgen.md) and [`crossgen2`](crossgen2-compilation-structure-enhancements.md)). The immediate unavailability of this code has been mitigated by a complex mechanism to have marshalling code generated at AOT time. + +The user experience of the built-in generation initially appears ideal, but there are several negative consequences that make the system costly in the long term: + +* Bug fixes in the marshaling system require an update to the entire runtime. +* New types require enhancements to the marshaling system for efficient marshal behavior. + * [`ICustomMarshaler`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.icustommarshaler) incurs a substantial performance penalty. +* Once a marshaling bug becomes expected behavior the bug is difficult to fix. This is due to user reliance on shipped behavior and since the marshaling system is built into the runtime there aren't ways to select previous or new behavior. + * Example involving COM marshaling: https://github.com/dotnet/coreclr/pull/23974. + +This is not to say the P/Invoke system should be completely redesigned. The current system is heavily used its current simplicity for consuming native assets is a benefit. Rather this new mechansim is designed to provide a way for marshaling code to be generated by an external tool, but integrate with `DllImportAttribute` in a natural way. + +**Note** This attribute is not designed to work with COM interop. The [`ComWrappers`][comwrappers_link] API should be used instead. + +### Requirements + +* [C# Function pointers][csharp_fptr_link]. + * https://github.com/dotnet/roslyn/issues/39865 + +## Design + +The Marshal Directive design is focused on a natural integration with existing uses of `DllImportAttribute`. The current and anticipated P/Invoke algorithm is presented below using a simple example. + +``` CSharp +[DllImportAttribute("Kernel32.dll")] +/* A */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount); +... +long count; +/* B */ QueryPerformanceCounter(out count); +``` + +At (A) in the above code snippet the runtime is told to look for an export name `QueryPerformanceCounter` in the `Kernel32.dll` binary. There are many additional attributes on the `DllImportAttribute` that help with export discovery and can influence the semantics of the generated IL Stub. Point (B) represents an invocation of the P/Invoke. The majority of the work occurs at (B) at runtime, since (A) is merely a declaration the compiler uses to embed the relevant details into the metadata. + +1) During invocation, the function declaration is determined to be an external call requiring marshaling. Given the defined properties in the `DllImportAttribute` instance as well as the metadata of the user-defined signature an IL Stub is generated. + + * An IL Stub is always generated when an assembly is compiled in `Debug`. In `Release` builds it is possible no IL Stub is generated and instead the JIT will elide the stub and inline the invocation. + + * **Marshal Directive**: The user would define a method to call instead of asking the runtime to generate one. This would not impact the scenario where the JIT has determined the P/Invoke can be inlined. + +2) The runtime attempts to find a binary with the name supplied in `DllImportAttribute`. + + * Discovery of the target binary is complicated and can be influenced by the [`AssemblyLoadContext`](https://docs.microsoft.com/dotnet/api/system.runtime.loader.assemblyloadcontext) and [`NativeLibrary`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.nativelibrary) classes. + +3) Once the binary is found and loaded into the runtime, it is queried for the expected export name. The name of the attributed function is used by default but this is configurable by the [`DllImportAttribute.EntryPoint`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.dllimportattribute.entrypoint) property. + + * This process is also influenced by additional `DllImportAttribute` properties as well as by the underlying platform. For example, the [Win32 API ANSI/UNICODE convention](https://docs.microsoft.com/windows/win32/intl/conventions-for-function-prototypes) on Windows is respected and a(n) `A`/`W` suffix may be appended to the function name if it is not immediately found. + +4) The IL Stub is called like any .NET method, but the address of the export is passed to the generated IL Stub via a 'hidden' argument. + + * **Marshal Directive**: The user's defined method would require an identical signature to that of the one typically defined, but a new argument of type `IntPtr` should be added as the first argument. The runtime would be responsible for providing the export to the user defined function. + +5) The IL Stub then marshals arguments as appropriate and invokes the export via the `calli` instruction. + + * **Marshal Directive**: The user's defined method would cast the supplied export to an appropriate C# function pointer type and invoke the call. + +6) Once the export returns control to the IL Stub the marshaling logic cleans up and ensures any returned data is marshaled back out to the calling function. + +An example of how the snippet above could be transformed is below. The example is using the proposed API in this document as well as the [C# function pointer proposal][csharp_fptr_link]. Observe points (C), the new attribute, (D), a `partial` class, and (E) the stub with additional `IntPtr` argument that will be called in lieu of a runtime generated IL Stub. + +``` CSharp +[DllImportAttribute("Kernel32.dll")] +/* C */ [MarshalDirectiveAttribute( + Type = MarshalDirectiveType.UserDefinedStub, + UserFunctionType = typeof(Stubs), + UserDefinedStub = nameof(QueryPerformanceCounterStub))] +/* A */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount); +... +long count; +/* B */ QueryPerformanceCounter(out count); + +/* D */ class partial Stubs +{ + /* E */ public static bool QueryPerformanceCounterStub(IntPtr fptr, out long lpPerformanceCount) + { + unsafe + { + long result = 0; + var tfptr = (delegate* stdcall)fptr; + bool success = tfptr(&result) != 0; + lpPerformanceCount = result; + return success; + } + } +} +``` + +The contents of the above stub are trivial and in a `Release` build wouldn't even exist. More complicated examples may require [GC handle pinning](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.gchandle) or native memory allocation. These complicated examples could be hand authored or generated through a third-party tool. + +In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types and it is these rules that once shipped become the defacto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unamanged environment. + +### Proposed API +``` CSharp +namespace System.Runtime.InteropServices +{ + /// + /// User defined marshaling directive + /// + public enum MarshalDirectiveType + { + /// + /// The attribute does nothing. + /// + Default = 0, + + /// + /// The user can supply a managed function that will be called in-place + /// of a generated function stub. + /// + /// + UserDefinedStub, + } + + /// + /// Attribute used to define how marshaling stubs should be defined for + /// the associated managed function declaration. + /// + [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)] + public sealed class MarshalDirectiveAttribute : Attribute + { + /// + /// Default constructor. + /// + public MarshalDirectiveAttribute() + { + this.Type = MarshalDirectiveType.Default; + } + + /// + /// The directive type. + /// + /// + /// The default value is . + /// + public MarshalDirectiveType Type { get; set; } + + /// + /// The type to use to search for user defined functions. + /// + public Type UserFunctionType { get; set; } + + /// + /// The name of the user defined stub to call. + /// Only used if is defined as + /// . + /// + /// + /// The supplied name will be searched for on + /// . + /// + /// The supplied function must conform to a signature of that matches + /// the managed definition of the function, except for the first argument + /// which will be supplied by the runtime. The first argument will be + /// a resolved native function pointer. + /// + /// For example, given the following P/Invoke signature: + /// + /// [DllImport("Kernel32.dll", ExactSpelling = true)] + /// [MarshalDirectiveAttribute( + /// Type = MarshalDirectiveType.UserDefinedStub, + /// UserFunctionType = typeof(Stubs), + /// UserDefinedStub = nameof(QueryPerformanceCounterStub))] + /// extern static unsafe bool QueryPerformanceCounter(long* lpPerformanceCount); + /// + /// + /// The supplied user defined stub should be: + /// + /// static unsafe bool QueryPerformanceCounterStub( + /// IntPtr nativeFunctionPtr, + /// long* lpPerformanceCount) { ... } + /// + /// + /// During the call to the stub, the supplied + /// would be a native function pointer. + /// + public string UserDefinedStub { get; set; } + } +} +``` + +## Questions + +* Can the above API be used to provide a reverse P/Invoke stub? +* Does the the Marshal Directive have any interaction with the [`SuppressGCTransitionAttribute`](https://github.com/dotnet/runtime/issues/30741)? +* Should the `MarshalDirectiveAttribute` be considered in potential support for the [typing of function pointers](https://github.com/dotnet/roslyn/issues/39865#issuecomment-600884703)? + +## References + +[P/Invoke][pinvoke_link] + +[Type Marshaling][typemarshal_link] + +[IL Stubs description][il_stub_link] + + +[dotnet_link]: https://docs.microsoft.com/dotnet/core/tools/dotnet +[typemarshal_link]: https://docs.microsoft.com/dotnet/standard/native-interop/type-marshaling +[pinvoke_link]: https://docs.microsoft.com/dotnet/standard/native-interop/pinvoke +[comwrappers_link]: https://github.com/dotnet/runtime/issues/1845 +[il_stub_link]: https://mattwarren.org/2019/09/26/Stubs-in-the-.NET-Runtime/ +[csharp_fptr_link]: https://github.com/dotnet/csharplang/blob/master/proposals/function-pointers.md \ No newline at end of file From 162dee8b13a8c625f63cc975b7404d3339ec0c17 Mon Sep 17 00:00:00 2001 From: Aaron Robinson Date: Thu, 19 Mar 2020 15:25:14 -0700 Subject: [PATCH 2/9] Realignment with existing Pinvoke generators. --- docs/design/features/MarshalDirective.md | 206 ---------------- .../features/source-generator-pinvokes.md | 233 ++++++++++++++++++ 2 files changed, 233 insertions(+), 206 deletions(-) delete mode 100644 docs/design/features/MarshalDirective.md create mode 100644 docs/design/features/source-generator-pinvokes.md diff --git a/docs/design/features/MarshalDirective.md b/docs/design/features/MarshalDirective.md deleted file mode 100644 index 94d07b24bdc41..0000000000000 --- a/docs/design/features/MarshalDirective.md +++ /dev/null @@ -1,206 +0,0 @@ -# User Defined Marshal Directive - -## Purpose - -The CLR possesses a rich built-in marshaling mechanism for interopability with native code that is handled at runtime. This system was written to free .NET developers from having to author complex and potentially ABI sensitive [type conversion code][typemarshal_link] from a managed to an unmanaged environment. The built-in system works with both [P/Invoke][pinvoke_link] (i.e. `DllImportAttribute`) and [COM interop](https://docs.microsoft.com/dotnet/standard/native-interop/cominterop). The generated portion is typically called an ["IL Stub"][il_stub_link] since the stub is generated by inserting IL instructions into a stream and then passing it off to the JIT for compilation. - -One consequence of this approach is that marshaling code is not immediately available post link for AOT scenarios (e.g. [`crossgen`](../../workflow/building/coreclr/crossgen.md) and [`crossgen2`](crossgen2-compilation-structure-enhancements.md)). The immediate unavailability of this code has been mitigated by a complex mechanism to have marshalling code generated at AOT time. - -The user experience of the built-in generation initially appears ideal, but there are several negative consequences that make the system costly in the long term: - -* Bug fixes in the marshaling system require an update to the entire runtime. -* New types require enhancements to the marshaling system for efficient marshal behavior. - * [`ICustomMarshaler`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.icustommarshaler) incurs a substantial performance penalty. -* Once a marshaling bug becomes expected behavior the bug is difficult to fix. This is due to user reliance on shipped behavior and since the marshaling system is built into the runtime there aren't ways to select previous or new behavior. - * Example involving COM marshaling: https://github.com/dotnet/coreclr/pull/23974. - -This is not to say the P/Invoke system should be completely redesigned. The current system is heavily used its current simplicity for consuming native assets is a benefit. Rather this new mechansim is designed to provide a way for marshaling code to be generated by an external tool, but integrate with `DllImportAttribute` in a natural way. - -**Note** This attribute is not designed to work with COM interop. The [`ComWrappers`][comwrappers_link] API should be used instead. - -### Requirements - -* [C# Function pointers][csharp_fptr_link]. - * https://github.com/dotnet/roslyn/issues/39865 - -## Design - -The Marshal Directive design is focused on a natural integration with existing uses of `DllImportAttribute`. The current and anticipated P/Invoke algorithm is presented below using a simple example. - -``` CSharp -[DllImportAttribute("Kernel32.dll")] -/* A */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount); -... -long count; -/* B */ QueryPerformanceCounter(out count); -``` - -At (A) in the above code snippet the runtime is told to look for an export name `QueryPerformanceCounter` in the `Kernel32.dll` binary. There are many additional attributes on the `DllImportAttribute` that help with export discovery and can influence the semantics of the generated IL Stub. Point (B) represents an invocation of the P/Invoke. The majority of the work occurs at (B) at runtime, since (A) is merely a declaration the compiler uses to embed the relevant details into the metadata. - -1) During invocation, the function declaration is determined to be an external call requiring marshaling. Given the defined properties in the `DllImportAttribute` instance as well as the metadata of the user-defined signature an IL Stub is generated. - - * An IL Stub is always generated when an assembly is compiled in `Debug`. In `Release` builds it is possible no IL Stub is generated and instead the JIT will elide the stub and inline the invocation. - - * **Marshal Directive**: The user would define a method to call instead of asking the runtime to generate one. This would not impact the scenario where the JIT has determined the P/Invoke can be inlined. - -2) The runtime attempts to find a binary with the name supplied in `DllImportAttribute`. - - * Discovery of the target binary is complicated and can be influenced by the [`AssemblyLoadContext`](https://docs.microsoft.com/dotnet/api/system.runtime.loader.assemblyloadcontext) and [`NativeLibrary`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.nativelibrary) classes. - -3) Once the binary is found and loaded into the runtime, it is queried for the expected export name. The name of the attributed function is used by default but this is configurable by the [`DllImportAttribute.EntryPoint`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.dllimportattribute.entrypoint) property. - - * This process is also influenced by additional `DllImportAttribute` properties as well as by the underlying platform. For example, the [Win32 API ANSI/UNICODE convention](https://docs.microsoft.com/windows/win32/intl/conventions-for-function-prototypes) on Windows is respected and a(n) `A`/`W` suffix may be appended to the function name if it is not immediately found. - -4) The IL Stub is called like any .NET method, but the address of the export is passed to the generated IL Stub via a 'hidden' argument. - - * **Marshal Directive**: The user's defined method would require an identical signature to that of the one typically defined, but a new argument of type `IntPtr` should be added as the first argument. The runtime would be responsible for providing the export to the user defined function. - -5) The IL Stub then marshals arguments as appropriate and invokes the export via the `calli` instruction. - - * **Marshal Directive**: The user's defined method would cast the supplied export to an appropriate C# function pointer type and invoke the call. - -6) Once the export returns control to the IL Stub the marshaling logic cleans up and ensures any returned data is marshaled back out to the calling function. - -An example of how the snippet above could be transformed is below. The example is using the proposed API in this document as well as the [C# function pointer proposal][csharp_fptr_link]. Observe points (C), the new attribute, (D), a `partial` class, and (E) the stub with additional `IntPtr` argument that will be called in lieu of a runtime generated IL Stub. - -``` CSharp -[DllImportAttribute("Kernel32.dll")] -/* C */ [MarshalDirectiveAttribute( - Type = MarshalDirectiveType.UserDefinedStub, - UserFunctionType = typeof(Stubs), - UserDefinedStub = nameof(QueryPerformanceCounterStub))] -/* A */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount); -... -long count; -/* B */ QueryPerformanceCounter(out count); - -/* D */ class partial Stubs -{ - /* E */ public static bool QueryPerformanceCounterStub(IntPtr fptr, out long lpPerformanceCount) - { - unsafe - { - long result = 0; - var tfptr = (delegate* stdcall)fptr; - bool success = tfptr(&result) != 0; - lpPerformanceCount = result; - return success; - } - } -} -``` - -The contents of the above stub are trivial and in a `Release` build wouldn't even exist. More complicated examples may require [GC handle pinning](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.gchandle) or native memory allocation. These complicated examples could be hand authored or generated through a third-party tool. - -In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types and it is these rules that once shipped become the defacto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unamanged environment. - -### Proposed API -``` CSharp -namespace System.Runtime.InteropServices -{ - /// - /// User defined marshaling directive - /// - public enum MarshalDirectiveType - { - /// - /// The attribute does nothing. - /// - Default = 0, - - /// - /// The user can supply a managed function that will be called in-place - /// of a generated function stub. - /// - /// - UserDefinedStub, - } - - /// - /// Attribute used to define how marshaling stubs should be defined for - /// the associated managed function declaration. - /// - [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)] - public sealed class MarshalDirectiveAttribute : Attribute - { - /// - /// Default constructor. - /// - public MarshalDirectiveAttribute() - { - this.Type = MarshalDirectiveType.Default; - } - - /// - /// The directive type. - /// - /// - /// The default value is . - /// - public MarshalDirectiveType Type { get; set; } - - /// - /// The type to use to search for user defined functions. - /// - public Type UserFunctionType { get; set; } - - /// - /// The name of the user defined stub to call. - /// Only used if is defined as - /// . - /// - /// - /// The supplied name will be searched for on - /// . - /// - /// The supplied function must conform to a signature of that matches - /// the managed definition of the function, except for the first argument - /// which will be supplied by the runtime. The first argument will be - /// a resolved native function pointer. - /// - /// For example, given the following P/Invoke signature: - /// - /// [DllImport("Kernel32.dll", ExactSpelling = true)] - /// [MarshalDirectiveAttribute( - /// Type = MarshalDirectiveType.UserDefinedStub, - /// UserFunctionType = typeof(Stubs), - /// UserDefinedStub = nameof(QueryPerformanceCounterStub))] - /// extern static unsafe bool QueryPerformanceCounter(long* lpPerformanceCount); - /// - /// - /// The supplied user defined stub should be: - /// - /// static unsafe bool QueryPerformanceCounterStub( - /// IntPtr nativeFunctionPtr, - /// long* lpPerformanceCount) { ... } - /// - /// - /// During the call to the stub, the supplied - /// would be a native function pointer. - /// - public string UserDefinedStub { get; set; } - } -} -``` - -## Questions - -* Can the above API be used to provide a reverse P/Invoke stub? -* Does the the Marshal Directive have any interaction with the [`SuppressGCTransitionAttribute`](https://github.com/dotnet/runtime/issues/30741)? -* Should the `MarshalDirectiveAttribute` be considered in potential support for the [typing of function pointers](https://github.com/dotnet/roslyn/issues/39865#issuecomment-600884703)? - -## References - -[P/Invoke][pinvoke_link] - -[Type Marshaling][typemarshal_link] - -[IL Stubs description][il_stub_link] - - -[dotnet_link]: https://docs.microsoft.com/dotnet/core/tools/dotnet -[typemarshal_link]: https://docs.microsoft.com/dotnet/standard/native-interop/type-marshaling -[pinvoke_link]: https://docs.microsoft.com/dotnet/standard/native-interop/pinvoke -[comwrappers_link]: https://github.com/dotnet/runtime/issues/1845 -[il_stub_link]: https://mattwarren.org/2019/09/26/Stubs-in-the-.NET-Runtime/ -[csharp_fptr_link]: https://github.com/dotnet/csharplang/blob/master/proposals/function-pointers.md \ No newline at end of file diff --git a/docs/design/features/source-generator-pinvokes.md b/docs/design/features/source-generator-pinvokes.md new file mode 100644 index 0000000000000..cee2ecd65e73e --- /dev/null +++ b/docs/design/features/source-generator-pinvokes.md @@ -0,0 +1,233 @@ +# Source Generator P/Invokes + +## Purpose + +The CLR possesses a rich built-in marshaling mechanism for interoperability with native code that is handled at runtime. This system was designed to free .NET developers from having to author complex and potentially ABI sensitive [type conversion code][typemarshal_link] from a managed to an unmanaged environment. The built-in system works with both [P/Invoke][pinvoke_link] (i.e. `DllImportAttribute`) and [COM interop](https://docs.microsoft.com/dotnet/standard/native-interop/cominterop). The generated portion is typically called an ["IL Stub"][il_stub_link] since the stub is generated by inserting IL instructions into a stream and then passing that stream to the JIT for compilation. + +A consequence of this approach is that marshaling code is not immediately available post-link for AOT scenarios (e.g. [`crossgen`](../../workflow/building/coreclr/crossgen.md) and [`crossgen2`](crossgen2-compilation-structure-enhancements.md)). The immediate unavailability of this code has been mitigated by a complex mechanism to have marshalling code generated at AOT time. + +The user experience of the built-in generation initially appears ideal, but there are several negative consequences that make the system costly in the long term: + +* Bug fixes in the marshaling system require an update to the entire runtime. +* New types require enhancements to the marshaling system for efficient marshal behavior. + * [`ICustomMarshaler`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.icustommarshaler) incurs a substantial performance penalty. +* Once a marshaling bug becomes expected behavior the bug is difficult to fix. This is due to user reliance on shipped behavior and since the marshaling system is built into the runtime there aren't ways to select previous or new behavior. + * Example involving COM marshaling: https://github.com/dotnet/coreclr/pull/23974. +* Debugging the auto-generated marshaling IL Stub is difficult for runtime developers and close to impossible for consumers of P/Invokes. + +This is not to say the P/Invoke system should be completely redesigned. The current system is heavily used and its simplicity for consuming native assets is a benefit. Rather this new mechanism is designed to provide a way for marshaling code to be generated by an external tool but integrate with `DllImportAttribute` in a way that isn't onerous on current .NET developers. + +The [Roslyn Compiler](https://github.com/dotnet/roslyn) team is working on a [Source Generator feature][source_gen_link] that will allow the generation of additional source files that can be added to an assembly during the compilation process - the runtime generation IL Stubs is an in-memory version of this scenario. + +**Note** This proposal is targeted at addressing P/Invoke improvements but could be adapted to work with COM interop utilizing the new [`ComWrappers`][comwrappers_link] API. + +### Requirements + +* [Source generators][source_gen_link] + * Branch: https://github.com/dotnet/roslyn/tree/features/source-generators + +## Design + +Using Source Generators is focused on integrating with existing uses of `DllImportAttribute` from an invocation point of view (i.e. callsites should not need to be updated). The idea behind Source Generators is that code for some scenarios can be precomputed using user declared types and logic thus avoiding the need to generator code at runtime. The desire is then to provide a way that existing code can continue to function but can be modified in a way that allows it to leverage this new compiler feature. + +### P/Invoke Walkthrough + +The P/Invoke algorithm is presented below using a simple example. + +``` CSharp +/* A */ [DllImportAttribute("Kernel32.dll")] +/* B */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount); +... +long count; +/* C */ QueryPerformanceCounter(out count); +``` + +At (A) in the above code snippet, the runtime is told to look for an export name `QueryPerformanceCounter` (B) in the `Kernel32.dll` binary. There are many additional attributes on the `DllImportAttribute` that help with export discovery and can influence the semantics of the generated IL Stub. Point (C) represents an invocation of the P/Invoke. Most of the work occurs at (C) at runtime, since (A) and (B) are merely declarations the compiler uses to embed the relevant details into assembly metadata that is read at runtime. + +1) During invocation, the function declaration is determined to be an external call requiring marshaling. Given the defined properties in the `DllImportAttribute` instance as well as the metadata of the user-defined signature an IL Stub is generated. + + * An IL Stub is always generated when an assembly is compiled in `Debug`. In `Release` builds it is possible no IL Stub is generated and instead the JIT will elide the stub and inline the invocation. We will come back to this inlining below. + +2) The runtime attempts to find a binary with the name supplied in `DllImportAttribute`. + + * Discovery of the target binary is complicated and can be influenced by the [`AssemblyLoadContext`](https://docs.microsoft.com/dotnet/api/system.runtime.loader.assemblyloadcontext) and [`NativeLibrary`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.nativelibrary) classes. + +3) Once the binary is found and loaded into the runtime, it is queried for the expected export name. The name of the attributed function is used by default but this is configurable by the [`DllImportAttribute.EntryPoint`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.dllimportattribute.entrypoint) property. + + * This process is also influenced by additional `DllImportAttribute` properties as well as by the underlying platform. For example, the [Win32 API ANSI/UNICODE convention](https://docs.microsoft.com/windows/win32/intl/conventions-for-function-prototypes) on Windows is respected and a(n) `A`/`W` suffix may be appended to the function name if it is not immediately found. + +4) The IL Stub is called like any .NET method, but the address of the export is passed to the generated IL Stub via a 'hidden' argument. + +5) The IL Stub then marshals arguments as appropriate and invokes the export via the `calli` instruction. + +6) Once the export returns control to the IL Stub the marshaling logic cleans up and ensures any returned data is marshaled back out to the calling function. + +### Source Generator Integration + +An example of how the previous P/Invoke snippet could be transformed is below. This example is using the proposed API in this document. The Source Generator currently has a limitation that we will ignore at present - the inability to modify user written code. This limitation is ignored to present an optimal user experience, but will be discussed later in case the current limitation becomes a hard requirement. + +`Program.cs` (Pre-generated code) + +``` CSharp +/* A */ [DllImportAttribute("Kernel32.dll")] +/* D */ [GenerateNativeImportAttribute] +/* B */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount); +... +long count; +/* C*/ QueryPerformanceCounter(out count); +``` + +Observe point (D), the new attribute. This attribute provides an indication to a Source Generator that the following declaration should be updated. + +`Program.cs` (In-memory updated code) + +``` CSharp +/* D */ [GenerateNativeImportAttribute] +/* B */ partial static bool QueryPerformanceCounter(out long lpPerformanceCount); +... +long count; +/* C */ QueryPerformanceCounter(out count); +``` + +During the source generation process the `DllImportAttribute` (A) would be removed. The `GenerateNativeImportAttribute` (D) would remain to provide a metadata indication that the function was auto-generated. Also note that the method declaration has been updated and marked `partial`. The Source Generator would then generate the source for this partial method. The invocation (C) remains unchanged. + +`Stubs.g.cs`: + +``` CSharp +/* E */ partial static bool QueryPerformanceCounter(out long lpPerformanceCount) +{ + unsafe + { + long result = 0; + bool success = QueryPerformanceCounter(&result) != 0; + lpPerformanceCount = result; + return success; + } +} + +[DllImportAttribute("Kernel32.dll")] +/* F */ private static extern int QueryPerformanceCounter(long* lpPerformanceCount); +``` + +The Source Generator would generate the implementation of the partial method (E) in a separate translation unit (`Stubs.g.cs`). At point (F) the `DllImportAttribute` declaration from the user's original declaration is copied into a private P/Invoke specifically for the generated code. The P/Invoke signature from the original declaration would be modified to contain only [blittable types][blittable_link] to ensure the JIT could inline the invocation. Finally note that the user's original function signature would remain in to avoid impacting existing callsites. + +In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types, and it is these rules that once shipped become the de facto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unmanaged environment. + +### Source Generator constraint mitigations + +In the current Source Generator design modification of any user written code is not permitted. This includes modification of any non-functional metadata (e.g. Attributes). The above design mentions modification of user source so the current design may be untenable. Let us define the following options and describe how the above design would be modified. + +* In-memory source modification - This would yield the ideal design above. + +* On disk source modification - Altering source on disk would require the `GenerateNativeImportAttribute` to contain all properties from `DllImportAttribute`. This is required since removing the `DllImportAttribute` would be impossible and thus would cause issues during build since the method must also be marked as `partial`. + +* No source modification - This aligns with the current Source Generator constraints. It would be possible to create a [Roslyn Analyzer and Code fix](https://github.com/dotnet/roslyn/wiki/Getting-Started-Writing-a-Custom-Analyzer-&-Code-Fix) to aid the developer in converting their `DllImportAttribute` marked functions to use `GenerateNativeImportAttribute`. Furthermore, the function would need to be updated to have the `partial` keyword and potentially the enclosing class. + +## Proposed APIs + +The ideal solution with support for minor user defined source updates using Source Generators: + +``` CSharp +namespace System.Runtime.InteropServices +{ + /// + /// Attribute used to indicate a Source Generator should create a function for marshaling + /// arguments instead of relying on the CLR to generate an IL Stub at runtime. + /// + [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)] + public sealed class GenerateNativeImportAttribute : Attribute + { + } +} +``` + +An alternative solution that merges the existing `DllImportAttribute` with the `GenerateNativeImportAttribute`. This option would be used if user source cannot be modified in any manner. + +``` CSharp +namespace System.Runtime.InteropServices +{ + /// + /// Attribute used to indicate a Source Generator should create a function for marshaling + /// arguments instead of relying on the CLR to generate an IL Stub at runtime. + /// + [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)] + public sealed class GenerateNativeImportAttribute : Attribute + { + /// + /// Enables or disables best-fit mapping behavior when converting Unicode characters + /// to ANSI characters. + /// + /// + public bool BestFitMapping; + + /// + /// Indicates the calling convention of an entry point. + /// + /// + public CallingConvention CallingConvention; + + /// + /// Indicates how to marshal string parameters to the method and controls name mangling. + /// + /// + public CharSet CharSet; + + /// + /// Indicates the name or ordinal of the DLL entry point to be called. + /// + /// + public string? EntryPoint; + + /// + /// Controls whether the System.Runtime.InteropServices.DllImportAttribute.CharSet + /// field causes the common language runtime to search an unmanaged DLL for entry-point + /// names other than the one specified. + /// + /// + public bool ExactSpelling; + + /// + /// Indicates whether unmanaged methods that have HRESULT or retval return values + /// are directly translated or whether HRESULT or retval return values are automatically + /// converted to exceptions. + /// + /// + public bool PreserveSig; + + /// + /// Indicates whether the callee calls the SetLastError Windows API function before + /// returning from the attributed method. + /// + /// + public bool SetLastError; + + /// + /// Enables or disables the throwing of an exception on an unmappable Unicode character + /// that is converted to an ANSI "?" character. + /// + /// + public bool ThrowOnUnmappableChar; + } +} +``` + +## Questions + +* Can the above API be used to provide a reverse P/Invoke stub? + +## References + +[P/Invoke][pinvoke_link] + +[Type Marshaling][typemarshal_link] + +[IL Stubs description][il_stub_link] + + +[dotnet_link]: https://docs.microsoft.com/dotnet/core/tools/dotnet +[typemarshal_link]: https://docs.microsoft.com/dotnet/standard/native-interop/type-marshaling +[pinvoke_link]: https://docs.microsoft.com/dotnet/standard/native-interop/pinvoke +[comwrappers_link]: https://github.com/dotnet/runtime/issues/1845 +[il_stub_link]: https://mattwarren.org/2019/09/26/Stubs-in-the-.NET-Runtime/ +[source_gen_link]: https://github.com/dotnet/roslyn/blob/master/docs/features/generators.md +[blittable_link]: https://docs.microsoft.com/dotnet/framework/interop/blittable-and-non-blittable-types \ No newline at end of file From 517421d10c1e698bab87d64c332a86f040db989b Mon Sep 17 00:00:00 2001 From: Aaron Robinson Date: Thu, 19 Mar 2020 17:51:55 -0700 Subject: [PATCH 3/9] PR feedback --- .../features/source-generator-pinvokes.md | 31 +++++++++---------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/docs/design/features/source-generator-pinvokes.md b/docs/design/features/source-generator-pinvokes.md index cee2ecd65e73e..c3b8bd05b951e 100644 --- a/docs/design/features/source-generator-pinvokes.md +++ b/docs/design/features/source-generator-pinvokes.md @@ -4,7 +4,7 @@ The CLR possesses a rich built-in marshaling mechanism for interoperability with native code that is handled at runtime. This system was designed to free .NET developers from having to author complex and potentially ABI sensitive [type conversion code][typemarshal_link] from a managed to an unmanaged environment. The built-in system works with both [P/Invoke][pinvoke_link] (i.e. `DllImportAttribute`) and [COM interop](https://docs.microsoft.com/dotnet/standard/native-interop/cominterop). The generated portion is typically called an ["IL Stub"][il_stub_link] since the stub is generated by inserting IL instructions into a stream and then passing that stream to the JIT for compilation. -A consequence of this approach is that marshaling code is not immediately available post-link for AOT scenarios (e.g. [`crossgen`](../../workflow/building/coreclr/crossgen.md) and [`crossgen2`](crossgen2-compilation-structure-enhancements.md)). The immediate unavailability of this code has been mitigated by a complex mechanism to have marshalling code generated at AOT time. +A consequence of this approach is that marshaling code is not immediately available post-link for AOT scenarios (e.g. [`crossgen`](../../workflow/building/coreclr/crossgen.md) and [`crossgen2`](crossgen2-compilation-structure-enhancements.md)). The immediate unavailability of this code has been mitigated by a complex mechanism to have marshalling code generated at AOT time. The [IL Linker][ilinker_link] is another tool that struggles with runtime generated code since it is unable to understand all potential used types without seeing what is generated. The user experience of the built-in generation initially appears ideal, but there are several negative consequences that make the system costly in the long term: @@ -46,8 +46,6 @@ At (A) in the above code snippet, the runtime is told to look for an export name 1) During invocation, the function declaration is determined to be an external call requiring marshaling. Given the defined properties in the `DllImportAttribute` instance as well as the metadata of the user-defined signature an IL Stub is generated. - * An IL Stub is always generated when an assembly is compiled in `Debug`. In `Release` builds it is possible no IL Stub is generated and instead the JIT will elide the stub and inline the invocation. We will come back to this inlining below. - 2) The runtime attempts to find a binary with the name supplied in `DllImportAttribute`. * Discovery of the target binary is complicated and can be influenced by the [`AssemblyLoadContext`](https://docs.microsoft.com/dotnet/api/system.runtime.loader.assemblyloadcontext) and [`NativeLibrary`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.nativelibrary) classes. @@ -56,9 +54,9 @@ At (A) in the above code snippet, the runtime is told to look for an export name * This process is also influenced by additional `DllImportAttribute` properties as well as by the underlying platform. For example, the [Win32 API ANSI/UNICODE convention](https://docs.microsoft.com/windows/win32/intl/conventions-for-function-prototypes) on Windows is respected and a(n) `A`/`W` suffix may be appended to the function name if it is not immediately found. -4) The IL Stub is called like any .NET method, but the address of the export is passed to the generated IL Stub via a 'hidden' argument. +4) The IL Stub is called like any other .NET method. -5) The IL Stub then marshals arguments as appropriate and invokes the export via the `calli` instruction. +5) The IL Stub marshals arguments as appropriate and invokes the export via the `calli` instruction. 6) Once the export returns control to the IL Stub the marshaling logic cleans up and ensures any returned data is marshaled back out to the calling function. @@ -70,7 +68,7 @@ An example of how the previous P/Invoke snippet could be transformed is below. T ``` CSharp /* A */ [DllImportAttribute("Kernel32.dll")] -/* D */ [GenerateNativeImportAttribute] +/* D */ [GeneratedDllImportAttribute] /* B */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount); ... long count; @@ -82,14 +80,14 @@ Observe point (D), the new attribute. This attribute provides an indication to a `Program.cs` (In-memory updated code) ``` CSharp -/* D */ [GenerateNativeImportAttribute] +/* D */ [GeneratedDllImportAttribute] /* B */ partial static bool QueryPerformanceCounter(out long lpPerformanceCount); ... long count; /* C */ QueryPerformanceCounter(out count); ``` -During the source generation process the `DllImportAttribute` (A) would be removed. The `GenerateNativeImportAttribute` (D) would remain to provide a metadata indication that the function was auto-generated. Also note that the method declaration has been updated and marked `partial`. The Source Generator would then generate the source for this partial method. The invocation (C) remains unchanged. +During the source generation process the `DllImportAttribute` (A) would be removed. The `GeneratedDllImportAttribute` (D) would remain to provide a metadata indication that the function was auto-generated. Also note that the method declaration has been updated and marked `partial`. The Source Generator would then generate the source for this partial method. The invocation (C) remains unchanged. `Stubs.g.cs`: @@ -111,7 +109,7 @@ During the source generation process the `DllImportAttribute` (A) would be remov The Source Generator would generate the implementation of the partial method (E) in a separate translation unit (`Stubs.g.cs`). At point (F) the `DllImportAttribute` declaration from the user's original declaration is copied into a private P/Invoke specifically for the generated code. The P/Invoke signature from the original declaration would be modified to contain only [blittable types][blittable_link] to ensure the JIT could inline the invocation. Finally note that the user's original function signature would remain in to avoid impacting existing callsites. -In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types, and it is these rules that once shipped become the de facto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unmanaged environment. +In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types, and it is these rules that once shipped become the de facto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unmanaged environment. With the IL Stub generation extracted from the runtime new type marshaling (e.g. `Span`) could be introduced without requiring an corresponding update to the runtime itself. The `Span` type is good example of a type that at present has no support for marshaling, but with Source Generators, users could update to the latest generator and have support without changing the runtime. ### Source Generator constraint mitigations @@ -119,9 +117,9 @@ In the current Source Generator design modification of any user written code is * In-memory source modification - This would yield the ideal design above. -* On disk source modification - Altering source on disk would require the `GenerateNativeImportAttribute` to contain all properties from `DllImportAttribute`. This is required since removing the `DllImportAttribute` would be impossible and thus would cause issues during build since the method must also be marked as `partial`. +* On disk source modification - Altering source on disk would require the `GeneratedDllImportAttribute` to contain all properties from `DllImportAttribute`. This is required since removing the `DllImportAttribute` would be impossible and thus would cause issues during build since the method must also be marked as `partial`. -* No source modification - This aligns with the current Source Generator constraints. It would be possible to create a [Roslyn Analyzer and Code fix](https://github.com/dotnet/roslyn/wiki/Getting-Started-Writing-a-Custom-Analyzer-&-Code-Fix) to aid the developer in converting their `DllImportAttribute` marked functions to use `GenerateNativeImportAttribute`. Furthermore, the function would need to be updated to have the `partial` keyword and potentially the enclosing class. +* No source modification - This aligns with the current Source Generator constraints. It would be possible to create a [Roslyn Analyzer and Code fix](https://github.com/dotnet/roslyn/wiki/Getting-Started-Writing-a-Custom-Analyzer-&-Code-Fix) to aid the developer in converting their `DllImportAttribute` marked functions to use `GeneratedDllImportAttribute`. Furthermore, the function would need to be updated to have the `partial` keyword and potentially the enclosing class. ## Proposed APIs @@ -135,13 +133,13 @@ namespace System.Runtime.InteropServices /// arguments instead of relying on the CLR to generate an IL Stub at runtime. /// [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)] - public sealed class GenerateNativeImportAttribute : Attribute + public sealed class GeneratedDllImportAttribute : Attribute { } } ``` -An alternative solution that merges the existing `DllImportAttribute` with the `GenerateNativeImportAttribute`. This option would be used if user source cannot be modified in any manner. +An alternative solution that merges the existing `DllImportAttribute` with the `GeneratedDllImportAttribute`. This option would be used if user source cannot be modified in any manner. ``` CSharp namespace System.Runtime.InteropServices @@ -151,7 +149,7 @@ namespace System.Runtime.InteropServices /// arguments instead of relying on the CLR to generate an IL Stub at runtime. /// [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)] - public sealed class GenerateNativeImportAttribute : Attribute + public sealed class GeneratedDllImportAttribute : Attribute { /// /// Enables or disables best-fit mapping behavior when converting Unicode characters @@ -229,5 +227,6 @@ namespace System.Runtime.InteropServices [pinvoke_link]: https://docs.microsoft.com/dotnet/standard/native-interop/pinvoke [comwrappers_link]: https://github.com/dotnet/runtime/issues/1845 [il_stub_link]: https://mattwarren.org/2019/09/26/Stubs-in-the-.NET-Runtime/ -[source_gen_link]: https://github.com/dotnet/roslyn/blob/master/docs/features/generators.md -[blittable_link]: https://docs.microsoft.com/dotnet/framework/interop/blittable-and-non-blittable-types \ No newline at end of file +[source_gen_link]: https://github.com/dotnet/roslyn/blob/features/source-generators/docs/features/source-generators.md +[blittable_link]: https://docs.microsoft.com/dotnet/framework/interop/blittable-and-non-blittable-types +[ilinker_link]: https://github.com/mono/linker \ No newline at end of file From 717a47e7474f2aad38fbdaecb69cbd824935f0cd Mon Sep 17 00:00:00 2001 From: Aaron Robinson Date: Wed, 25 Mar 2020 11:59:05 -0700 Subject: [PATCH 4/9] Feedback from PR --- .../features/source-generator-pinvokes.md | 78 +++++++------------ 1 file changed, 30 insertions(+), 48 deletions(-) diff --git a/docs/design/features/source-generator-pinvokes.md b/docs/design/features/source-generator-pinvokes.md index c3b8bd05b951e..a5b9f948cdcc6 100644 --- a/docs/design/features/source-generator-pinvokes.md +++ b/docs/design/features/source-generator-pinvokes.md @@ -4,7 +4,7 @@ The CLR possesses a rich built-in marshaling mechanism for interoperability with native code that is handled at runtime. This system was designed to free .NET developers from having to author complex and potentially ABI sensitive [type conversion code][typemarshal_link] from a managed to an unmanaged environment. The built-in system works with both [P/Invoke][pinvoke_link] (i.e. `DllImportAttribute`) and [COM interop](https://docs.microsoft.com/dotnet/standard/native-interop/cominterop). The generated portion is typically called an ["IL Stub"][il_stub_link] since the stub is generated by inserting IL instructions into a stream and then passing that stream to the JIT for compilation. -A consequence of this approach is that marshaling code is not immediately available post-link for AOT scenarios (e.g. [`crossgen`](../../workflow/building/coreclr/crossgen.md) and [`crossgen2`](crossgen2-compilation-structure-enhancements.md)). The immediate unavailability of this code has been mitigated by a complex mechanism to have marshalling code generated at AOT time. The [IL Linker][ilinker_link] is another tool that struggles with runtime generated code since it is unable to understand all potential used types without seeing what is generated. +A consequence of this approach is that marshaling code is not immediately available post-link for AOT scenarios (e.g. [`crossgen`](../../workflow/building/coreclr/crossgen.md) and [`crossgen2`](crossgen2-compilation-structure-enhancements.md)). The immediate unavailability of this code has been mitigated by a complex mechanism to have marshalling code generated by during AOT compilation. The [IL Linker][ilinker_link] is another tool that struggles with runtime generated code since it is unable to understand all potential used types without seeing what is generated. The user experience of the built-in generation initially appears ideal, but there are several negative consequences that make the system costly in the long term: @@ -15,7 +15,7 @@ The user experience of the built-in generation initially appears ideal, but ther * Example involving COM marshaling: https://github.com/dotnet/coreclr/pull/23974. * Debugging the auto-generated marshaling IL Stub is difficult for runtime developers and close to impossible for consumers of P/Invokes. -This is not to say the P/Invoke system should be completely redesigned. The current system is heavily used and its simplicity for consuming native assets is a benefit. Rather this new mechanism is designed to provide a way for marshaling code to be generated by an external tool but integrate with `DllImportAttribute` in a way that isn't onerous on current .NET developers. +This is not to say the P/Invoke system should be completely redesigned. The current system is heavily used and its simplicity for consuming native assets is a benefit. Rather this new mechanism is designed to provide a way for marshaling code to be generated by an external tool but work with existing `DllImportAttribute` practices in a way that isn't onerous on current .NET developers. The [Roslyn Compiler](https://github.com/dotnet/roslyn) team is working on a [Source Generator feature][source_gen_link] that will allow the generation of additional source files that can be added to an assembly during the compilation process - the runtime generation IL Stubs is an in-memory version of this scenario. @@ -26,9 +26,22 @@ The [Roslyn Compiler](https://github.com/dotnet/roslyn) team is working on a [So * [Source generators][source_gen_link] * Branch: https://github.com/dotnet/roslyn/tree/features/source-generators +* Support for non-`void` return types in [`partial`](https://docs.microsoft.com/dotnet/csharp/language-reference/keywords/partial-method) methods. + * https://github.com/dotnet/csharplang/issues/3301 + ## Design -Using Source Generators is focused on integrating with existing uses of `DllImportAttribute` from an invocation point of view (i.e. callsites should not need to be updated). The idea behind Source Generators is that code for some scenarios can be precomputed using user declared types and logic thus avoiding the need to generator code at runtime. The desire is then to provide a way that existing code can continue to function but can be modified in a way that allows it to leverage this new compiler feature. +Using Source Generators is focused on integrating with existing `DllImportAttribute` practices from an invocation point of view (i.e. callsites should not need to be updated). The idea behind Source Generators is that code for some scenarios can be precomputed using user declared types and logic thus avoiding the need to generator code at runtime. + +**Goals** + +* Allow P/Invoke interop evolution independently on runtime. +* High performance: No reflection at runtime, compatible in an AOT scenario. + +**Non-Goals** + +* 100 % parity with existing P/Invoke marshaling rules. +* Zero code change for the developers. ### P/Invoke Walkthrough @@ -62,37 +75,26 @@ At (A) in the above code snippet, the runtime is told to look for an export name ### Source Generator Integration -An example of how the previous P/Invoke snippet could be transformed is below. This example is using the proposed API in this document. The Source Generator currently has a limitation that we will ignore at present - the inability to modify user written code. This limitation is ignored to present an optimal user experience, but will be discussed later in case the current limitation becomes a hard requirement. +An example of how the previous P/Invoke snippet could be transformed is below. This example is using the proposed API in this document. The Source Generator has a restriction of no user code modification so that is reflected in the design and mitigations for easing code adoption is presented later. -`Program.cs` (Pre-generated code) +`Program.cs` (User written code) ``` CSharp -/* A */ [DllImportAttribute("Kernel32.dll")] -/* D */ [GeneratedDllImportAttribute] -/* B */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount); +/* A */ [GeneratedDllImportAttribute("Kernel32.dll")] +/* B */ partial static bool QueryPerformanceCounter(out long lpPerformanceCount); ... long count; /* C*/ QueryPerformanceCounter(out count); ``` -Observe point (D), the new attribute. This attribute provides an indication to a Source Generator that the following declaration should be updated. +Observe point (A), the new attribute. This attribute provides an indication to a Source Generator that the following declaration represents a native export that will be called via Source Generated stub. -`Program.cs` (In-memory updated code) - -``` CSharp -/* D */ [GeneratedDllImportAttribute] -/* B */ partial static bool QueryPerformanceCounter(out long lpPerformanceCount); -... -long count; -/* C */ QueryPerformanceCounter(out count); -``` - -During the source generation process the `DllImportAttribute` (A) would be removed. The `GeneratedDllImportAttribute` (D) would remain to provide a metadata indication that the function was auto-generated. Also note that the method declaration has been updated and marked `partial`. The Source Generator would then generate the source for this partial method. The invocation (C) remains unchanged. +During the source generation process the metadata in the `GeneratedDllImportAttribute` (A) would be used to generate a stub and invoke the desired native export. Also note that the method declaration is marked `partial`. The Source Generator would then generate the source for this partial method. The invocation (C) remains unchanged to that of usage involving `DllImportAttribute`. `Stubs.g.cs`: ``` CSharp -/* E */ partial static bool QueryPerformanceCounter(out long lpPerformanceCount) +/* D */ partial static bool QueryPerformanceCounter(out long lpPerformanceCount) { unsafe { @@ -104,42 +106,22 @@ During the source generation process the `DllImportAttribute` (A) would be remov } [DllImportAttribute("Kernel32.dll")] -/* F */ private static extern int QueryPerformanceCounter(long* lpPerformanceCount); +/* E */ private static extern int QueryPerformanceCounter(long* lpPerformanceCount); ``` -The Source Generator would generate the implementation of the partial method (E) in a separate translation unit (`Stubs.g.cs`). At point (F) the `DllImportAttribute` declaration from the user's original declaration is copied into a private P/Invoke specifically for the generated code. The P/Invoke signature from the original declaration would be modified to contain only [blittable types][blittable_link] to ensure the JIT could inline the invocation. Finally note that the user's original function signature would remain in to avoid impacting existing callsites. +The Source Generator would generate the implementation of the partial method (D) in a separate translation unit (`Stubs.g.cs`). At point (E) a `DllImportAttribute` declaration is created based on the user's original declaration (A) for a private P/Invoke specifically for the generated code. The P/Invoke signature from the original declaration would be modified to contain only [blittable types][blittable_link] to ensure the JIT could inline the invocation. Finally note that the user's original function signature would remain in to avoid impacting existing callsites. In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types, and it is these rules that once shipped become the de facto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unmanaged environment. With the IL Stub generation extracted from the runtime new type marshaling (e.g. `Span`) could be introduced without requiring an corresponding update to the runtime itself. The `Span` type is good example of a type that at present has no support for marshaling, but with Source Generators, users could update to the latest generator and have support without changing the runtime. -### Source Generator constraint mitigations - -In the current Source Generator design modification of any user written code is not permitted. This includes modification of any non-functional metadata (e.g. Attributes). The above design mentions modification of user source so the current design may be untenable. Let us define the following options and describe how the above design would be modified. +### Adoption of Source Generator for existing code -* In-memory source modification - This would yield the ideal design above. +In the current Source Generator design modification of any user written code is not permitted. This includes modification of any non-functional metadata (e.g. Attributes). The above design therefore introduces a new attribute and signature for consumption of a native export. Therefore, in order to consume Source Generator's users would need to update their source and adoption could be stunted by this. -* On disk source modification - Altering source on disk would require the `GeneratedDllImportAttribute` to contain all properties from `DllImportAttribute`. This is required since removing the `DllImportAttribute` would be impossible and thus would cause issues during build since the method must also be marked as `partial`. +As a mitigation it would be possible to create a [Roslyn Analyzer and Code fix](https://github.com/dotnet/roslyn/wiki/Getting-Started-Writing-a-Custom-Analyzer-&-Code-Fix) to aid the developer in converting their `DllImportAttribute` marked functions to use `GeneratedDllImportAttribute`. Furthermore, the function would need to be updated to have the `partial` keyword and potentially the enclosing class. -* No source modification - This aligns with the current Source Generator constraints. It would be possible to create a [Roslyn Analyzer and Code fix](https://github.com/dotnet/roslyn/wiki/Getting-Started-Writing-a-Custom-Analyzer-&-Code-Fix) to aid the developer in converting their `DllImportAttribute` marked functions to use `GeneratedDllImportAttribute`. Furthermore, the function would need to be updated to have the `partial` keyword and potentially the enclosing class. - -## Proposed APIs - -The ideal solution with support for minor user defined source updates using Source Generators: - -``` CSharp -namespace System.Runtime.InteropServices -{ - /// - /// Attribute used to indicate a Source Generator should create a function for marshaling - /// arguments instead of relying on the CLR to generate an IL Stub at runtime. - /// - [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)] - public sealed class GeneratedDllImportAttribute : Attribute - { - } -} -``` +## Proposed API -An alternative solution that merges the existing `DllImportAttribute` with the `GeneratedDllImportAttribute`. This option would be used if user source cannot be modified in any manner. +Given the Source Generator restrictions and potential confusion about overloaded attribute usage, the new attribute merges the existing `DllImportAttribute` with `GeneratedDllImportAttribute`. ``` CSharp namespace System.Runtime.InteropServices From f9270e4a98099b573ff6420f9be90d2ecbc9d8a8 Mon Sep 17 00:00:00 2001 From: Aaron Robinson Date: Wed, 25 Mar 2020 12:12:34 -0700 Subject: [PATCH 5/9] Update docs/design/features/source-generator-pinvokes.md Co-Authored-By: Jan Kotas --- docs/design/features/source-generator-pinvokes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/features/source-generator-pinvokes.md b/docs/design/features/source-generator-pinvokes.md index a5b9f948cdcc6..ee9186cc5540f 100644 --- a/docs/design/features/source-generator-pinvokes.md +++ b/docs/design/features/source-generator-pinvokes.md @@ -121,7 +121,7 @@ As a mitigation it would be possible to create a [Roslyn Analyzer and Code fix]( ## Proposed API -Given the Source Generator restrictions and potential confusion about overloaded attribute usage, the new attribute merges the existing `DllImportAttribute` with `GeneratedDllImportAttribute`. +Given the Source Generator restrictions and potential confusion about overloaded attribute usage, the new `GeneratedDllImportAttribute` attribute mirrors the existing `DllImportAttribute`. ``` CSharp namespace System.Runtime.InteropServices @@ -211,4 +211,4 @@ namespace System.Runtime.InteropServices [il_stub_link]: https://mattwarren.org/2019/09/26/Stubs-in-the-.NET-Runtime/ [source_gen_link]: https://github.com/dotnet/roslyn/blob/features/source-generators/docs/features/source-generators.md [blittable_link]: https://docs.microsoft.com/dotnet/framework/interop/blittable-and-non-blittable-types -[ilinker_link]: https://github.com/mono/linker \ No newline at end of file +[ilinker_link]: https://github.com/mono/linker From caa8af39065953f1caa95b192a41f709fd588bed Mon Sep 17 00:00:00 2001 From: Aaron Robinson Date: Wed, 25 Mar 2020 13:54:44 -0700 Subject: [PATCH 6/9] Update Questions section to FAQs --- docs/design/features/source-generator-pinvokes.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/design/features/source-generator-pinvokes.md b/docs/design/features/source-generator-pinvokes.md index ee9186cc5540f..2f02f5c9ac7f8 100644 --- a/docs/design/features/source-generator-pinvokes.md +++ b/docs/design/features/source-generator-pinvokes.md @@ -111,7 +111,7 @@ During the source generation process the metadata in the `GeneratedDllImportAttr The Source Generator would generate the implementation of the partial method (D) in a separate translation unit (`Stubs.g.cs`). At point (E) a `DllImportAttribute` declaration is created based on the user's original declaration (A) for a private P/Invoke specifically for the generated code. The P/Invoke signature from the original declaration would be modified to contain only [blittable types][blittable_link] to ensure the JIT could inline the invocation. Finally note that the user's original function signature would remain in to avoid impacting existing callsites. -In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types, and it is these rules that once shipped become the de facto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unmanaged environment. With the IL Stub generation extracted from the runtime new type marshaling (e.g. `Span`) could be introduced without requiring an corresponding update to the runtime itself. The `Span` type is good example of a type that at present has no support for marshaling, but with Source Generators, users could update to the latest generator and have support without changing the runtime. +In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types, and it is these rules that once shipped become the de facto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unmanaged environment. With the IL Stub generation extracted from the runtime new type marshaling (e.g. [`Span`](https://docs.microsoft.com/dotnet/api/system.span-1)) could be introduced without requiring an corresponding update to the runtime itself. The `Span` type is good example of a type that at present has no support for marshaling, but with this proposal users could update to the latest generator and have support without changing the runtime. ### Adoption of Source Generator for existing code @@ -191,10 +191,16 @@ namespace System.Runtime.InteropServices } ``` -## Questions +## FAQs * Can the above API be used to provide a reverse P/Invoke stub? + * No. Reverse P/Invoke invocation is performed via a delegate and integrating this proposal would prove difficult. Alternative approach is to leverage the [`NativeCallableAttribute`](https://github.com/dotnet/runtime/issues/32462) feature. + +* How will users get error messages during source generator? + + * The Source Generator API will be permitted to provide warnings and errors through the [Roslyn SDK](https://docs.microsoft.com/dotnet/csharp/roslyn-sdk/). + ## References [P/Invoke][pinvoke_link] From 61a77bb6f60cc308e35e94303ee1c91d22c57c3a Mon Sep 17 00:00:00 2001 From: Aaron Robinson Date: Wed, 25 Mar 2020 15:46:30 -0700 Subject: [PATCH 7/9] Clarifying updates. --- .../features/source-generator-pinvokes.md | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/design/features/source-generator-pinvokes.md b/docs/design/features/source-generator-pinvokes.md index 2f02f5c9ac7f8..1f60fd3ec9d0d 100644 --- a/docs/design/features/source-generator-pinvokes.md +++ b/docs/design/features/source-generator-pinvokes.md @@ -23,7 +23,7 @@ The [Roslyn Compiler](https://github.com/dotnet/roslyn) team is working on a [So ### Requirements -* [Source generators][source_gen_link] +* [Source Generators][source_gen_link] * Branch: https://github.com/dotnet/roslyn/tree/features/source-generators * Support for non-`void` return types in [`partial`](https://docs.microsoft.com/dotnet/csharp/language-reference/keywords/partial-method) methods. @@ -31,11 +31,11 @@ The [Roslyn Compiler](https://github.com/dotnet/roslyn) team is working on a [So ## Design -Using Source Generators is focused on integrating with existing `DllImportAttribute` practices from an invocation point of view (i.e. callsites should not need to be updated). The idea behind Source Generators is that code for some scenarios can be precomputed using user declared types and logic thus avoiding the need to generator code at runtime. +Using Source Generators is focused on integrating with existing `DllImportAttribute` practices from an invocation point of view (i.e. callsites should not need to be updated). The idea behind Source Generators is that code for some scenarios can be precomputed using user declared types, metadata, and logic thus avoiding the need to generate code at runtime. **Goals** -* Allow P/Invoke interop evolution independently on runtime. +* Allow P/Invoke interop evolution independently of runtime. * High performance: No reflection at runtime, compatible in an AOT scenario. **Non-Goals** @@ -87,11 +87,11 @@ long count; /* C*/ QueryPerformanceCounter(out count); ``` -Observe point (A), the new attribute. This attribute provides an indication to a Source Generator that the following declaration represents a native export that will be called via Source Generated stub. +Observe point (A), the new attribute. This attribute provides an indication to a Source Generator that the following declaration represents a native export that will be called via a generated stub. During the source generation process the metadata in the `GeneratedDllImportAttribute` (A) would be used to generate a stub and invoke the desired native export. Also note that the method declaration is marked `partial`. The Source Generator would then generate the source for this partial method. The invocation (C) remains unchanged to that of usage involving `DllImportAttribute`. -`Stubs.g.cs`: +`Stubs.g.cs` (Source Generator code) ``` CSharp /* D */ partial static bool QueryPerformanceCounter(out long lpPerformanceCount) @@ -113,11 +113,11 @@ The Source Generator would generate the implementation of the partial method (D) In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types, and it is these rules that once shipped become the de facto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unmanaged environment. With the IL Stub generation extracted from the runtime new type marshaling (e.g. [`Span`](https://docs.microsoft.com/dotnet/api/system.span-1)) could be introduced without requiring an corresponding update to the runtime itself. The `Span` type is good example of a type that at present has no support for marshaling, but with this proposal users could update to the latest generator and have support without changing the runtime. -### Adoption of Source Generator for existing code +### Adoption of Source Generator -In the current Source Generator design modification of any user written code is not permitted. This includes modification of any non-functional metadata (e.g. Attributes). The above design therefore introduces a new attribute and signature for consumption of a native export. Therefore, in order to consume Source Generator's users would need to update their source and adoption could be stunted by this. +In the current Source Generator design modification of any user written code is not permitted. This includes modification of any non-functional metadata (e.g. Attributes). The above design therefore introduces a new attribute and signature for consumption of a native export. In order to consume Source Generators, users would need to update their source and adoption could be stunted by this requirement. -As a mitigation it would be possible to create a [Roslyn Analyzer and Code fix](https://github.com/dotnet/roslyn/wiki/Getting-Started-Writing-a-Custom-Analyzer-&-Code-Fix) to aid the developer in converting their `DllImportAttribute` marked functions to use `GeneratedDllImportAttribute`. Furthermore, the function would need to be updated to have the `partial` keyword and potentially the enclosing class. +As a mitigation it would be possible to create a [Roslyn Analyzer and Code fix](https://github.com/dotnet/roslyn/wiki/Getting-Started-Writing-a-Custom-Analyzer-&-Code-Fix) to aid the developer in converting `DllImportAttribute` marked functions to use `GeneratedDllImportAttribute`. Additionally, the function signature would need to be updated to remove the `extern` keyword and add the `partial` keyword to the function and potentially the enclosing class. ## Proposed API @@ -201,6 +201,10 @@ namespace System.Runtime.InteropServices * The Source Generator API will be permitted to provide warnings and errors through the [Roslyn SDK](https://docs.microsoft.com/dotnet/csharp/roslyn-sdk/). +* Is it be possible to completely replicate the marshaling rules in the current built-in system using existing .NET APIs? + + * No. There are rules and semantics that would be difficult to replicate with the current .NET API surface. Additional .NET APIs will likely need to be added in order to allow a Source Generator implementation to provide identical semantics with the built-in system. + ## References [P/Invoke][pinvoke_link] From dc9c3a00037a33f6e7ef88f7c8fd16a1610911c2 Mon Sep 17 00:00:00 2001 From: Aaron Robinson Date: Sun, 29 Mar 2020 21:36:27 -0700 Subject: [PATCH 8/9] Update docs/design/features/source-generator-pinvokes.md Co-Authored-By: Andrew Boyarshin --- docs/design/features/source-generator-pinvokes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/features/source-generator-pinvokes.md b/docs/design/features/source-generator-pinvokes.md index 1f60fd3ec9d0d..f48ff42695254 100644 --- a/docs/design/features/source-generator-pinvokes.md +++ b/docs/design/features/source-generator-pinvokes.md @@ -201,7 +201,7 @@ namespace System.Runtime.InteropServices * The Source Generator API will be permitted to provide warnings and errors through the [Roslyn SDK](https://docs.microsoft.com/dotnet/csharp/roslyn-sdk/). -* Is it be possible to completely replicate the marshaling rules in the current built-in system using existing .NET APIs? +* Will it be possible to completely replicate the marshaling rules in the current built-in system using existing .NET APIs? * No. There are rules and semantics that would be difficult to replicate with the current .NET API surface. Additional .NET APIs will likely need to be added in order to allow a Source Generator implementation to provide identical semantics with the built-in system. From 20ac6554c583a7f65a20c289d527bd08d7729de7 Mon Sep 17 00:00:00 2001 From: Aaron R Robinson Date: Sun, 29 Mar 2020 21:44:24 -0700 Subject: [PATCH 9/9] Feedback on FAQs --- docs/design/features/source-generator-pinvokes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/features/source-generator-pinvokes.md b/docs/design/features/source-generator-pinvokes.md index f48ff42695254..dcd0944e8d44c 100644 --- a/docs/design/features/source-generator-pinvokes.md +++ b/docs/design/features/source-generator-pinvokes.md @@ -203,7 +203,7 @@ namespace System.Runtime.InteropServices * Will it be possible to completely replicate the marshaling rules in the current built-in system using existing .NET APIs? - * No. There are rules and semantics that would be difficult to replicate with the current .NET API surface. Additional .NET APIs will likely need to be added in order to allow a Source Generator implementation to provide identical semantics with the built-in system. + * No. There are rules and semantics that would be difficult to replicate with the current .NET API surface. Additional .NET APIs will likely need to be added in order to allow a Source Generator implementation to provide identical semantics with the built-in system (e.g. Respecting the semantics of [`DllImportAttribute.SetLastError`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.dllimportattribute.setlasterror)). ## References