Initial proposal for P/Invokes via Source Generators

docs/design/features/MarshalDirective.md

@@ -0,0 +1,206 @@
+# User Defined Marshal Directive
+
+## Purpose
+
+The CLR possesses a rich built-in marshaling mechanism for interopability with native code that is handled at runtime. This system was written to free .NET developers from having to author complex and potentially ABI sensitive [type conversion code][typemarshal_link] from a managed to an unmanaged environment. The built-in system works with both [P/Invoke][pinvoke_link] (i.e. `DllImportAttribute`) and [COM interop](https://docs.microsoft.com/dotnet/standard/native-interop/cominterop). The generated portion is typically called an ["IL Stub"][il_stub_link] since the stub is generated by inserting IL instructions into a stream and then passing it off to the JIT for compilation. 
+
+One consequence of this approach is that marshaling code is not immediately available post link for AOT scenarios (e.g. [`crossgen`](../../workflow/building/coreclr/crossgen.md) and [`crossgen2`](crossgen2-compilation-structure-enhancements.md)). The immediate unavailability of this code has been mitigated by a complex mechanism to have marshalling code generated at AOT time.
+
+The user experience of the built-in generation initially appears ideal, but there are several negative consequences that make the system costly in the long term:
+
+* Bug fixes in the marshaling system require an update to the entire runtime.
+* New types require enhancements to the marshaling system for efficient marshal behavior.
+  * [`ICustomMarshaler`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.icustommarshaler) incurs a substantial performance penalty.
+* Once a marshaling bug becomes expected behavior the bug is difficult to fix. This is due to user reliance on shipped behavior and since the marshaling system is built into the runtime there aren't ways to select previous or new behavior.
+  * Example involving COM marshaling: https://github.com/dotnet/coreclr/pull/23974.
+
+This is not to say the P/Invoke system should be completely redesigned. The current system is heavily used its current simplicity for consuming native assets is a benefit. Rather this new mechansim is designed to provide a way for marshaling code to be generated by an external tool, but integrate with `DllImportAttribute` in a natural way.
+
+**Note** This attribute is not designed to work with COM interop. The [`ComWrappers`][comwrappers_link] API should be used instead.
+
+### Requirements
+
+* [C# Function pointers][csharp_fptr_link].
+  * https://github.com/dotnet/roslyn/issues/39865
+
+## Design
+
+The Marshal Directive design is focused on a natural integration with existing uses of `DllImportAttribute`. The current and anticipated P/Invoke algorithm is presented below using a simple example.
+
+``` CSharp
+[DllImportAttribute("Kernel32.dll")]
+/* A */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount);
+...
+long count;
+/* B */ QueryPerformanceCounter(out count);
+```
+
+At (A) in the above code snippet the runtime is told to look for an export name `QueryPerformanceCounter` in the `Kernel32.dll` binary. There are many additional attributes on the `DllImportAttribute` that help with export discovery and can influence the semantics of the generated IL Stub. Point (B) represents an invocation of the P/Invoke. The majority of the work occurs at (B) at runtime, since (A) is merely a declaration the compiler uses to embed the relevant details into the metadata.
+
+1) During invocation, the function declaration is determined to be an external call requiring marshaling. Given the defined properties in the `DllImportAttribute` instance as well as the metadata of the user-defined signature an IL Stub is generated.
+
+    * An IL Stub is always generated when an assembly is compiled in `Debug`. In `Release` builds it is possible no IL Stub is generated and instead the JIT will elide the stub and inline the invocation.
+
+    * **Marshal Directive**: The user would define a method to call instead of asking the runtime to generate one. This would not impact the scenario where the JIT has determined the P/Invoke can be inlined.
+
+2) The runtime attempts to find a binary with the name supplied in `DllImportAttribute`.
+
+    * Discovery of the target binary is complicated and can be influenced by the [`AssemblyLoadContext`](https://docs.microsoft.com/dotnet/api/system.runtime.loader.assemblyloadcontext) and [`NativeLibrary`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.nativelibrary) classes.
+
+3) Once the binary is found and loaded into the runtime, it is queried for the expected export name. The name of the attributed function is used by default but this is configurable by the [`DllImportAttribute.EntryPoint`](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.dllimportattribute.entrypoint) property.
+
+    * This process is also influenced by additional `DllImportAttribute` properties as well as by the underlying platform. For example, the [Win32 API ANSI/UNICODE convention](https://docs.microsoft.com/windows/win32/intl/conventions-for-function-prototypes) on Windows is respected and a(n) `A`/`W` suffix may be appended to the function name if it is not immediately found.
+
+4) The IL Stub is called like any .NET method, but the address of the export is passed to the generated IL Stub via a 'hidden' argument.
+
+    * **Marshal Directive**: The user's defined method would require an identical signature to that of the one typically defined, but a new argument of type `IntPtr` should be added as the first argument. The runtime would be responsible for providing the export to the user defined function.
+
+5) The IL Stub then marshals arguments as appropriate and invokes the export via the `calli` instruction.
+
+    * **Marshal Directive**: The user's defined method would cast the supplied export to an appropriate C# function pointer type and invoke the call.
+
+6) Once the export returns control to the IL Stub the marshaling logic cleans up and ensures any returned data is marshaled back out to the calling function.
+
+An example of how the snippet above could be transformed is below. The example is using the proposed API in this document as well as the [C# function pointer proposal][csharp_fptr_link]. Observe points (C), the new attribute, (D), a `partial` class, and (E) the stub with additional `IntPtr` argument that will be called in lieu of a runtime generated IL Stub.
+
+``` CSharp
+[DllImportAttribute("Kernel32.dll")]
+/* C */ [MarshalDirectiveAttribute(
+        Type = MarshalDirectiveType.UserDefinedStub,
+        UserFunctionType = typeof(Stubs),
+        UserDefinedStub = nameof(QueryPerformanceCounterStub))]
+/* A */ extern static bool QueryPerformanceCounter(out long lpPerformanceCount);
+...
+long count;
+/* B */ QueryPerformanceCounter(out count);
+
+/* D */  class partial Stubs
+{
+    /* E */ public static bool QueryPerformanceCounterStub(IntPtr fptr, out long lpPerformanceCount)
+    {
+        unsafe
+        {
+            long result = 0;
+            var tfptr = (delegate* stdcall<long*, int>)fptr;
+            bool success = tfptr(&result) != 0;
+            lpPerformanceCount = result;
+            return success;
+        }
+    }
+}
+```
+
+The contents of the above stub are trivial and in a `Release` build wouldn't even exist. More complicated examples may require [GC handle pinning](https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.gchandle) or native memory allocation. These complicated examples could be hand authored or generated through a third-party tool.
+
+In this system it is not defined how marshaling of specific types would be performed. The built-in runtime has complex rules for some types and it is these rules that once shipped become the defacto standard - often times regardless if the behavior is a bug or not. The design here is not concerned with how the arguments go from a managed to unamanged environment.
+
+### Proposed API
+``` CSharp
+namespace System.Runtime.InteropServices
+{
+    /// <summary>
+    /// User defined marshaling directive
+    /// </summary>
+    public enum MarshalDirectiveType
+    {
+        /// <summary>
+        /// The attribute does nothing.
+        /// </summary>
+        Default = 0,
+
+        /// <summary>
+        /// The user can supply a managed function that will be called in-place
+        /// of a generated function stub.
+        /// </summary>
+        /// <seealso cref="MarshalDirectiveAttribute.UserDefinedStub"/>
+        UserDefinedStub,
+    }
+
+    /// <summary>
+    /// Attribute used to define how marshaling stubs should be defined for
+    /// the associated managed function declaration.
+    /// </summary>
+    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)]
+    public sealed class MarshalDirectiveAttribute : Attribute
+    {
+        /// <summary>
+        /// Default constructor.
+        /// </summary>
+        public MarshalDirectiveAttribute()
+        {
+            this.Type = MarshalDirectiveType.Default;
+        }
+
+        /// <summary>
+        /// The directive type.
+        /// </summary>
+        /// <remarks>
+        /// The default value is <see cref="MarshalDirectiveType.Default "/>.
+        /// </remarks>
+        public MarshalDirectiveType Type { get; set; }
+
+        /// <summary>
+        /// The type to use to search for user defined functions.
+        /// </summary>
+        public Type UserFunctionType { get; set; }
+
+        /// <summary>
+        /// The name of the user defined stub to call.
+        /// Only used if <see cref="MarshalDirectiveAttribute.Type"/> is defined as
+        /// <see cref="MarshalDirectiveType.UserDefinedStub"/>.
+        /// </summary>
+        /// <remarks>
+        /// The supplied name will be searched for on
+        /// <see cref="MarshalDirectiveAttribute.UserFunctionType"/>.
+        ///
+        /// The supplied function must conform to a signature of that matches
+        /// the managed definition of the function, except for the first argument
+        /// which will be supplied by the runtime. The first argument will be
+        /// a resolved native function pointer.
+        ///
+        /// For example, given the following P/Invoke signature:
+        /// <code>
+        /// [DllImport("Kernel32.dll", ExactSpelling = true)]
+        /// [MarshalDirectiveAttribute(
+        ///         Type = MarshalDirectiveType.UserDefinedStub, 
+        ///         UserFunctionType = typeof(Stubs),
+        ///         UserDefinedStub = nameof(QueryPerformanceCounterStub))]
+        /// extern static unsafe bool QueryPerformanceCounter(long* lpPerformanceCount);
+        /// </code>
+        ///
+        /// The supplied user defined stub should be:
+        /// <code>
+        /// static unsafe bool QueryPerformanceCounterStub(
+        ///     IntPtr nativeFunctionPtr,
+        ///     long* lpPerformanceCount) { ... }
+        /// </code>
+        ///
+        /// During the call to the stub, the supplied <see cref="System.IntPtr"/>
+        /// would be a native function pointer.
+        /// </remarks>
+        public string UserDefinedStub { get; set; }
+    }
+}
+```
+
+## Questions
+
+* Can the above API be used to provide a reverse P/Invoke stub?
+* Does the the Marshal Directive have any interaction with the [`SuppressGCTransitionAttribute`](https://github.com/dotnet/runtime/issues/30741)?
+* Should the `MarshalDirectiveAttribute` be considered in potential support for the [typing of function pointers](https://github.com/dotnet/roslyn/issues/39865#issuecomment-600884703)?
+
+## References
+
+[P/Invoke][pinvoke_link]
+
+[Type Marshaling][typemarshal_link]
+
+[IL Stubs description][il_stub_link]
+
+<!-- Common links -->
+[dotnet_link]: https://docs.microsoft.com/dotnet/core/tools/dotnet
+[typemarshal_link]: https://docs.microsoft.com/dotnet/standard/native-interop/type-marshaling
+[pinvoke_link]: https://docs.microsoft.com/dotnet/standard/native-interop/pinvoke
+[comwrappers_link]: https://github.com/dotnet/runtime/issues/1845
+[il_stub_link]: https://mattwarren.org/2019/09/26/Stubs-in-the-.NET-Runtime/
+[csharp_fptr_link]: https://github.com/dotnet/csharplang/blob/master/proposals/function-pointers.md