Supporting "Win32" APIs not included with Windows SDK

[kennykerr]

How do we generate a .winmd for Win32-style components that don't ship with the Windows SDK and thus won't be included in the windows.win32.winmd that this project produces? Here's an example of such an API:

https://www.nuget.org/packages/Microsoft.Web.WebView2

Hopefully most such components will provide WinRT APIs, but it sounds like Project Reunion may also eventually provide Win32-style APIs, so we may need a story for this.

Such components would typically also refer to common Win32 types so we'd want them to refer to windows.win32.winmd rather than duplicate those definitions.

[wravery]

I was able to get the specific example of Win32 WebView2 working with another partition under generation/scraper/Partitions, but that adds them to the Windows.Win32.winmd binary instead of a standalone binary. Some of the pwsh and C# tools depend on the current directory structure and include everything under the generation root, which makes splitting the outputs into separate winmd files much more difficult.

I was thinking it might work best to mirror the entire generation directory structure, e.g. in a folder called external. If it could also include some information about the NuGet packages, either in JSON or CSV, maybe even packages.config, then the generation could be entirely config-driven instead of needing to code any package names or versions directly into the pwsh scripts. The external build should also include things like the enum and type remaps from the generation directory so they reference consistent types from the base winmd.

Alternatively, there could be a different mode for all of the relevant tooling that lets us put the external partitions somewhere in the existing generation folder hierarchy, but I think that might end up more complicated than it's worth.

[wravery]

I think I found a C# compiler bug. 😄

[kennykerr]

Thanks for experimenting, Bill! This is related to microsoft/wdkmetadata#1 in the sense that the WDK request may well also necessitatea producing a seperate winmd that depends on Windows.Win32.winmd in much the same way as any other 3rd party components would.

[wravery]

I think a possible workaround for the compiler bug might be to redeclare VARIANT in an external namespace using an auto type.

[wravery]

To do this out of tree (i.e. in a standalone project), I think the winmd needs to carry more information. For example, all of the --remap type names passed to ClangSharpPInvokeGenerator in baseRemap.rsp need to be there to reliably map the types seen in another header to the correct types defined in Windows.Win32.winmd. I'm thinking of adding an attribute to Windows.Win32.Interop.dll to carry this information, e.g. [RemapFrom("tagVARIANT")]. The MetadataSyntaxTreeCleaner should be able to inject those attributes, but since the --remap parameter means something different in baseRemap.rsp and remap.rsp I may need to rename the parameter in remap.rsp to be able to pass baseRemap.rsp to ClangSharpSourceToWinmd along with remap.rsp.

The other thing I thought it would need is the namespace information, but it looks like types are still globally unique when the PInvoke/FFI for a projection is generated, so it should be possible to enumerate all the types and build a map of the type names to the namespace used in the metadata.

[wravery]

The standalone project would probably also resemble the complete pipeline of this project quite a bit, but hopefully as more of a turnkey solution.

[wravery]

I integrated #553 (and #549) with the SDK being developed in the sources/msbuild directory. It depends on that SDK, but if/when that PR is merged and the SDK is ready I think that addresses all of this issue.

[sotteson1]

We now have two samples in the code that show how to use the external scarper nuget package. It's up to external projects to use the scraper and publish their own .winmd files.

[riverar]

@sotteson1 None of them seem to work. Do you have a sample that demonstrates usage outside the source tree?

[sotteson1]

@sotteson1 None of them seem to work. Do you have a sample that demonstrates usage outside the source tree?

I updated win32metadata\sources\msbuild\GeneratorSample\GeneratorSample.proj to use the right version of what's in nuget.org. To build it, I did "dotnet build" from that directory and it worked. Let me know if it doesn't for you. There might be something in your configuration that is needed to build that I don't know about.