Proposal: Open-source WinMDExp

[wjk]

Proposal: Open-source WinMDExp tool

Summary

Does what the title says, really.

Rationale

This tool is vital for compiling .NET libraries targeting Windows Runtime, yet it is still proprietary. I have looked at this library using ILSpy to see if I could recreate it, but it is complex and hard for me to parse.

Scope

Capability	Priority
Open-source WinMDExp (either here, or in another repo)	Must
Make WinMDExp compatible with portable PDBs	Must
Eventually port it to .NET Core	Should
Make it run on non-Windows platforms, perhaps as part of Roslyn	Could
Make WinMDExp compatible with embedded PDBs	Could

Notes

This tool doesn't have to go in this repo (in fact, I don't know with certainty where it should go), but I thought this would be the best place to put this proposal. dotnet/roslyn#23456 (comment) indicated that WinMDExp is not owned by Roslyn.

The portable PDB compatibility is listed as "Must" per dotnet/sdk#3042 (comment).

[ranjeshj]

This tool is not part of WinUI. @jesbis do you know where this proposal needs to be routed ?

[ranjeshj]

@wjk This is not owned by the WinUI team, could you please file a feedback request though the feedback hub app and share a link here? We can try to route that to the appropriate team.

[wjk]

@ranjeshj https://aka.ms/AA75kce

[ranjeshj]

@wjk Thanks. I'll forward it to the appropriate folks.

[weltkante]

Is there a way to track this for interested parties? The feedback hub request seems to be not accessible.

[ranjeshj]

@weltkante I agree that feedback hub does not help with back and forth conversations. We can use continue to use this issue for that.

The WinMDExp tool is used to allow users to define a WinRT type in a .NET assembly to create a WinMD for consumption by other WinRT based systems (e.g. C++). Can you provide more details on what are you trying to achieve ?

@AaronRobinsonMSFT as FYI

[weltkante]

I'm often working with custom native/managed projections in interop for desktop applications, previously I've been working on COM TLB generation because the builtin tools (regasm) no longer meet our requirements. As far as I understand WinMD is the modern replacement, so for some of our use cases it may be useful to use the newer runtime instead of the old TLB/IDispatch based one.

At this time it is just a point of interest and I don't have any concrete plans, previously I never examined whether IInspectable could be of any use over IDispatch because writing and maintaining IDL text files and running them through a compiler instead of generating TLBs directly from our models was too much trouble to be worth looking at, but if there were an open source version of a WinMD exporter I'd certainly study it. Thats why I asked, its often hard to find out when such things are released.

[AaronRobinsonMSFT]

@weltkante This is something we have been trying to figure out the best way forward. For the upcoming work in .NET 5 to support WinUI et al we are working with the same team that owns CppWinRT to create a CsWinRT solution for consuming WinRT from .NET. This would make consuming WinMDs from .NET no longer needed because the interop assemblies would be pre-generated using this new CsWinRT tool. This represents the consumption side of WinRT from .NET. Now we get to the solution that WinMDExp is trying to solve.

I believe for our v1 approach we are going to recommend users follow the same path as C++ and start with MIDL 3.0. We can appreciate this isn't a great experience because it forces users to author interfaces in another language, this brings us to the v2 of our support. Right now a possible approach would be to create a new attribute to indicate which types to expose via a new iteration of the WinMDExp or similar tool that would generate a MIDL 3.0 file or WinMD for the user.

The WinMDExp tool has some limitations with respect to what it can export and is a lossy process similar to the TlbExp scenarios involving COM and as such in its current form we are reluctant to simply open-source it. All that said, we do recognize the need for a polished UX for exposing WinRT types from a .NET project and are looking at possible solutions.

/cc @jeffschwMSFT @jkoritzinsky

[weltkante]

Your suggestion is probably fine for general developers but won't be useful for advanced users who want to customize their WinMD generation.

The WinMDExp tool has some limitations with respect to what it can export and is a lossy process similar to the TlbExp scenarios involving COM and as such in its current form we are reluctant to simply open-source it.

I don't quite follow that argument, you already provide a version of the tool within VS don't you? So users of it will already run into any shortcomings it has. This was exactly the reason why we had to write a replacement for regasm/tlbexp, the tools simply have too many issues. If anything having the source of WinMDExp would make it easier to build a customized version working around those issues (and maybe even contribute back improvements).

Having a quick look at the IL of WinMDExp it looks like mostly being built around CCI which seems already open sourced so even if WinMDExp uses a customized version it probably shouldn't be much of a problem to open source it.

That said I can understand if you currently don't have the capacities or desire of maintaining the tool in public. Its not blocking for us, just an opportunity and I only want to follow the discussion and see where this leads. (Though if you want to know more about our use case and regasm/tlbexp issues I can elaborate.)

[AaronRobinsonMSFT]

Your suggestion is probably fine for general developers but won't be useful for advanced users who want to customize their WinMD generation.

Can you elaborate on this more. By "your suggestion" I am assume the one about authoring in MIDL 3.0 which provides much more customization than anything the WinMDExp tool can provide.

I don't quite follow that argument, you already provide a version of the tool within VS don't you?

Indeed we do, but that is intended for .NET Framework scenarios. It is also missing Portable PDB support - which is very sad. It will work for .NET Core, but that is possibly going to change in .NET 5 since we are reworking our entire projection system with the aforementioned CsWinRT tool in mind.

Having a quick look at the IL of WinMDExp it looks like mostly being built around CCI which seems already open sourced so even if WinMDExp uses a customized version it probably shouldn't be much of a problem to open source it.

There are two sides to this that must be considered. The technical and the business. Your technical reading is generally true, but ownership is difficult and proliferation of a WinMDExp tool that isn't provided by Windows but looks and feels similar is a large security liability. Making something open-source is expensive from a business perspective as well. It isn't simply moving the source to a new repo on github. It involves a fair amount of legal work and coordination with how Microsoft will or will not "support" the source/product going forward which can impact Microsoft's reputation.

Though if you want to know more about our use case and regasm/tlbexp issues I can elaborate.

We would very much like to understand your scenarios and see how we can help support them. Feel free to share here or if you like my email can be found in my github profile.

[weltkante]

(just calling it regasm here because thats what we used before moving to a custom solution, but "regasm /tlb" is probably similar to tlbexp)

Can you elaborate on this more. By "your suggestion" I am assume the one about authoring in MIDL 3.0 which provides much more customization than anything the WinMDExp tool can provide.

I meant both the IDL suggestion and following it up with an attributed model later (which I imagine would be similar to what regasm does where it looks at attributes to determine what to export).

While technically IDL would be a solution it means you have to write everything twice (once IDL and once in C#). We'd rather stay with the sole definition being the managed code, but any builtin tool to generate TLB/WinMD from managed assembly is probably not going to be customizable. The primary concern is that we have to make the entire interop stack part of the public API surface of the managed assemblies in order for classic tooling to export it, while in reality we only want to generate internal TLB/WinMD for marshaling/interop purposes with our own code and implement it within the managed assembly. Secondary concerns are finetuning of how signatures are mapped.

Our primary usecase of TLB/IDispatch is interop with unmanaged code we write ourselves. Having TLBs is important because otherwise you don't get proper COM apartment marshaling. The other important usecase is communication between 32 and 64 bit processes, I don't know if WinMD/IInspectable is cross-process/bitness marshalable, but if it is it would be a prime target to replace in our stack.

The major motivation for looking at WinMD is that we could make use of the much richer WinRT language projection, currently with the TLB based system we have to stick to primitive types and arrays.

To be honest, ad hoc I couldn't find any information about whether apartment handling or cross process (and cross bitness) marshaling is supported by WinRT, so maybe the whole idea to update from TLB/IDispatch to WinMD/IInspectable doesn't even make sense. Like said before, right now its mostly out of interest, need to do more research to see if it would lead anywhere.

details about our experience with regasm and where it failed

• VS integration for regasm was broken since forever when you were doing anything else than a solution with a single project. When it tried to unregister the previous version it needed to load the assembly, but dependencies may already have been recompiled making it impossible to unregister the previous version due to not being able to load it. This caused garbage leftovers in the registry leading to errors later on at runtime.

  I have no idea if WinMD files can/need to be registered so might not be a problem at all

• regasm doesn't support anycpu setups properly (i.e. registering a TLB both for 32 bit and 64 bit) - we need that to make use of COM support for marshaling between processes of different bitness. While you can work around it by running regasm repeatedly with different settings we did write our own registration logic to do it cleanly.

  I have no idea how WinRT handles cross-process / cross-bitness marshaling (i.e. if it is possible at all, needs registration, etc.)

• regasm forces you to make everything public, even though the .NET framework can easily support COM objects implemented internally. Using a customized TLB generation we can use our own rules for what to export, the C# code is setup to implement interfaces from an external TLB via ComImported interfaces, but the TLB itself is generated from this very .NET assembly. It works because the .NET compiler doesn't need to resolve ComImports, meaning we only have to maintain the interface declarations in one place while still being able to use TLB based marshaling. Also not having to expose the whole interop infrastructure as public API from the managed assemblies is great.

• sometimes we want to fine-tune how a method signature is mapped in the TLB vs. how the C# signature looks, while still keeping a single definition

The last two points can easily be achieved with custom IDL, but we don't want to maintain two distinct declarations of every interface. Writing a custom IDL generator would probably the way to go to solve this, but this is basically on the same level of complexity as writing a custom WinMD generator in the first place, so its definitely not something I'd attempt without having a reference implementation to start with.

For what its worth the only reason I even attempted custom TLB generation is because the coreclr source contained most of the relevant logic. It has been removed from coreclr long ago because it was never used and just been a relic from older source, but it served well as a reference for building the custom TLB generator.

[stevenbrix]

I'm often working with custom native/managed projections in interop for desktop applications,

@weltkante, so are you saying that you have native desktop applications that consume managed components?

[weltkante]

Yes, we have applications which are mixing managed/native code and are communicating through COM interfaces/objects. That relationship goes both ways on all levels (we have native hosting managed, but also managed hosting native)