Support for Mono's DllImport(@"__Internal") ?

[tido64]

Can we add support for loading symbols within the hosting application as documented here?

As a Mono extension, if the library being loaded is __Internal, then the main program is searched for method symbols. This is equivalent to calling dlopen(3) with a filename of NULL. This allows you to P/Invoke methods that are within an application that is embedding Mono.

This will make it easier for existing applications to expose method symbols to managed code without having to 1) separating desired methods to a separate dynamic library or 2) passing tables of function pointers to managed code on initialization.

[jkotas]

cc @yizhang82

[gkhanna79]

@yizhang82 Please fix the milestone as applicable.

[jkoritzinsky]

@jkotas @AaronRobinsonMSFT I'm looking into working on this next. It fits in well with our goal of improving our hosting story for .NET Core 3.0 as well as improving compatibility with Mono. What do you think?

[jkotas]

The managed NativeLibrary APIs that we are adding for 3.0 allow people to resolve __Internal to whatever module they wish. It means we have a basic story for this in 3.0 already. You have to write a bit of code for it, but I think that's acceptable - you are not blocked.

A great hosting story is not a theme for 3.0. I would wait for doing anything more here until a great end-to-end hosting story is a theme.

[AaronRobinsonMSFT]

@jkotas I would argue the managed COM hosting theme I am working on is probably just that scenario. Being able to load a managed COM server and then have that COM server communicate with the native host seems very natural to me and something I have done. I think this is actually a valuable task for the following reasons:

1. There is some demand (i.e. this issue).
2. A scenario does exist, perhaps simple/contrived, but it does exist and the fact there is an issue demonstrates a non-COM scenario is in the mind of the community.
3. Has no blocking issues which means it could be accomplished at this time and can be written in an abstract way meaning the implementation can change, but the community can be unblocked immediately.
4. Prior art/aligning with Mono scenarios.
5. It can easily be tested/validated.

I don't see a real strong reason not to start on this, unless there is something more pressing at the moment.

[jkotas]

I do not think __Internal is relevant for COM hosting. __Internal would be relevant for custom hosting/embedding in native apps. We are not investing much in this area for .NET Core 3.0. Internal user story https://devdiv.visualstudio.com/DevDiv/_queries/edit/650238/. It does not mean that this is unimportant, nor that we are doing absolute zero. It just means that it did not fit.

In any case, this would need more detailed design proposal first.

[jkoritzinsky]

Do we have any docs or guidance written down anywhere on writing up design proposals? I know we have some docs for the corefx API review process. Is it a similar process or something different?

I just want to make sure that when I eventually write up a design doc for something that I am doing it correctly.

[jkotas]

There is no prescribed format. Smaller proposals just get written as a few paragraphs in the tracking issue. Larger proposals get written as standalone docs, e.g. look under https://github.com/dotnet/coreclr/tree/master/Documentation/design-docs

I think this one would be likely on the smaller proposal side.

[AaronRobinsonMSFT]

I do not think __Internal is relevant for COM hosting. __Internal would be relevant for custom hosting/embedding in native apps.

@jkotas Sure it is. I have a native application that is using a managed COM server. My managed COM server relies on some native function in my native application. The managed COM server is merely a subset of the native hosting scenario. Am I missing something?

[jkotas]

Then you are mixing COM hosting with a custom hosting. It is not a pure COM hosting anymore.

[AaronRobinsonMSFT]

Then you are mixing COM hosting with a custom hosting. It is not a pure COM hosting anymore.

@jkotas I don't understand that statement. If I have a native EXE that does a CoCreateInstance() of a managed COM server, then CoreCLR will be instantiated via the upcoming *.comhost.dll. This is pure COM hosting, nothing custom about it. The managed COM server may then rely on some sub-system within the native EXE that did the CoCreateInstance(), perhaps some customized localization or resource management that the native EXE provides to the managed COM server. The only way to handle this would be to use the __Internal. I don't see how custom hosting of the CoreCLR impacts this scenario.

[jkotas]

The host and component should communicate via COM interfaces only in the pure COM hosting. Once there are side channels, it is not a pure COM hosting anymore.

The only way to handle this would be to use the __Internal

The server and host can exchange COM interface to call back and forth. It was the way to do this for last 20 years, no __Internal needed.

[CoffeeFlux]

Is there any chance of revisiting this in light of .NET 5? Embedding is a much more common scenario for Mono, and from our perspective this is important functionality for mobile.

[jkotas]

@CoffeeFlux Are you talking about embedding in general; or specifically about __Internal ?

[CoffeeFlux]

__Internal specifically. For context, I'm refactoring parts of the Mono loader alongside implementing NativeLibrary and realized that we're currently enabling __Internal on all builds, despite Core not having it. I'm highly loathe to remove it since you'd be essentially mandating a more complex NativeLibrary setup for a lot of scenarios, notably iOS, but it means we have a runtime discrepancy I'd prefer to resolve.

[jkotas]

If it helps, I think it is fine for __Internal to be recognized in Mono for statically linked embedding cases, like iOS. CoreCLR does not have equivalent of it and we do not plan to change that for .NET 5 currently. If we ever get to supporting statically linked embedding with CoreCLR, we can look into handling __Internal for that case too.

Much more fundamental difference between CoreCLR and Mono native library loading is that Mono loads everything as RTLD_GLOBAL by default (correct me if it is not the case). We are intentionally not doing that in CoreCLR. Do you plan to reconcile this difference?

[filmor]

I don't really get what's the issue with supporting __Internal.

The problem I'm facing right now is that I'm trying to make Python.NET work reasonably with .NET Core. However, in the usual "Python embeds .NET" use-case, the missing __Internal (or something equivalent) complicates things quite a bit. There is a quite common setup (e.g. Debian and Anaconda), in which the python executable is statically linked against libpython and there is an additional libpython next to it. However, if this libpython is now RTLD_LOCAL loaded alongside the python executable, all the pesky global variables (small int optimisation, interned strings) will point to different places, depending on whether I P/Invoke a Python API function or whether it's called from the main application, leading to a huge messup.

In Mono, this is not an issue as I can just use __Internal and the symbols are picked from the python executable in all cases. In .NET Core, I have (right now) as far as I can see to get the handle from dlopen(NULL) manually and then inject this into the DllImportResolver. It's not even possible to just define DllImport(null) or something like this as this is caught in the library.

[jkotas]

I have (right now) as far as I can see to get the handle from dlopen(NULL) manually and then inject this into the DllImportResolver

Yes, it is how DllImportResolver is meant to be used to customize DllImport lookups. I believe that a small amount of extra code should be acceptable to make more advanced interop scenarios like this one work.

[tannergooding]

The managed NativeLibrary APIs that we are adding for 3.0 allow people to resolve __Internal to whatever module they wish. It means we have a basic story for this in 3.0 already. You have to write a bit of code for it, but I think that's acceptable - you are not blocked.

@jkotas, for this it would essentially be doing the following, is that correct?

        static void Main(string[] args)
        {
            NativeLibrary.SetDllImportResolver(Assembly.GetEntryAssembly(), OnResolveDllImport);

            int result = AddIntegers(3, 5);
            Console.WriteLine(result);
        }

        public static IntPtr OnResolveDllImport(string libraryName, Assembly assembly, DllImportSearchPath? searchPath)
        {
            if (libraryName == "__Internal")
            {
                var module = typeof(ClassLibrary1.Class1).Module;
                return Marshal.GetHINSTANCE(module);
            }

            // Insert additional resolution logic as needed

            return IntPtr.Zero;
        }

        [DllImport("__Internal")]
        public static extern int AddIntegers(int lhs, int rhs);

Testing with C++/CLI it appears to all work and I wouldn't expect any difference if the entry points were bound another way (such as python or embedding the CLR in a native host).
This has an additional benefit that it isn't restricted to "__Internal" as the name, you can use any name and treat it as "internal".

[jkotas]

Yes, it is the pattern to use for this. For the @filmor 's python case, it would need to PInvoke dlopen with the right flags instead of Marshal.GetHINSTANCE (ideally just once and caching the result).

[filmor]

Sure, I can (and will) do this, it just seems complicated for something that has a well-defined and -established meaning on all platforms, as far as I know.

Now I have to write two loader variants, that also can't be pure .NET Standard (without a lot of reflection) and select them at runtime. What is particularly annoying is that I can't even piggy-back on the accompanied NativeLibrary.TryLoad as it doesn't handle the null case.

[jkotas]

something that has a well-defined and -established meaning on all platforms

What is it specifically? (just want to make sure that I understand your point)

[AaronRobinsonMSFT]

Just a small note, the NativeLibrary isn't in .NET Standard. It must be .Net Core 3.0+. Also even if we did the work in .NET Core, it wouldn't work on .NET Framework which adheres to .NET Standard.

I accept that this confusion is mostly our fault with naming. For clarity can we assume your scenario for a "standard" approach means sharing between .NET Core and Mono?

[filmor]

Yeah, sorry, I'm currently working on this so I'm jumping a bit around between approaches. For me it would be perfect if .NET Core handled DllImport("__Internal") like Mono does, i.e. using dlopen(NULL). This seems reasonably well-defined for me and is established by a few years of Mono legacy. If this was supported, I could stick to .NET Standard for the DLL that performs the actual assembly loading.

The rest of my comment was more about that the alternative (i.e. custom code for .NET Core) for this is still a bit more work than it should be since I can't do it in a platform-independent manner from within C# as the NativeLibrary class doesn't support this particular case.

[AaronRobinsonMSFT]

@jkoritzinsky @elinor-fung Lets think about what this means for the .NET 6 time frame. I think we are getting close to the critical mass of requests.

[jkoritzinsky]

We have an API proposal to make it much easier to enable this experience in #56331, which will be added in 7.0.