[Proposal] Functional Interfaces

[sakno]

This proposal is inspired by FI from Java but it's not just a copy. It's aimed to solve some problems with delegates and has different implementation.

Motivation

We have several ways to pass executable code as an argument to method:

1. Using delegates
2. Using function/method pointer as proposed in Function Pointer Syntax #2917
3. Using generic parameter constrained with interface type

The last approach typically used to avoid on-heap allocation if variables are captured in read-only manner. It may look like as following:

static void Sort<T, TComparer>(T[] array, TComparer comparer)
  where TComparer : struct, System.Collections.Generic.IComparer<T>;

This allows to declare custom value type implementing IComparer<T> interface and place additional context to the fields of such value type. Then I can instantiate value type on the stack with all necessary context passed through constructor and use it in delegate-like manner.

Of course, I can achieve the same goal with Comparer<T> delegate but it requires allocation on the heap plus allocation of captured context. Additionally, it's not possible to capture variables of ref-like value types.

With #1148, it will be possible to use this pattern with ref-like value types. However, the type itself should be still hand-written.

readonly struct MyComparer : IComparer<string>
{
  private readonly StringComparison _comparison;

  MyComparer(StringComparison comparison) => _comparison = comparison;

  public int Compare(string x, string y) => string.Compare(x, y, _comparison);
}

It's a little bit verbose. What if compiler will allow to convert lambda expression and its captured context to automatically generated type implementing required interface?

Goals

• Minimal impact on existing syntax
• Allow capturing variables of ref struct types
• Reduce on-heap allocation
• Compatibility with related proposals

Non-goals

This proposal is not about

• Replacement of delegates
• Special syntax for declaration of functional interfaces
• Java Interop

Proposal

The key idea is to allow lambda expression to be passed as parameter if it satisfied to one of the following conditions:

1. Target type is functional interface
2. Target type is generic type that is constrained by functional interface. Additional constraints such as struct, ref struct, class are also allowed.

The semantics of FI is the same as in Java: it may have multiple DIMs and single method without default implementation. For instance, IComparer<T> can be treated as functional interface.

In this case, calling of Sort method can be done without boilerplate code:

StringComparison comparison = StringComparison.Ordinal;
Sort(new [] {"Hello", "World"}, (x, y) => string.Compare(x, y, comparison));

C# compiler uses translation strategy depends on the target type of lambda expression:

1. Delegate type. Nothing special here, compiler creates display class for captured locals and then creates delegate instance. Capturing variables of ref-like structs is not allowed.
2. Functional interface. Compiler generates display class implementing functional interface. All captured locals placed as fields. Capturing variables of ref-like structs is not allowed.
3. Generic type constrained with functional interface. Compiler generates value type implementing functional interface. All captured locals placed as fields. Mutability of locals is handled in the same way as for delegates. Capturing variables of ref-like structs is not allowed.
4. Generic type constrained with struct and functional interface. The same as previous one.
5. Generic type constrained with ref struct and functional interface. Compiler generates ref-like value type implementing interface (which is allowed by IL). Capturing variables of ref-like structs is allowed. Mutability of locals is allowed only if target parameter is by-ref.

The last use case requires special procedure for handling mutability of captured variables. However, it's easier than it might seem at first glance. We know that value of ref-like struct exists on the stack only, so compiler just emits the code that rewrites the mutated local variables.

For instance, the following code

static void Sort<T, TComparer>(T[] array, ref TComparer comparer)
  where TComparer : ref struct, System.Collections.Generic.IComparer<T>;

int x = 0;
Sort(new []{"Hello", "World"}, (x, y) => { x = 10; return string.Compare(x, y, StringComparison.Ordinal); });

can be translated to

[CompilerGenerated]
internal ref struct LambdaExpression$1 : IComparer<string>
{
  internal int _captured_x;

  public void Compare(string x, string y)
  {
    _captured_x = 10;
    return string.Compare(x, y, StringComparison.Ordinal);
  }
}

int x = 0;
LambdaExpression$1 closure = new LambdaExpression$1 { _captured_x = x }
Sort(new []{"Hello", "World"}, ref closure);
x = closure._captured_x;

Resolving issues with rvalue

In the current version of C# compiler, lambda expression can be used in rvalue position only if target type is delegate type. This proposal extends context where lambda expression is allowed so it's necessary to cover translation strategy for such cases.

Target type is functional interface:

IComparer<T> comparer = (x, y) => string.Compare(x, y, StringComparison.Ordinal);

Should be translated to compiler-generated class.

Target type is var. In this case compiler should allow to apply new operator to interface type:

var comparer = new IComparer<T>((x, y) => string.Compare(x, y, StringComparison.Ordinal);
//or just simply
var comparer = new IComparer<T>(string.Compare);

Should be translated to compiler-generated value type. Very similar to anonymous type instantiation using new { } syntax that already exists in C#.

The last use case with ref-like value type is under consideration:

ref var comparer = (x, y) => string.Compare(x, y, StringComparison.Ordinal);
//or
var comparer =  new ref IComparer<T>((x, y) => string.Compare(x, y, StringComparison.Ordinal);

Should be translated to compiler-generated ref-like value type. This is the only place when new syntax should be introduced. However, it can be solved with #2975.

IMO, this proposal might replace #302 which is rejected now.

[CyrusNajmabadi]

It appears that Shapes subsumes this: #164

[quinmars]

Related: #2517

[sakno]

@CyrusNajmabadi , AFAIK shapes don't explain the translation of lambda expression and closures. Did I miss something?

[Sergio0694]

It appears that Shapes subsumes this: #164

Hey @CyrusNajmabadi - I'm not sure I understand your comment here, could you elaborate on that?
From what I can see, the shapes feature would be used in cases of static extensions to types, while the main point of this proposal would be to avoid allocations in cases where you need to capture a state. That is, it'll allow the compiler to have stack-only display types for closures being created as ref structs, and passed by reference to the calle.

For instance, consider how an API like string.Create<TState> could change. Right now it's like this:

// Just an example, I know there is string(char, int):
char c = '$';

// Super awkward syntax and function signature to enable delegate caching
string text = string.Create(10, c, (span, state) => span.Fill(state));

Whereas with this proposal you could have the following:

// API definition
string Create<TFunc>(int length, ref TFunc func)
    where TFunc : struct, ISpanAction<char>;

// Usage:
char c = '$';

string text = string.Create(10, ref span => span.Fill(c));

That ref in the lambda expression could be used to tell the compiler to generate a ref delegate.
Behind the scenes, it would just do this:

// Display type
struct <Test2>DisplayStruct_1 : ISpanAction<char>
{
    public char c;

    public void Invoke(Span<char> span)
    {
        span.Fill(c);
    }
}

// And transform the code as:
char c = '$';

<Main>DisplayStruct_1 state = default;
state.c = c;
string text = string.Create(10, ref state);

This would make both the API surface and the actual usage much less verbose and easy to use.
The only workaround today is to either just pay the allocation/overhead for a full blown display class every time, or just force users to create the necessary closure value type manually, which is pretty annoying and time consuming.

I feel like this would enable devs to write more efficient code more easily, while also specifying intent (indicating that the input delegate will be immediately consumed), and make the existing struct + interface "value delegate" feature more accessible 😄

[ZacharyPatten]

This issue is related to #2904 (and probably has some overlap)

In #2904 I was proposing new explicit syntax that would result in compilation to struct+interface. This issue is slightly different but I think they may share the same end goal to some degree.

[Sergio0694]

@ZacharyPatten From what I see, both proposals would require some changes to the C# language too. This one in particular would likely require allowing ref in a lambda expression to indicate the intent to use this feature. The C# compiler would then create the necessary value type behind the scenes, matching the target interface constraint. This would essentially just be a small expansion over the standard lambda expression syntax, the rest would be the same.

I think there's some overlap in the two issues, yes. At the very least, the intent seems to be very similar - they're both proposing ways to improve the usage of constrained value type delegates and make them more accessible and widespread 😊

[HaloFour]

[CompilerGenerated]
internal ref struct LambdaExpression$1 : IComparer<string>

ref structs are not permitted to implement interfaces. Yes, IL allows it, but it creates a very dangerous situation in which it would be possible to cast to struct to the interface which would result in boxing on the heap.

[Sergio0694]

Yup, Michal had a good example of why this could cause issues in cases where default interface methods are present.
See his original comment in #1148 here.

Personally, especially for an initial version of this feature, I'd be 100% fine with not having the generate closure value type be a ref struct. It would require no language changes there, and the only limitation would be that it wouldn't be possible to capture ref struct types - but after all that's the same limitation we have with lambda expressions today, so it would be fine I think 🤔

[sakno]

@HaloFour , casting is not possible in C# or any other .NET language. Compiler can emit interface impl for ref struct. Casting to interface is the same as casting to object data type from which any ref struct derived implicitly. But I agree that the default interface method can cause boxing.

[HaloFour]

@sakno

Casting to interface is the same as casting to object data type from which any ref struct derived implicitly.

And equally forbidden. Casting a struct to object results in boxing which results in heap allocation.

[sakno]

@HaloFour , I understand that. It is forbidden by compiler as well as casting ref struct to interface. As I said, it's a problem only if interface has default method. So I agree with you and @Sergio0694 , at the moment it's not possible to support capturing of ref struct variables. However, it can be possible in future with #2975 proposal which is mentioned in my first post.

[timcassell]

It looks to me like this has all the same benefits as #1060, but is much simpler and more straight forward. Are there any differences besides declaration syntax that I'm missing?

[sakno]

@timcassell , we can exclude capturing ref structs for simplicity. However, I don't see any issues with that if ref struct constraint on generic type will be implemented (I saw that appropriate proposal exists already). From other side, ref struct can implement interface if it doesn't have default implementation (that could be checked by compiler easily). From #1060 personally I don't like many aspects:

• Magic ValueFunc<int, bool>.Reference<TPredImpl> inner type
• Presence of ValueFunc counterparts in BCL in addition to well-known delegate types such as Func and Action
• Magic unsafe constrained call in IL that cannot be expressed in C#
• Verbosity
• Special syntax to express struct lambda (while the current proposal uses type inference without introducing new contexts for existing keywords)

[timcassell]

5. Generic type constrained with ref struct and functional interface. Compiler generates ref-like value type implementing interface (which is allowed by IL). Capturing variables of ref-like structs is allowed. Mutability of locals is allowed only if target parameter is by-ref.

I think since C# supports ref fields in ref structs now, mutability of locals can be allowed without requiring the parameter to be by-ref. The display struct would just be filled with refs to the locals, instead of the actual values.

[jaredpar]

I think since C# supports ref fields in ref structs now, mutability of locals can be allowed without requiring the parameter to be by-ref. The display struct would just be filled with refs to the locals, instead of the actual values.

There are some cases that can be solved by this but it's a case where the details get complicated. For example you can't capture a ref struct as a ref field yet so it can't solve mutability for that. The lifetime rules are also a problem that needs to get worked out, particularly if you start silently capturing locals as ref. Probably end up in a place where you'd want such parameters marked as scoped.

[jaredpar]

Overall like the motivations and general outline of the proposal here. Think the challenge with this proposal, and really any proposal that moves capture into struct, is mutations. Effectively how do make the fact a struct capture was used transparent to the customer? Keep in mind it's not just about making sure that mutations done within the lambda are written back, it's also about making sure that mutations done in other contexts are observed by the lambda. Consider for example the following:

int local = 42;
M(
  () => local = 13,
  () => Console.WriteLine(local));

void M(Action a1, Action a2)
{
  a1();
  a2();
}

This needs to print "13". Even though the second lambda in this case doesn't mutate local it still must participate in some manner of ref capture in order to observe the mutation of other lambdas. This means the compiler can't do tricks like only use struct capture if there are no mutations. It can only use struct capture if there are no mutations in pretty much every code path.

This behavior pattern in the above sample needs to be true even if you expand the set of types a lambda can target to include functional interfaces. I do not think that LDM would accept that whether or not mutations are observable is dependent upon whether the target type is a delegate or functional interface. Instead I think LDM would maintain the position that lambdas allow and observe mutations.

There are a few mitigations around this:

1. A lambda which has the static modifier by definition does not participate in mutations. Those closures could be safely moved to a struct. A decent portion of these are already amortized into delegate caches by the compiler though.
2. As the proposal mentions, functional interface parameters could be declared as ref. That would let the struct capture be passed by ref so that mutations could be observed. This has a few caveats though:
   1. It means that API authors need to buy into the idea of defining functional interface parameters as ref. That goes against current guidance. It also creates friction for code that doesn't use lambdas as they now need to use ref to pass around values.
   2. When the method has other ref or ref struct parameters then ref-safety comes into play and that could get a bit messy at the call site. Yes we could work the rules out here and make sure it's all safe. I'm a bit concerned about how customers would be able to interpret / understand the error messages though.
3. As the proposal mentions, functional interface parameters could be type parameters with the allows ref struct anti-constraint. This solves some of the problems posed by having them as ref parameters but creates new ones like inability to mutate or observe mutations of ref struct locals.

These are very real challenges of integrating struct closures into existing lambda syntax that don't have good answers (at least that I can think of). What my mind keeps coming back to is that for a proposal like this to be successful it would likely require a change to lambda syntax that indicated it used value capture and have the compiler consider struct based capture for those cases.