Proposal: Virtual extension methods

[gafter]

Add support for virtual extension methods - methods in interfaces with concrete implementations. A class that implements such an interface is required to have a single most specific implementation for the interface method inherited from its base classes or interfaces. Virtual extension methods enable an API author to add methods to an interface in future versions without breaking source or binary compatibility with existing implementations of that interface.

These are similar to Java's "Default Methods".

This feature requires corresponding support in the CLI/CLR. Programs that take advantage of this feature cannot run on earlier versions of the platform.

[thomaslevesque]

I really liked the idea of default methods when this feature was announced for Java 8. In some ways, it solves the same problem as extension methods, but it has one advantage and one drawback:

• Pro: the method can be overriden by derived classes to provide a more specific implementation
• Con: you can't extend interfaces you don't own

If C# could have default methods in addition to extension methods, it would be quite awesome. The first use case that comes to mind is a Sort method for IList<T>.

[theraot]

Extension method are pretty much that, in the sense that the compiler will use an actual method before looking for extension methods. So you can have that Sort method for IList<T> already.

---

On the other hand, these "virtual extension method" as described by @gafter are meant to serve as default implementations of the interface methods.

---

I'll call them vem for the purpose of this post, ok? ok...

Consider if you want to be able to declare an interface with a vem for some or all of it's members, and you want to be able to share these with other assemblies. Ok, that means that those implementations got to be stored in the assembly somewhere...

---

Edit: the following was wrote under the assumption that these vems could be used to implement interface members. That is not necessarily the case.

---

Since part the intention is that you will be able to use an assembly compiled against a version of an interface that had less methods than the current one, and also have the current one fill-in the missing methods... a more clever virtual resolution will be required, since the objects exported by the assembly don't have the methods you want - nor are aware of their existence.

In that situation we know that the missing methods will only be called from new code (since old code is not aware of them anyway). So it is sensible to to implement these vems in similar ways to extension methods. maybe using another attribute that the compiler knows about, so the compiler should be able to find them to generate the calls - these calls are resolved at compile time.

---

Now... you would also want to define these vems for interfaces that you did not declare (for example IList<T>), so we will need a syntax to specify these without creating a new interface... (allowing multiple vems from multiple authors per interface) and then how does the compiler decide which one to use if there is a collision?

Consider the case where you have multiple interface inheritance and various vems causing collisions, in that case the call must be marked ambiguous, it must be possible to call the vem directly (as an static method, the same way it happens with current extension methods) in those cases.

Edit: the decision of whatever to use the vem is done in runtime [except if it is called statically], but deciding which one to use is done at compile time.

---

If it is allowed to declare these vems without declaring the interface at the same time (I mean, if it is allowed to declare these vems for an interface declared somewhere else), that means that:

• Similar syntax to declaring a extension method.
• It is conceivable to have vems without interfaces.

Note: calling vems must be transparent, no new syntax should be necessary to call them.

[thomaslevesque]

Extension method are pretty much that, in the sense that the compiler will use an actual method before looking for extension methods. So you can have that Sort method for IList already.

No, you can't; extension methods are resolved at compile time, not at runtime. So if you have a variable of type IList<T>, and you call a Sort extension method on it, it won't matter if the concrete type actually has a Sort instance method; it will never be called, since the compiler will already have transformed the call to a static method call.

Now, if the variable is known at compile time to be List<T>, it's a different matter; the compiler will indeed pick the instance method.

[theraot]

Ok... so the code has to be able to decide if the IList<T> is actually a List<T> (or other implementation), and do so at runtime because there is no way to know all the implementations beforehand, and still have intellisense...

So... the compiler would generate a method that will query the type, decide to use either the specific method if available or the default implementation, then the calls are resolved to this method. Then at runtime it is decided whatever or not to use the default implementation, when writing code you would have intellisense because the compiler knows the method will be available.

Note: That hopefully could be optimized for known implementations.

This needs memorization, so the check is done only once per type per AppDomain. There will be a small penalty on each call due to the indirection, and a slightly bigger on the first call due to the check. Then it would be a matter of adding syntactic sugar to it.

What implementation to fallback to, can be decided at compile time, but whatever it is used or not, is decided at runtime. That implementation can be chosen beforehand because we know that if any new method is added in the dependency assembly it is not used in old code. The code of the default implementation must be referenced (cannot be copied to the current assembly) because they may change from a version to another of the dependency assembly.

It could be done with a similar syntax to current extension methods (perhaps using the virtual keyword) and have the compiler rewrite the code to add the runtime checks. Although that approach will not solve the problem of implementing interface methods, API authors could add these without the need to add new methods to the interface. I believe that would solve both problems for most cases.

---

Note: The virtual extension methods are to be chosen at compile time. If it is possible to override a virtual extension method with another virtual extension method, then those local to the assembly (or may be by namespace too) could override those imported from other assemblies.

It most also be noted that overriding a virtual extension method with an specific implementation is done without keyword, because it is intended to keep source compatibility.

It may be weird to use the keyword virtual if you don't use the keyword override, maybe we can call them dynamic extension methods, or something else?

[gafter]

@theraot What you're describing does not make sense to me. In this proposal the VM would use a vtable to cache the method to be invoked, just as it does today for interfaces. The CLR's method resolution mechanism for these methods would be enhanced to look through interface implementations if there is no implementation in the class hierarchy of the instance type. This is exactly what Java does today.

[theraot]

@gafter hmmm... seems to be you are trying to solve something different. What I gather is that you start from that these are members of the interface. I did start from that these are extension methods, having them to implement interface members being a secondary thing.

If you start by thinking about extension methods, and go on to make them "virtual" as to allow what @thomaslevesque proposes of extending an already existing interface (IList<T>) to provide a genal and default implementation of a method (Sort) [up to there extension methods will do], but have it use the specific implementation based on type of the object at runtime [that extension methods don't do], then you will end up with a different solution.

And, in reality if the API author wants to add new methods to existing interfaces and not break binary nor source, they can add extension methods. The only difference is that we want to use the specific implementation if available.

I suspect it would not be much different implementation, although I was imagining it more like dynamic...

Is the compiler able to pre-generate these vtable caches for known implementations at compile time? If no... then that's a suggestion.

[HaloFour]

@gafter I've love to hear comments/concerns regarding how C# might mitigate multiple inheritance concerns that arise from this.

For example:

public interface IFoo {
    void M() { }
}

public interface IBar {
    void M() { }
}

public class Baz : IFoo, IBar { }

Baz baz = new Baz();
baz.M();

In Java this is illegal and Baz is required to implement M(). But of course M() could be added to the interface after the fact in which case Baz wouldn't supply an implementation.

[gafter]

@HaloFour I suspect the Java runtime method resolution algorithm would make sense for CLR/C# as well.

[fschmied]

In order to preserve binary compatibility when new members with a default implementation are added to existing interfaces, virtual extension methods do need CLR support.

However, if CLR support were not available, virtual extension methods could also be implemented as a language-only syntactic sugar feature, see #73. This could be seen as a "version 1" variant of the feature (with CLR support coming later, for example) that only provides source code compatibility, but not binary compatibility.

Also, virtual extension methods are not only about adding new members to existing interfaces; they are also about providing default implementations for interface members commonly (but not always) implemented in a certain way. (E.g., IEnumerable<T> providing a default implementation for the non-generic IEnumerable members which just delegate to the generic ones.)

[HaloFour]

The problem with that implementation is that it doesn't provide the biggest benefit to actual default implementations, that is being able to amend an existing interface with new members without breaking any existing code. If it was handled as syntax candy entirely by the compiler then any code implementing that interface would require to be recompiled.

I'd rather see this implemented "properly" with CLR support, assuming that the CLR could be amended to support it within the C# 7.0 time frame. If that's not the case then maybe consider the alternatives, but even then my preference might just be to hold off on the language feature until the CLR supports it.

[fschmied]

@HaloFour I agree that CLR support would be best, of course. However, I disagree that without the binary compatibility (which can only be provided by the CLR), the biggest benefit is lost.

First, to me as a framework auther and a framework user, the biggest benefit is that of having overridable default implementations for interface members that are most often, but not always, implemented the same way. This would

• allow me to "mix in" overridable behavior into my class simply by implementing an interface containing virtual extension methods,
• allow me to provide overridable default implementations for convenience methods (see e.g., LINQ operators - wouldn't it be nice if an implementation of IEnumerable<T> could override Count and Any to have better performance, for example?), and
• allow me to provide default implementations for methods inherited from base interfaces (see e.g., IEnumerable<T>, where the base IEnumerable.GetEnumerator() method is always implemented by delegating to the generic one).

Second, as a framework or library author, the feature of being able to add members to existing interfaces without breaking source code compatibility would still be useful even if recompilation of implementers was required. While binary backward compatibility is really important to people implementing the .NET BCL, it is not as important to people implementing other libraries. A breaking change just requiring recompilation is still much better than a breaking change requiring modification of all implementers.

Therefore, in my opinion, the feature would be better with, but still very valuable even without CLR support. And thus, I would not want it to be postponed just because the CLR can't be changed by the time C# 7 is released.

[vladd]

Dear all, isn't this functionality just a simpler version of mixins? If we are considering having mixins in the language, we don't need to just copy a feature from other language.
The motivation from Java tutorial doesn't seem to be very convincing for me: changing the interface which is implemented by other's classes is anyway a breaking change, and having the other's code silently compiling without forcing the developers to review the needed adjustments is potentially dangerous.
Besides that, confirming to an interface and sharing the implementation should be potentially orthogonal things, right? With mixins we would add a possibility to opt in for code sharing while requiring the interface implementation. With the proposed feature one needs to opt out of the code sharing, which is again error-prone.

[gafter]

@vladd The problematic issue with multiple inheritance is state. But unlike mixins, this proposal does not allow the addition of state to interfaces. The default method implementation provided in an interface would be a correct (if inefficient) one based on other methods of the interface. This is like the existing extension method feature but these methods are virtual and can be overridden in subclasses for efficiency.

[vladd]

@gafter
My main point is that extending an existing and published interface is indeed a breaking change, so making it undercover (so that the developers won't have to adjust their code) is not what I would endorse.

Having said that, I am not proposing adding state to interfaces. I think the needed functionality could be provided with mixins by the code like this (speculative syntax):

(old interface)

interface I { TimeSpan ExpectedDuration { get; } }

(new interface)

interface I
{
    TimeSpan ExpectedDuration { get; }
    TimeSpan ActualDuration { get; }
}

mixin ActualDurationImpl(T) where T : I
{
    public TimeSpan ActualDuration { get { return this.ExpectedDuration; } }
}

so that the implementers of the I interface would have an easy way to conform to the updated interface, but not get the new unreviewed implementation implicitly!

[gafter]

@vladd In what sense is adding a virtual extension method a breaking change? The whole point of this feature is that adding them is not a breaking change.

[vladd]

@gafter Well, what if a class has already a method named like the one being added (but has different semantics)? What if the added method just doesn't fit into the semantics of the class implementing the original interface, so now it would be impossible to implement the updated interface? Say, if we had interface IDrink { void Consume(); }, class Poison : IDrink { ... }, and now adding bool DoesTasteGood() to IDrink?
This all must be a breaking semantical change, if even it's technically possible to make the code compilable.

[HaloFour]

@vladd

Well, what if a class has already a method named like the one being added (but has different semantics)?

If the class already had the method and that class was in a different assembly that was not recompiled after the interface was changed then nothing would change. The method on the class would effectively shadow the interface method and callers to that method would call the class method directly. Any new code written to call the interface method would call that method directly since the class method does not implement it.

If the assembly containing the class is then recompiled the compiler would consider the class method to implement the interface method and going forward calls to either would call the class method.

What if the added method just doesn't fit into the semantics of the class implementing the original interface, so now it would be impossible to implement the updated interface?

C# has implicit implementation, this is not really a problem like it is in Java.

As for the semantics argument, you could pretty much invent scenarios to invalidate every feature proposed by claiming that the semantics can't be guaranteed. I recall a pro-Java article written shortly after C# was first released claiming that delegates were stupid because of this very reason. Apparently a FireDelegate delegate was less descriptive than an IFireable interface and you could accidentally launch a missile or something silly.

• To note, I hate this "virtual extension method" name. The "extension" moniker simply doesn't fit. I know that we probably don't want to use Java's naming convention of "default methods" because, ya know, ew Java, but we shouldn't adopt their habit of giving things wacky names to avoid the obvious name that someone else might've already used.

[vladd]

@HaloFour

If the assembly containing the class is then recompiled the compiler would consider the class method to implement the interface method and going forward calls to either would call the class method.

Exactly, and that's my point. This is dangerous. Your class now overrides the default method implementation with some arbitrary code, without the conscious decision of the developer, and without any warnings on recompilation.

[vladd]

@HaloFour

As for the semantics argument, you could pretty much invent scenarios to invalidate every feature proposed by claiming that the semantics can't be guaranteed.

No, my point was that adding a new interface method is a breaking semantical change, and as such ought to be a breaking syntactical change as well. That is, every class implementing the updated interface must get some developer's attention after the change.

[thomaslevesque]

Well, what if a class has already a method named like the one being added (but has different semantics)?

This problem is easily addressed : make it mandatory to explicitly override the interface method (same as when you override a method from an abstract class). If you don't, then the method already in the class hides the one from the interface (as if you had added the new modifier), and it won't be called through the interface.

[vladd]

@thomaslevesque ... unless the existing method already overrides some other interface's method?

[jveselka]

What if these virtual extension methods were explicitly implemented interface methods. They could still be overriden in implementations (but again only as explicit interface imlementation) and it would avoid problems @vladd pointed out.

[gafter]

See #73 for a detailed description.

[orthoxerox]

I wonder if the name for this concept was chosen well. To me a virtual extension method is literally an extension method that is virtual, i.e., something to solve the expression problem with, e.g.:

public class Base
{
    public virtual int FooVirt() => 1;

    public int FooStat() => -1;
}

public class Derived : Base
{
    public override int FooVirt() => 2;

    public new int FooStat() => -2;
}

public static class Ext
{
    public static virtual int BarVirt(this Base @this) => 3;

    public static virtual int BarVirt(this Derived @this) => 4;

    public static int BarStat(this Base @this) => -3;

    public static int BarStat(this Derived @this) => -4;

    public static void Test()
    {
        Base b = new Derived();
        Console.WriteLine(b.FooVirt()); //2
        Console.WriteLine(b.FooStat()); //-1
        Console.WriteLine(b.BarVirt()); //4
        Console.WriteLine(b.BarStat()); //-3
    }
}

[alrz]

@gafter If we ever have an Option<T> type, couldn't it be something like this?

default Option<T> TryGetNext() {
  return MoveNext() ? Some(Current) : None();
}

Also with this we can define a non-generic default method on top of generic interfaces, like

interface IEnumerable<out T> : IEnumerable {
  default IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();
  IEnumerator<T> GetEnumerator();
}

interface IEnumerator<out T> : IDisposable, IEnumerator {
  default object IEnumerator.Current => (object) this.Current;
  default Option<T> Next() => MoveNext() ? Some(this.Current) : None();
  // ... 
}

interface IAsyncEnumerator<T> : IDisposable {
    Task<Option<T>> NextAsync();
}

[MouseProducedGames]

This seems less a default, and more of a fallback. Ie., something to use if the class doesn't implement it.

Perhaps a name something like "fallback interface method" might work better. Except for "fall" looking like "fail".

...Maybe a different name.

[timgoodman]

Is there any chance we could get a syntactic sugar for implementation-through-composition, like Jon Skeet suggests in his answer here?
http://stackoverflow.com/questions/255553/is-it-possible-to-implement-mixins-in-c

That would allow you to support default methods (putting the default implementation in its own class, e.g. DefaultIEnumerator) which the consumer of the interface opts in to by adding a single line of code to their class. It also lets the consumer of the interface choose a different implementation as their default. Plus it's useful for doing composition on your own classes. And it's really just a syntactic sugar, no need for CLR changes.

public class Foo : ISomeInterface
{
    private DefaultISomeInterface impl implements ISomeInterface; // new keyword: implements

    public void OneMethodOfISomeInterface()
    {
        // Specialize just this method
    }
}

[gafter]

This proposal is now tracked at dotnet/csharplang#52