PoC of passing Python Lambdas as Java lambdas

[cmacdonald]

This is a first implementation of passing lambdas to Java.

It works based on several premises:

• we can detect Python lambdas arguments as callable()
• the target Java argument will be a functional interface.
• we can instantiate a proxy for that interface that maps the functional method to the Python lambda

There is one unit test provided, which demonstrates the following:

List<Integer> squares = numbers.stream().map(i -> i*i).collect(Collectors.toList());

in Python as:

squares = numbers.stream().map(lambda i : i * i).collect(Collectors.toList())

Areas for improvement:

• more unit tests are needed.
  • This is only one of many functional interfaces in the JDK, and with varying return types
  • we also need to test error conditions - e.g. a lambda that returns the wrong type, or wrong number of arguments
  • test function reference rather than lambdas
• Modifiers.isStatic() is not available within jnius_conversion.pxi ? I used a bitwise operation, which would be faster anyway.
• We need the python proxy object, created in populate_args() to last as long as the Java object. I tried to do this with the list holds, but thats not sufficient, probably because the actual invocation from Java to Python doesn't happen until later (same challenge as some proxy classes dissapear #511); holds should be removed.

Finally, I wanted to add: it only took a few hours to get this working - a true testament to the flexibility/extensibility of jnius!

[tshirtman]

  * test function reference rather than lambdas

I expect it to work exactly the same, with the additional benefit of avoiding issues with gc, as the user is more likely to hold on to a reference to the function than with a lambda.
In fact we should probably document the limitation that if their lambda/function is likely to be called later, they need to keep a reference to it, or to disable the gc (or to cross fingers very hard, but that won't work).

* Modifiers.isStatic() is not available within jnius_conversion.pxi ?  I used a bitwise operation, which would be faster anyway.

I'd need to check but it might be that it can't use the registered Object class that we use in reflect to be able to use base methods/attributes.

* We need the python proxy object, created in populate_args() to last as long as the Java object. I tried to do this with the list holds, but thats not sufficient, probably because the actual invocation from Java to Python doesn't happen until later (same challenge as #511); holds should be removed.

Right that'll be user responsability imho, as we can't know (afaik) if java still holds a reference to it, maybe we could introspect if the proxy object is still referenced by java, but that would need to be something we run regularly? maybe we could tie it to python's gc though. (https://docs.python.org/3/library/gc.html#gc.callbacks my idea would be to save the "holds" in a persistent list, and to run through them in the gc callback to remove them from the list if java doesn't reference their proxy anymore, i can give it a try later if you agree that's sensible).

Finally, I wanted to add: it only took a few hours to get this working - a true testament to the flexibility/extensibility of jnius!

happy to hear that :)

[cmacdonald]

Right that'll be user responsibility imho, as we can't know (afaik) if java still holds a reference to it, maybe we could introspect if the proxy object is still referenced by java, but that would need to be something we run regularly?

Yes, I had a think about this point. At the Java end, we would need a HashMap<Long,WeakReference>, along with a corresponding ReferenceQueue. This could be a static method in NativeInvocationHandler. Every instance of NativeInvocationHandler would be added to this map.

At some periodic time (perhaps initiated by Python GC), we would need to walk the values and see which items had been "enqueued" (i.e. Java GCd). We could then delete the corresponding Python object.

GC challenges aside or not, I don't think that #515 should delay a 1.3.0 release of Jnius.

[cmacdonald]

I'm beginning to think that we should merge this, and deal with the wider GC stuff separately (i.e. currently its "user responsibility" to maintain their Lambda).

[cmacdonald]

removed two extraneous commits: 6781b12 & c8aa447, and updated to current HEAD.

[tshirtman]

Sorry for not coming back to this before, i now see no reason to wait more to merge this, so it can be part of next release.

[cmacdonald]

Super, thanks for merging!