[FEATURE REQUEST]: udf support for dynamic assemblies

[fwaris]

In a recent talk Michael Rys showed udfs being used in a notebook setting with Azure Synapse.

I was under the impression udfs worked in compiled assemblies only.

Not sure the how the notebooks are coded but when I try to define a udf in F# interactive, I get the following error:

System.NotSupportedException: The invoked member is not supported in a dynamic assembly.
   at System.Reflection.Emit.InternalAssemblyBuilder.get_Location()
   at Microsoft.Spark.Utils.UdfSerDe.SerializeType(Type type)

It would be nice if the creation of Spark udfs in most interactive environments was supported. Let us know if there any plans for this.

[MikeRys]

Can you provide the code and how you distribute it across the cells? UDFs do work in Synapse notebooks, but you have to keep the definition of the function and the registration in the same cell. E.g., Write and call UDFs in .NET for Apache Spark interactive environments. | Microsoft Docs<https://docs.microsoft.com/en-us/dotnet/spark/how-to-guides/dotnet-interactive-udf-issue> Best regards Michael From: fwaris ***@***.***> Sent: Tuesday, July 20, 2021 5:58 PM To: dotnet/spark ***@***.***> Cc: Subscribed ***@***.***> Subject: [dotnet/spark] [FEATURE REQUEST]: udfs support for dynamic assemblies (#957) In a recent talk Michael Rys<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchannel9.msdn.com%2FShows%2FOn-NET%2FWhats-New-in-NET-for-Apache-Spark-v111&data=04%7C01%7Cmrys%40microsoft.com%7C6da042b90e3445ed924408d94be298a1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637624258869844454%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yBLwhWgARVnGVdeYqmuMylK1c3NYNROgyEqFXW0%2BmsQ%3D&reserved=0> showed udfs being used in a notebook setting with Azure Synapse. I was under the impression udfs worked in compiled assemblies only. Not sure the how the notebooks are coded but when I try to define a udf in F# interactive, I get the following error: System.NotSupportedException: The invoked member is not supported in a dynamic assembly. at System.Reflection.Emit.InternalAssemblyBuilder.get_Location() at Microsoft.Spark.Utils.UdfSerDe.SerializeType(Type type) It would be nice if the creation of Spark udfs in most interactive environments was supported. Let us know if there any plans for this. - You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fspark%2Fissues%2F957&data=04%7C01%7Cmrys%40microsoft.com%7C6da042b90e3445ed924408d94be298a1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637624258869854405%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2Rn19T%2BonDxQdz481YaunMQaW5IkY9kVmpYlbxQOBN4%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZXGJCZGLZPCDVOLIOK573TYYLRZANCNFSM5AW5SU3Q&data=04%7C01%7Cmrys%40microsoft.com%7C6da042b90e3445ed924408d94be298a1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637624258869854405%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5vy5GAsxttoJUAcXXHaPNqclrzDJhRy8wGMlx%2BJDOlU%3D&reserved=0>.

[fwaris]

Michael,
Thanks for responding.

I have attached a zip file containing a README.MD with instructions and code to recreate the above issue.

I have been using Spark interactively from F# on my local machine for quite some time now. Its been working well when using plain SQL to manipulate data.

I could not define UDFs in interactive F# code because the incrementally compile F# code resides in a dynamic assembly.

However now that UDFs can be defined in notebook cells, I am hoping that we can achieve the same in F# interactive.

(Note that F# has had a REPL since 2010 and we are quite used to interactive programming in F#. Tooling support is good; intellisense and compiler checks are supported even in script files.)

sparkInteractive.zip

[MikeRys]

Thanks. I think this is best communicated to the .NET interactive team. Please open a feature request issue on their repo at https://github.com/dotnet/interactive. Cheers Michael From: fwaris ***@***.***> Sent: Wednesday, July 21, 2021 4:44 AM To: dotnet/spark ***@***.***> Cc: Michael Rys ***@***.***>; Comment ***@***.***> Subject: Re: [dotnet/spark] [FEATURE REQUEST]: udf support for dynamic assemblies (#957) Michael, Thanks for responding. I have attached a zip file containing a README.MD with instructions and code to recreate the above issue. I have been using Spark interactively from F# on my local machine for quite some time now. Its been working well when using plain SQL to manipulate data. I could not define UDFs in interactive F# code because the incrementally compile F# code resides in a dynamic assembly. However now that UDFs can be defined in notebook cells, I am hoping that we can achieve the same in F# interactive. (Note that F# has had a REPL since 2010 and we are quite used to interactive programming in F#. Tooling support is good; intellisense and compiler checks are supported even in script files.) sparkInteractive.zip<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fspark%2Ffiles%2F6855220%2FsparkInteractive.zip&data=04%7C01%7Cmrys%40microsoft.com%7C03b0adb0c7fe42c0da4408d94c3cdf79%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637624646602396291%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4R7T%2FrZ7dbKGB61Lv7JrRAV%2BTtPe%2BNmXcmPKWUJQ38o%3D&reserved=0> - You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fspark%2Fissues%2F957%23issuecomment-884125882&data=04%7C01%7Cmrys%40microsoft.com%7C03b0adb0c7fe42c0da4408d94c3cdf79%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637624646602406249%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9rUKwZ7UQ62EdzRDEv7yKTKCyFDBY7n%2FlFTkM517s3A%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZXGJFFK62JLAFZAA5PLODTY2XJDANCNFSM5AW5SU3Q&data=04%7C01%7Cmrys%40microsoft.com%7C03b0adb0c7fe42c0da4408d94c3cdf79%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637624646602406249%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BM4MKgNxUP%2BMpknUkgBI%2F%2BcNnmrFnEdeyfqkJF%2FfjbA%3D&reserved=0>.

[GeorgeS2019]

@fwaris
@MikeRys

Is there a solution of doing UDF in the latest of Polyglot notebook without having

Error: System.NotSupportedException: BinaryFormatter serialization and deserialization are disabled within this application. See https://aka.ms/binaryformatter for more information.