[FEATURE REQUEST]: Support GroupedMapUdf in Spark-3.0.0

[suhsteve]

Spark has changed the expected contents of an Arrow RecordBatch from 2.4 and 3.0. Spark 3.0 expects the results of the GroupedMap to be a column of StructType instead of separating each column from the DataFrame.

SPARK-27163

Spark2.4

• https://github.com/apache/spark/blob/807e0a484d1de767d1f02bd8a622da6450bdf940/python/pyspark/serializers.py#L219-L261
• https://github.com/apache/spark/blob/807e0a484d1de767d1f02bd8a622da6450bdf940/sql/core/src/main/scala/org/apache/spark/sql/execution/python/FlatMapGroupsInPandasExec.scala#L140-L148

Spark3.0

• https://github.com/apache/spark/blob/3fdfce3120f307147244e5eaf46d61419a723d50/python/pyspark/sql/pandas/serializers.py#L135-L193
• https://github.com/apache/spark/blob/3fdfce3120f307147244e5eaf46d61419a723d50/sql/core/src/main/scala/org/apache/spark/sql/execution/python/FlatMapGroupsInPandasExec.scala#L94
• https://github.com/apache/spark/blob/3fdfce3120f307147244e5eaf46d61419a723d50/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PandasGroupUtils.scala#L38-L54

To support these changes we need to use Arrow's StructArray and StructType. Unfortunately it is currently unsupported ARROW-6972

[rapoth]

@eerhardt Would you be able to help with ARROW-6972? I'm not sure if anyone from the Arrow community is already working on it. Looks like we cannot support GroupedMap for Spark 3.0 in .NET without that functionality.

[eerhardt]

@eerhardt Would you be able to help with ARROW-6972?

I should be able to look into it later this week.

[rapoth]

Great, thank you so much in advance!

[pgovind]

Just adding a note that I'll be working on this starting tomorrow or Monday :) I'll update this post as I make progress!

[rapoth]

I'm closing this issue considering it is done. Thank you all!