Arrow enabled implementation and comparison with pySpark

[voltcode]

It seems that using Arrow enabled Pandas could greatly improve pySpark efficiency. Did you compare against Arrow enabled Pandars or the version without? Is this detail something you could highlight in the comparison? As far as I know ,NET does not have proper Arrow bindings, and it would be of great benefit if it did, did you consider creating it? There's a bit of a discussion in dotnet/machinelearning#69 if you need more context.

[rapoth]

@voltcode: Thank you for your comment! We totally agree and we are working on more benchmarks as I type this :) For a comparison against Arrow enabled Pandas in Python, having efficient .NET support for Arrow is a must.

We are already working very closely with the Arrow community towards getting the initial version done over the next few weeks. @eerhardt from our team has been contributing several PRs lately to Apache Arrow:

• ARROW-4997: [C#] ArrowStreamReader doesn't consume whole stream and doesn't implement sync read. apache/arrow#4017
• ARROW-5019: [C#] ArrowStreamWriter doesn't work on a non-seekable stream apache/arrow#4052
• ARROW-4839: [C#] Add NuGet package metadata and instructions. apache/arrow#3891
• ARROW-4502: [C#] Add support for zero-copy reads apache/arrow#3736
• ARROW-4737: run C# tests in CI apache/arrow#3788
• ARROW-4543: [C#] Update Flat Buffers code to latest version apache/arrow#3637
• ARROW-4435: Minor fixups to csharp .sln and .csproj file apache/arrow#3529

Our plan is that once we have a reasonable Arrow .NET integration, we will run more benchmarks. This is a great point that you made. We are very open to hearing your take on any configurations we should pay special attention to. Since benchmarking is a sensitive topic, we want to be totally transparent about what we are running and how we are running - you can find all the benchmarking code we are using here. Our goal is to have community members guide us towards getting sensible benchmarks.

[eerhardt]

@voltcode - Arrow enabled Pandas UDFs are something we are actively working on. The preliminary numbers are promising when there are lots of records being serialized between the JVM and .NET, and we are working on making them better.

As far as I know ,NET does not have proper Arrow bindings, and it would be of great benefit if it did, did you consider creating it?

A few months ago @chutchinson contributed the initial implementation of the .NET Apache Arrow library and we have been contributing to it as well. The .NET Arrow library shipped on nuget.org (https://www.nuget.org/packages/Apache.Arrow/) as part of the 0.13 release of Apache Arrow earlier this month, and more improvements are on the way. If anyone is interested in helping make the .NET Apache Arrow library better, check out the open .NET Arrow issues, or try out the Arrow NuGet package and log any suggestions/bugs/feedback. Contributions are always welcome.

[voltcode]

@rapoth thank you for the clarification. The benchmark link doesn't work - can you provide a new one?

[imback82]

@voltcode Here is the benchmark link: https://github.com/dotnet/spark/tree/master/benchmark

[rapoth]

@voltcode: Made a linking mistake 🤦 I fixed my original post. Thanks for catching!

[imback82]

cc: @Niharikadutta