.NET for Apache Spark 0.9 Release Notes

New Features and Improvements

Expose DataStreamWriter.Foreach API (#387)
Support UDF that returns Row object (#376, #406, #411)
Support for Bucketizer (#378)
Support DateType (#420)

Breaking Changes

The prior versions (<0.9) of Microsoft.Spark.Worker are not compatible with this release due to the internal changes related to UDFs.

Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

Spark Version	microsoft-spark JAR
2.3.*	microsoft-spark-2.3.x-0.9.0.jar
2.4.0	microsoft-spark-2.4.x-0.9.0.jar
2.4.1
2.4.3
2.4.4
2.4.5
2.4.2	Not supported