.NET for Apache Spark 0.5 Release Notes

New Features and Improvements

Support for DeltaTable APIs in Delta Lake (#236)
Support for Spark 2.3.4/2.4.4 (#232, #240)
Support for UDF taking a Row object as input (#214)
Exposing new APIs:
- Column.IsIn (#171)
- SparkSession.Range(), *ActiveSession(), *DefaultSession() (#225, #266)
- SparkSession.Catalog (#231)
- SparkFiles (#255)
- DataFrameReader.Jdbc() (#262)
- RelationalGroupedDataset.Agg(), Sum(), Mean(), Min(), Max(), Avg() (#263, #265)
- DataStreamWriter.PartitionBy (#270)
Ground work to support dotnet-try as C# REPL (#251)
Bug fix in generating Nuget package (#234)
Support for MapType (#235)

Breaking Changes

None, but the new Microsoft.Spark.Worker needs to be used to enable UDF taking Row object as input (#214).

Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

Spark Version	microsoft-spark JAR
2.3.*	microsoft-spark-2.3.x-0.5.0.jar
2.4.0	microsoft-spark-2.4.x-0.5.0.jar
2.4.1
2.4.3
2.4.4
2.4.2	Not supported