Skip to content

Latest commit

 

History

History
51 lines (45 loc) · 2.26 KB

release-0.4.md

File metadata and controls

51 lines (45 loc) · 2.26 KB

.NET for Apache Spark 0.4 Release Notes

New Features and Improvements

  • Revamped loading assemblies used inside UDFs (#180)
  • Support for Vector UDFs (#127)
  • Support for Grouped Map UDFs (#143)
  • Resolve the signer mismatch issue (#186)
  • Ability to launch a debugger from the worker (#150)
  • Compatibility check for Microsoft.Spark.dll in the worker (#170)
  • Update Apache.Arrow to v0.14.1 (#167)
  • Support for RuntimeConfig in SparkSession (#184)
  • Support for Trigger in DataStreamWriter (#153)
  • The ability to use --archives option to deploy the worker binaries and assemblies that UDFs depend on (#187)

Breaking Changes

  • Due to the changes in loading assemblies (#180), the Microsoft.Spark.Worker executable is no longer compatible with the previous versions.
  • The --class org.apache.spark.deploy.DotnetRunner option in spark-submit is changed to --class org.apache.spark.deploy.dotnet.DotnetRunner to resolve the signer mismatch issue (#186).

Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

Spark Version microsoft-spark JAR
2.3.* microsoft-spark-2.3.x-0.4.0.jar
2.4.0 microsoft-spark-2.4.x-0.4.0.jar
2.4.1
2.4.3
2.4.2 Not supported