From 32789e0f844aa3c9a09286287456d780410c97ba Mon Sep 17 00:00:00 2001 From: Niharika Dutta Date: Sat, 27 Mar 2021 08:46:29 -0700 Subject: [PATCH] Prep 1.1 release (#757) --- README.md | 2 +- benchmark/scala/pom.xml | 2 +- docs/release-notes/1.1/release-1.1.md | 217 ++++++++++++++++++++++++++ eng/Versions.props | 2 +- src/scala/pom.xml | 2 +- 5 files changed, 221 insertions(+), 4 deletions(-) create mode 100644 docs/release-notes/1.1/release-1.1.md diff --git a/README.md b/README.md index 721e37a7d..2b75e5c20 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ 2.3 - v1.0.0 + v1.1.0 2.4* diff --git a/benchmark/scala/pom.xml b/benchmark/scala/pom.xml index c1f61f288..656297d99 100644 --- a/benchmark/scala/pom.xml +++ b/benchmark/scala/pom.xml @@ -3,7 +3,7 @@ 4.0.0 com.microsoft.spark microsoft-spark-benchmark - 1.0.0 + 1.1.0 2019 UTF-8 diff --git a/docs/release-notes/1.1/release-1.1.md b/docs/release-notes/1.1/release-1.1.md new file mode 100644 index 000000000..3503d1076 --- /dev/null +++ b/docs/release-notes/1.1/release-1.1.md @@ -0,0 +1,217 @@ +# .NET for Apache Spark 1.1 Release Notes + +### Deprecation notice for Spark 2.3 + +We are planning to drop the support for Spark 2.3 in the 2.0 release, which will be the next release. +The last Spark 2.3 release (2.3.4) was back in September 2019, and no new release is planned for Spark 2.3. Since there have been no new features introduced for Spark 2.3 in the last few releases of .NET for Apache Spark, if you are relying on Spark 2.3, you should be able to continue using .NET for Apache Spark 1.x. + +### New Features/Improvements + +* Support for Arrow 2.0 and GroupedMapUdf in Spark 3.0.0 ([#711](https://github.com/dotnet/spark/pull/711)) +* Use pattern matching in arrow test utils to improve readability ([#725](https://github.com/dotnet/spark/pull/725)) +* Check whether file is found before trying to dereference it ([#759](https://github.com/dotnet/spark/pull/759)) +* Ml/feature hasher has only internal contructors ([#761](https://github.com/dotnet/spark/pull/761)) +* Support for stop words removers ([#726](https://github.com/dotnet/spark/pull/726)) +* Support for adding NGram functionality ([#734](https://github.com/dotnet/spark/pull/734)) +* Add support for SQLTransformer ML feature ([#781](https://github.com/dotnet/spark/pull/781)) +* Add skeletal support for FileSystem extension ([#787](https://github.com/dotnet/spark/pull/787)) +* Using (processId, threadId) as key to mantain threadpool executor instead of only threadId ([#793](https://github.com/dotnet/spark/pull/793)) +* Support for Hyperspace 0.4.0 ([#815](https://github.com/dotnet/spark/pull/815)) +* Support for Delta Lake 0.8.0 ([#823](https://github.com/dotnet/spark/pull/823)) +* Add support for Spark 3.0.2 ([#833](https://github.com/dotnet/spark/pull/833)) +* Add DOTNET_WORKER__DIR environment variable ([#861](https://github.com/dotnet/spark/pull/861)) +* Add spark.dotnet.ignoreSparkPatchVersionCheck conf to ignore patch version in DotnetRunner ([#862](https://github.com/dotnet/spark/pull/862)) + +### Bug Fixes + +* Fix signer information mismatch issue ([#752](https://github.com/dotnet/spark/pull/752)) +* Fix package-worker.ps1 to handle output path with ":" ([#742](https://github.com/dotnet/spark/pull/742)) +* Fix for using Broadcast variables in Databricks ([#766](https://github.com/dotnet/spark/pull/766)) +* Fix macOS Catalina Permissions ([#784](https://github.com/dotnet/spark/pull/784)) +* Fix for memory leak in JVMObjectTracker ([#801](https://github.com/dotnet/spark/pull/801)) +* Bug Fix for Spark 3.x - Avoid converting converted Row values ([#868](https://github.com/dotnet/spark/pull/868)) +* Add 'Z' to the string format in Timestamp.ToString() to indicate UTC time ([#871](https://github.com/dotnet/spark/pull/871)) + +### Infrastructure / Documentation / Etc. + +* Fix flaky CallbackTests.TestCallbackHandlers Test ([#745](https://github.com/dotnet/spark/pull/745)) +* Run E2E tests on Linux in build pipeline and add Backward/Forward E2E tests ([#737](https://github.com/dotnet/spark/pull/737)) +* Update dotnet-interactive deprecated feed ([#807](https://github.com/dotnet/spark/pull/807), [#808](https://github.com/dotnet/spark/pull/808)) +* Remove unnecessary RestoreSources ([#812](https://github.com/dotnet/spark/pull/812)) +* Migrating master to main branch ([#847](https://github.com/dotnet/spark/pull/847), [#849](https://github.com/dotnet/spark/pull/849)) + +### Breaking Changes + +* None + +### Known Issues + +* Broadcast variables do not work with [dotnet-interactive](https://github.com/dotnet/interactive) ([#561](https://github.com/dotnet/spark/pull/561)) +* UDFs defined using class objects with closures does not work with [dotnet-interactive](https://github.com/dotnet/interactive) ([#619](https://github.com/dotnet/spark/pull/619)) +* In [dotnet-interactive](https://github.com/dotnet/interactive) blocking Spark methods that require external threads to unblock them does not work. ie `StreamingQuery.AwaitTermination` requires `StreamingQuery.Stop` to unblock ([#736](https://github.com/dotnet/spark/pull/736)) +* UDFs don't work in Linux with Spark 2.3.0 ([#753](https://github.com/dotnet/spark/issues/753)) + +### Compatibility + +#### Backward compatibility + +The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker. + + + + + + + + + + + + + + +
Oldest compatible Microsoft.Spark.Worker versionIncompatible features
v1.0.0GroupedMap in Spark 3.0 is not compatible with Worker 1.0 (#654)*
+* This is not a breaking change since this feature never worked with Worker 1.0.0. + +#### Forward compatibility + +The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with. + + + + + + + + + + + + +
Oldest compatible .NET for Apache Spark release version
v1.0.0
+ +### Supported Spark Versions + +The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Spark Versionmicrosoft-spark JAR
2.3.*microsoft-spark-2-3_2.11-1.1.0.jar
2.4.0microsoft-spark-2-4_2.11-1.1.0.jar
2.4.1
2.4.3
2.4.4
2.4.5
2.4.6
2.4.7
2.4.2Not supported
3.0.0microsoft-spark-3-0_2.12-1.1.0.jar
3.0.1
3.0.2
+ +### Supported Delta Versions + +The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Delta VersionMicrosoft.Spark.Extensions.Delta
0.1.01.1.0
0.2.0
0.3.0
0.4.0
0.5.0
0.6.0
0.6.1
0.7.0
0.8.0
+ +### Supported Hyperspace Versions + +The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with: + + + + + + + + + + + + + + + + + + + + + + + +
Hyperspace VersionMicrosoft.Spark.Extensions.Hyperspace
0.1.01.1.0
0.2.0
0.3.0
0.4.0
diff --git a/eng/Versions.props b/eng/Versions.props index 04b66403c..fc82f0815 100644 --- a/eng/Versions.props +++ b/eng/Versions.props @@ -1,7 +1,7 @@ - 1.0.0 + 1.1.0 prerelease diff --git a/src/scala/pom.xml b/src/scala/pom.xml index bb0b408ae..e852b5b7a 100644 --- a/src/scala/pom.xml +++ b/src/scala/pom.xml @@ -7,7 +7,7 @@ ${microsoft-spark.version} UTF-8 - 1.0.0 + 1.1.0