Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prep 0.11.0 release #507

Merged
merged 23 commits into from
May 7, 2020
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
03b7939
Adding section for UDF serialization
Niharikadutta Apr 20, 2020
4ef693d
removing guides from master
Niharikadutta Apr 20, 2020
81145ca
Merge latest from master
Niharikadutta May 6, 2020
49a437d
Prep changes for 0.11.0 version
Niharikadutta May 6, 2020
014e359
Merge branch 'master' into nidutta/0.11.0-release
imback82 May 6, 2020
bde7231
Adding table for breaking changes
Niharikadutta May 6, 2020
ac0db5e
Merge branch 'nidutta/0.11.0-release' of github.com:Niharikadutta/spa…
Niharikadutta May 6, 2020
598ea15
table formatting error
Niharikadutta May 6, 2020
4ddfb2c
Added Compatibility section
Niharikadutta May 6, 2020
d748beb
Added description for compatibility table
Niharikadutta May 6, 2020
dfadf09
Added link to PRs of incompatible features
Niharikadutta May 6, 2020
e9eb4c8
testing remove the link name
Niharikadutta May 6, 2020
d27baae
test 2
Niharikadutta May 6, 2020
fb625b3
test3
Niharikadutta May 6, 2020
0dfe622
using href tags
Niharikadutta May 6, 2020
62f91c4
using href tags to remove link url from table
Niharikadutta May 6, 2020
63f2cac
PR review changes
Niharikadutta May 6, 2020
e06399a
PR comment changes
Niharikadutta May 6, 2020
b843cd8
PR comment changes
Niharikadutta May 6, 2020
a8dab1f
add forward compatibility table
elvaliuliuliu May 6, 2020
da5585a
resolve comments
elvaliuliuliu May 6, 2020
7b72e24
Merge pull request #1 from elvaliuliuliu/elva/addFC
Niharikadutta May 6, 2020
e7cd165
Merge branch 'master' into nidutta/0.11.0-release
imback82 May 7, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
<tbody align="center">
<tr>
<td >2.3.*</td>
<td rowspan=6><a href="https://github.com/dotnet/spark/releases/tag/v0.10.0">v0.10.0</a></td>
<td rowspan=6><a href="https://github.com/dotnet/spark/releases/tag/v0.11.0">v0.11.0</a></td>
</tr>
<tr>
<td>2.4.0</td>
Expand Down
2 changes: 1 addition & 1 deletion benchmark/scala/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.microsoft.spark</groupId>
<artifactId>microsoft-spark-benchmark</artifactId>
<version>0.10.0</version>
<version>0.11.0</version>
<inceptionYear>2019</inceptionYear>
<properties>
<encoding>UTF-8</encoding>
Expand Down
101 changes: 101 additions & 0 deletions docs/release-notes/0.11/release-0.11.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# .NET for Apache Spark 0.11 Release Notes

### New Features and Improvements

* Ability to pass and return [corefxlab](https://github.com/dotnet/corefxlab) DataFrames to UDF APIs ([#277](https://github.com/dotnet/spark/pull/277))
* Support for ML TF-IDF (Term frequency-inverse document frequency) feature vectorization method ([#394](https://github.com/dotnet/spark/pull/394))
* Support for TimestampType in `DataFrame.Collect()`, `CreateDataFrame` and UDFs ([#428](https://github.com/dotnet/spark/pull/428))
* Support for Broadcast Variables ([#414](https://github.com/dotnet/spark/pull/414))
* Support for ML feature Word2Vec ([#491](https://github.com/dotnet/spark/pull/491))
* Streamline logging when there is a failure ([#439](https://github.com/dotnet/spark/pull/439))


### Breaking Changes

* `SparkSession.Catalog` is changed from a method to a property ([#508](https://github.com/dotnet/spark/pull/508))

### Compatibility

#### Backward compatibility

The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker.

<table>
Niharikadutta marked this conversation as resolved.
Show resolved Hide resolved
<thead>
<tr>
<th>Oldest compatible Microsoft.Spark.Worker version</th>
<th>Incompatible features</th>
</tr>
</thead>
<tbody align="center">
<tr>
<td rowspan=4>v0.9.0</td>
<td>DataFrame with Grouped Map UDF <a href="https://github.com/dotnet/spark/pull/277">(#277)</a></td>
</tr>
<tr>
<td>DataFrame with Vector UDF <a href="https://github.com/dotnet/spark/pull/277">(#277)</a></td>
</tr>
<tr>
<td>Support for Broadcast Variables <a href="https://github.com/dotnet/spark/pull/414">(#414)</a></td>
</tr>
<tr>
<td>Support for TimestampType <a href="https://github.com/dotnet/spark/pull/428">(#428)</a></td>
</tr>
</tbody>
</table>

#### Forward compatibility

The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with.

<table>
<thead>
<tr>
<th>Oldest compatible .NET for Apache Spark release version</th>
</tr>
</thead>
<tbody align="center">
<tr>
<td>v0.9.0</td>
</tr>
</tbody>
</table>

### Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

<table>
<thead>
<tr>
<th>Spark Version</th>
<th>microsoft-spark JAR</th>
</tr>
</thead>
<tbody align="center">
<tr>
<td>2.3.*</td>
<td>microsoft-spark-2.3.x-0.11.0.jar</td>
</tr>
<tr>
<td>2.4.0</td>
<td rowspan=5>microsoft-spark-2.4.x-0.11.0.jar</td>
</tr>
<tr>
<td>2.4.1</td>
</tr>
<tr>
<td>2.4.3</td>
</tr>
<tr>
<td>2.4.4</td>
</tr>
<tr>
<td>2.4.5</td>
</tr>
<tr>
<td>2.4.2</td>
<td><a href="https://github.com/dotnet/spark/issues/60">Not supported</a></td>
</tr>
</tbody>
</table>
2 changes: 1 addition & 1 deletion eng/Versions.props
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<VersionPrefix>0.10.0</VersionPrefix>
<VersionPrefix>0.11.0</VersionPrefix>
<PreReleaseVersionLabel>prerelease</PreReleaseVersionLabel>
<RestoreSources>
$(RestoreSources);
Expand Down
2 changes: 1 addition & 1 deletion src/scala/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<version>${microsoft-spark.version}</version>
<properties>
<encoding>UTF-8</encoding>
<microsoft-spark.version>0.10.0</microsoft-spark.version>
<microsoft-spark.version>0.11.0</microsoft-spark.version>
</properties>

<modules>
Expand Down