Support .NET Core 3.1 #291

imback82 · 2019-10-12T23:36:26Z

Now that .NET Core 3.0 is officially released, we are moving worker binaries from netcoreapp2.1 to netcoreapp3.1.

imback82 · 2019-10-12T23:40:42Z

I will also update docs that are related to building from source.

imback82 · 2019-10-13T00:16:17Z

cc: @eerhardt, can you check if I am missing anything migrating to netcore 3.0? Thanks!

eerhardt · 2019-10-14T14:36:42Z

docs/building/windows-instructions.md

@@ -20,7 +20,7 @@ Building Spark .NET on Windows

 If you already have all the pre-requisites, skip to the [build](windows-instructions.md#building) steps below.

-  1. Download and install the **[.NET Core SDK](https://dotnet.microsoft.com/download/dotnet-core/2.1)** - installing the SDK will add the `dotnet` toolchain to your path. .NET Core 2.1, 2.2 and 3.0 preview are supported.
+  1. Download and install the **[.NET Core 3.0 SDK](https://dotnet.microsoft.com/download/dotnet-core/3.0)** - installing the SDK will add the `dotnet` toolchain to your path.
  2. Install any edition of **[Visual Studio 2019](https://www.visualstudio.com/downloads/)** or [Visual Studio 2017](https://www.visualstudio.com/downloads/). The Community version is completely free. When configuring your installation, include these components at minimum:


Since we are now building for netcoreapp3.0, we now require VS 2019 Update 3. So this should be updated.

updated to install 16.4 or higher. thanks!

eerhardt · 2019-10-14T14:40:26Z

src/csharp/Extensions/Microsoft.Spark.Extensions.Delta/Microsoft.Spark.Extensions.Delta.csproj

@@ -1,7 +1,7 @@
 <Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
-    <TargetFramework>netstandard2.0</TargetFramework>
+    <TargetFrameworks>netstandard2.0;netstandard2.1</TargetFrameworks>


You don't need to make this change. You can just target netstandard2.0 and everything will work find on .NET Core 3.0.

The only reason you would multi-target for standard 2.0 and 2.1 is if you wanted to use a new API in 2.1 that wasn't available in 2.0, which it doesn't appear that we are doing.

For consideration with targeting netstandard2.1:

.NET Framework won't support .NET Standard 2.1 or later versions. For more details, see the announcement of .NET Standard 2.1.

Source: https://docs.microsoft.com/en-us/dotnet/standard/net-standard

That's why it is targeting both netstandard2.0 and netstandard2.1. .NET Framework will use the netstandard2.0 compatible version.

eerhardt · 2019-10-14T14:40:55Z

src/csharp/Microsoft.Spark/Microsoft.Spark.csproj

@@ -1,7 +1,7 @@
 <Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
-    <TargetFramework>netstandard2.0</TargetFramework>
+    <TargetFrameworks>netstandard2.0;netstandard2.1</TargetFrameworks>


You don't need to make this change. You can just target netstandard2.0 and everything will work find on .NET Core 3.0.

The only reason you would multi-target for standard 2.0 and 2.1 is if you wanted to use a new API in 2.1 that wasn't available in 2.0, which it doesn't appear that we are doing.

For consideration with targeting netstandard2.1:

.NET Framework won't support .NET Standard 2.1 or later versions. For more details, see the announcement of .NET Standard 2.1.

Source: https://docs.microsoft.com/en-us/dotnet/standard/net-standard

eerhardt · 2019-10-14T14:41:44Z

src/csharp/Microsoft.Spark.UnitTest/Microsoft.Spark.UnitTest.csproj

@@ -10,7 +10,7 @@
    <PackageReference Include="System.Memory" Version="4.5.2" />
  </ItemGroup>

-  <ItemGroup Condition=" '$(TargetFramework)' != 'netcoreapp2.1' ">
+  <ItemGroup Condition=" '$(TargetFramework)' != 'netcoreapp3.0' ">


This can just get removed. There is never a case where TargetFramework != netcoreapp3.0.

eerhardt

I just want to make sure you are aware of the .NET Core support policy:

https://github.com/dotnet/core/blob/master/microsoft-support.md
https://github.com/dotnet/core/blob/master/roadmap.md

3.0 is the Current release of .NET Core. While 2.1 is still the LTS (long-term support) version. 3.1 will be the next LTS version shipping next month. Which means 3.0 will go out of support in about 4 months. So we will need to upgrade to 3.1 when it comes out. This shouldn't block us from upgrading to 3.0 today, but I just wanted to ensure you were aware of the policy.

imback82 · 2019-10-14T14:57:33Z

3.0 is the Current release of .NET Core. While 2.1 is still the LTS (long-term support) version. 3.1 will be the next LTS version shipping next month. Which means 3.0 will go out of support in about 4 months. So we will need to upgrade to 3.1 when it comes out. This shouldn't block us from upgrading to 3.0 today, but I just wanted to ensure you were aware of the policy.

Thanks @eerhardt. In this case, I will leave this PR open and update it to 3.1 when it's released next month.

AeroXuk · 2019-10-29T14:23:58Z

src/csharp/Microsoft.Spark.Experimental/Microsoft.Spark.Experimental.csproj


  <PropertyGroup>
-    <TargetFramework>netstandard2.0</TargetFramework>
+    <TargetFrameworks>netstandard2.0;netstandard2.1</TargetFrameworks>


See other netstandard2.1 comments on this PR.

imback82 · 2019-12-03T19:41:51Z

Now that .NET Core 3.1 is released, I will resume this work.

imback82 · 2020-01-04T17:49:50Z

src/csharp/Microsoft.Spark/Microsoft.Spark.csproj

@@ -1,7 +1,7 @@
 <Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
-    <TargetFrameworks>netstandard2.0;netcoreapp2.1;netstandard2.1</TargetFrameworks>


@eerhardt Should I add netcoreapp3.1 instead of replacing netcoreapp2.1?

For libraries (like Microsoft.Spark.csproj), you don't need to target all the frameworks you support running on. You just need to target the lowest/earliest frameworks that you support.

For example, if you just target netstandard2.0, your library will run on any desktop .NET Framework 4.6.1 or higher. And it will run on .NET Core 2.0 or higher.

The reason you would multi-target a library is so you can have different code that will execute when running on different frameworks. For example, .NET Core 3.1 adds a new API you want to use, so you multi-target to netstandard2.0;netcoreapp3.1. And now you can use #if conditions in your code to compile different code when building for netstandard2.0, and use the new API when building for netcoreapp3.1.

Since you don't have netcoreapp3.1 specific code in this library, it isn't necessary to put netcoreapp3.1 in this list at all.

The only framework-specific code I see in this project is:

spark/src/csharp/Microsoft.Spark/Sql/Types/DataType.cs

Lines 208 to 219 in 8dc789f

private static string NormalizeTypeName(string typeName)

{

#if !NETSTANDARD2_0

return string.Create(typeName.Length - 4, typeName, (span, name) =>

{

name.AsSpan(0, name.Length - 4).ToLower(span, CultureInfo.CurrentCulture);

});

#else

return typeName.Substring(0, typeName.Length - 4).ToLower();

#endif

}

}

So in all actuality, if you want to drop 1st class support for .NET Core 2.1, you would really only need:

<TargetFrameworks>netstandard2.0;netstandard2.1</TargetFrameworks> on this line. (Because .NET Core 3.1 is compatible with netstandard2.1, so the netstandard2.1 compiled assemblies will be used when running on .NET Core 3.1.)

Thanks @eerhardt. netcoreapp2.1 was added from this discussion: #364 (comment). I will remove netcoreapp2.1 for now and will add it if it is absolutely needed.

The only effect this will cause is if someone is on .NET Core 2.1 and uses the Microsoft.Spark library, the above code will use the slower Substring/ToLower code since the netstandard2.0 built version will be used. That doesn't seem too bad, especially if you aren't supporting .NET Core 2.1 on the worker anymore.

Yea, and this code is driver side and not used by worker anyway. Thanks @eerhardt!

initial checkin

6e74159

imback82 added 3 commits October 12, 2019 16:49

testing updating netcore in build.

a41aa1c

Fix script that publishes worker

22eb9eb

build mvn fix

900c63e

imback82 self-assigned this Oct 13, 2019

imback82 added the enhancement New feature or request label Oct 13, 2019

Update docs

c99dfff

imback82 mentioned this pull request Oct 13, 2019

Adding Hardware acceleration version of Queries 1 and 8 in csharp #274

Open

4 tasks

Merge branch 'master' into netcore3.0

14aa619

eerhardt reviewed Oct 14, 2019

View reviewed changes

AeroXuk reviewed Oct 29, 2019

View reviewed changes

imback82 mentioned this pull request Nov 9, 2019

[Documentation] Deploy Worker and Udf Binaries #329

Merged

Merge branch 'master' into netcore3.0

66fe798

imback82 changed the title ~~Support .NET Core 3.0~~ Support .NET Core 3.1 Jan 3, 2020

imback82 added 4 commits January 3, 2020 13:18

address comments

1698aca

update docs

a29d8e5

revert changes

00b3bb2

clean up

bc08290

imback82 commented Jan 4, 2020

View reviewed changes

address PR comments

c0efdcb

imback82 merged commit f83a1eb into dotnet:master Jan 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support .NET Core 3.1 #291

Support .NET Core 3.1 #291

imback82 commented Oct 12, 2019 •

edited

Loading

imback82 commented Oct 12, 2019

imback82 commented Oct 13, 2019

eerhardt Oct 14, 2019

imback82 Jan 3, 2020

eerhardt Oct 14, 2019

AeroXuk Oct 29, 2019

eerhardt Oct 29, 2019

imback82 Jan 3, 2020

eerhardt Oct 14, 2019

AeroXuk Oct 29, 2019

imback82 Jan 3, 2020

eerhardt Oct 14, 2019

eerhardt left a comment

imback82 commented Oct 14, 2019

AeroXuk Oct 29, 2019

imback82 commented Dec 3, 2019

imback82 Jan 4, 2020

eerhardt Jan 6, 2020

imback82 Jan 6, 2020

eerhardt Jan 6, 2020 •

edited

Loading

imback82 Jan 6, 2020

	private static string NormalizeTypeName(string typeName)
	{
	#if !NETSTANDARD2_0
	return string.Create(typeName.Length - 4, typeName, (span, name) =>
	{
	name.AsSpan(0, name.Length - 4).ToLower(span, CultureInfo.CurrentCulture);
	});
	#else
	return typeName.Substring(0, typeName.Length - 4).ToLower();
	#endif
	}
	}

Support .NET Core 3.1 #291

Support .NET Core 3.1 #291

Conversation

imback82 commented Oct 12, 2019 • edited Loading

imback82 commented Oct 12, 2019

imback82 commented Oct 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eerhardt left a comment

Choose a reason for hiding this comment

imback82 commented Oct 14, 2019

Choose a reason for hiding this comment

imback82 commented Dec 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eerhardt Jan 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imback82 commented Oct 12, 2019 •

edited

Loading

eerhardt Jan 6, 2020 •

edited

Loading