Support for DeltaTable #236

AFFogarty · 2019-09-05T00:43:16Z

This PR adds support for DeltaTable, which provides CRUD operations for Delta tables.

This PR introduces a new project called Microsoft.Spark.Extensions.Delta. This new project will contain all of the code related to Delta support in .Net. If we want to add support for other third-party libraries, we will create new projects in the Microsoft.Spark.Extensions namespace.

The tests require the Delta Jar to be installed in the $SPARK_HOME/jars/ directory.

…/delta_binding

src/csharp/Microsoft.Spark/Microsoft.Spark.csproj

src/csharp/Microsoft.Spark.E2ETest/Microsoft.Spark.E2ETest.csproj

src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTableTests.cs

...p/Microsoft.Spark.Extensions.Delta.UnitTest/Microsoft.Spark.Extensions.Delta.UnitTest.csproj

src/csharp/Microsoft.Spark.Extensions.Delta/Tables/DeltaMergeBuilder.cs

imback82

Please fix unit test failures.

imback82 · 2019-09-05T04:45:03Z

src/csharp/Microsoft.Spark.Extensions.Delta/Microsoft.Spark.Extensions.Delta.csproj

@@ -0,0 +1,11 @@
+<Project Sdk="Microsoft.NET.Sdk">


@rapoth the package version should be follow the delta package version, not Spark .NET version right?

src/csharp/Microsoft.Spark.Extensions.Delta/Tables/DeltaTable.cs

src/csharp/Microsoft.Spark/Microsoft.Spark.csproj

suhsteve · 2019-09-05T18:39:17Z

Please fix unit test failures.

@AFFogarty most likely failing due to missing delta jar. I would start by looking here and probably download and add the delta jars to each spark distro(where applicable).

src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTableTests.cs

…eTests.cs Co-Authored-By: Steve Suh <suhsteve@gmail.com>

src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTableTests.cs

…eTests.cs Co-Authored-By: Steve Suh <suhsteve@gmail.com>

…DeltaMergeBuilder.cs Co-Authored-By: Steve Suh <suhsteve@gmail.com>

…spark into feature/delta_binding

…/delta_binding

AFFogarty · 2019-09-09T21:24:23Z

Using InternalsAssemblyNames builds locally but seems to fail on the Azure pipeline build agent:

  <PropertyGroup>
    <InternalsAssemblyNames>Microsoft.Spark;Microsoft.Spark.E2ETest</InternalsAssemblyNames>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="IgnoresAccessChecksToGenerator" Version="0.4.0" PrivateAssets="All" />
  </ItemGroup>

src/csharp/Extensions/Microsoft.Spark.Extensions.Delta.E2ETest/DeltaTableTests.cs

imback82 · 2019-09-10T04:05:51Z

...ons/Microsoft.Spark.Extensions.Delta.E2ETest/Microsoft.Spark.Extensions.Delta.E2ETest.csproj

+  </ItemGroup>
+
+  <PropertyGroup>
+    <InternalsAssemblyNames>Microsoft.Spark;Microsoft.Spark.E2ETest</InternalsAssemblyNames>


Do you need internal classes of Microsoft.Spark here? What are they?

We need it for Microsoft.Spark.Versions.

src/csharp/Microsoft.Spark.E2ETest/SparkFixture.cs

…/delta_binding

suhsteve

LGTM

src/csharp/Microsoft.Spark.E2ETest/SparkFixture.cs

imback82

Thanks for working on this @AFFogarty. I have few minor comments.

src/csharp/Extensions/Microsoft.Spark.Extensions.Delta.E2ETest/DeltaFixture.cs

...ons/Microsoft.Spark.Extensions.Delta.E2ETest/Microsoft.Spark.Extensions.Delta.E2ETest.csproj

src/csharp/Extensions/Microsoft.Spark.Extensions.Delta.E2ETest/SparkCollection.cs

src/csharp/Extensions/Microsoft.Spark.Extensions.Delta.E2ETest/DeltaFixture.cs

src/csharp/Extensions/Microsoft.Spark.Extensions.Delta.E2ETest/DeltaTableTests.cs

AFFogarty · 2019-09-16T22:13:58Z

Addressed all comments.

imback82

LGTM

AFFogarty added 11 commits September 3, 2019 11:32

Move to separate project

1737ef3

Merge branch 'master' of https://github.com/dotnet/spark into feature…

ac44da3

…/delta_binding

More method comments

5ad8b5c

Clean up usings

0e0a5c0

Clean up

d80b4d3

Add test

5fe9dea

License

def3cd2

Cleanup

b77730f

Tests pass

c7b9607

Rename tempdir

066e8e9

Merge branch 'master' of https://github.com/dotnet/spark into feature…

4ddba3c

…/delta_binding

rapoth requested review from suhsteve and imback82 September 5, 2019 00:49

rapoth reviewed Sep 5, 2019

View reviewed changes

src/csharp/Microsoft.Spark/Microsoft.Spark.csproj Show resolved Hide resolved

AFFogarty added 2 commits September 4, 2019 18:09

Cleanup

e9c2bc2

More cleanup

51df47e

AFFogarty changed the title ~~WIP: Support for DeltaTable~~ Support for DeltaTable Sep 5, 2019

suhsteve reviewed Sep 5, 2019

View reviewed changes

imback82 reviewed Sep 5, 2019

View reviewed changes

suhsteve reviewed Sep 5, 2019

View reviewed changes

src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTableTests.cs Outdated Show resolved Hide resolved

suhsteve reviewed Sep 5, 2019

View reviewed changes

src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTableTests.cs Outdated Show resolved Hide resolved

AFFogarty and others added 4 commits September 5, 2019 16:12

Update src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTabl…

3d678fd

…eTests.cs Co-Authored-By: Steve Suh <suhsteve@gmail.com>

Update src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTabl…

887fb83

…eTests.cs Co-Authored-By: Steve Suh <suhsteve@gmail.com>

Update src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTabl…

8ad247c

…eTests.cs Co-Authored-By: Steve Suh <suhsteve@gmail.com>

Address some comments

6abe065

suhsteve reviewed Sep 6, 2019

View reviewed changes

src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTableTests.cs Outdated Show resolved Hide resolved

Update src/csharp/Microsoft.Spark.Extensions.Delta.UnitTest/DeltaTabl…

51c01b7

…eTests.cs Co-Authored-By: Steve Suh <suhsteve@gmail.com>

imback82 assigned AFFogarty Sep 6, 2019

AFFogarty and others added 9 commits September 9, 2019 13:07

Update src/csharp/Extensions/Microsoft.Spark.Extensions.Delta/Tables/…

ede3665

…DeltaMergeBuilder.cs Co-Authored-By: Steve Suh <suhsteve@gmail.com>

Fix spacing

723fad8

Merge branch 'feature/delta_binding' of https://github.com/AFFogarty/…

609ea22

…spark into feature/delta_binding

Merge branch 'master' of https://github.com/dotnet/spark into feature…

c876f2c

…/delta_binding

Simplify packagesArg

244f4bf

Use IgnoresAccessChecksToGenerator

a2ed899

Don't need using

d39c3ec

Fixed internals for E2E

1979d36

Reference base project

bd879a2

AFFogarty added 3 commits September 9, 2019 15:35

Add props

a3b98cc

Is shipping

a84d5e6

Remove props

4ae00d5

imback82 reviewed Sep 10, 2019

View reviewed changes

AFFogarty added 8 commits September 10, 2019 09:21

Var

630dce7

Only use Delta for delta tests

664bc8f

Merge branch 'master' of https://github.com/dotnet/spark into feature…

9abb18e

…/delta_binding

Clean up usings

279a221

Merge branch 'master' of https://github.com/dotnet/spark into feature…

04ea308

…/delta_binding

Store SparkFixture environment variable names in a class

93d76c3

Fix comment

d7be117

Fix long lines

6e50762

suhsteve previously approved these changes Sep 11, 2019

View reviewed changes

src/csharp/Microsoft.Spark.E2ETest/SparkFixture.cs Outdated Show resolved Hide resolved

More unique name

15c3f27

AFFogarty dismissed suhsteve’s stale review via 15c3f27 September 11, 2019 22:06

imback82 reviewed Sep 14, 2019

View reviewed changes

Addressed Terry's comments

d57edbe

imback82 approved these changes Sep 17, 2019

View reviewed changes

imback82 merged commit b8edbbf into dotnet:master Sep 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for DeltaTable #236

Support for DeltaTable #236

AFFogarty commented Sep 5, 2019

imback82 left a comment

imback82 Sep 5, 2019

suhsteve commented Sep 5, 2019

AFFogarty commented Sep 9, 2019 •

edited

Loading

imback82 Sep 10, 2019

AFFogarty Sep 10, 2019

suhsteve left a comment

imback82 left a comment

AFFogarty commented Sep 16, 2019

imback82 left a comment

Support for DeltaTable #236

Support for DeltaTable #236

Conversation

AFFogarty commented Sep 5, 2019

imback82 left a comment

Choose a reason for hiding this comment

imback82 Sep 5, 2019

Choose a reason for hiding this comment

suhsteve commented Sep 5, 2019

AFFogarty commented Sep 9, 2019 • edited Loading

imback82 Sep 10, 2019

Choose a reason for hiding this comment

AFFogarty Sep 10, 2019

Choose a reason for hiding this comment

suhsteve left a comment

Choose a reason for hiding this comment

imback82 left a comment

Choose a reason for hiding this comment

AFFogarty commented Sep 16, 2019

imback82 left a comment

Choose a reason for hiding this comment

AFFogarty commented Sep 9, 2019 •

edited

Loading