Code

Broadcast variable support#414

Code

Merged

imback82 merged 62 commits intodotnet:masterdotnet/spark:masterfrom Niharikadutta:nidutta/BroadcastVariableSupportNiharikadutta/spark:nidutta/BroadcastVariableSupportCopy head branch name to clipboardApr 2, 2020

+479-33

Commits

Commits on Feb 4, 2020

• First commit for initial design draft of Broadcast variable support.

  Niharikadutta

  committed
  0c38446

  Copy full SHA for 0c38446

Commits on Feb 10, 2020

• Adding code skeleton

  Niharikadutta

  committed
  0e5296c

  Copy full SHA for 0e5296c

Commits on Feb 12, 2020

• Added python helper function to call broadcast in scala side

  Niharikadutta

  committed
  e03033e

  Copy full SHA for e03033e

Commits on Feb 19, 2020

• Added encryption support and working code with encryption disabled

  Niharikadutta

  committed
  89ae9e1

  Copy full SHA for 89ae9e1

• removed unused code, cleanup

  Niharikadutta

  committed
  85b05c6

  Copy full SHA for 85b05c6

Commits on Feb 20, 2020

• pushing rough draft of worker side broadcast processing

  Niharikadutta

  committed
  0cb06cd

  Copy full SHA for 0cb06cd

Commits on Feb 22, 2020

• Added working test cases for broadcast without encryption and cleaned code up

  Niharikadutta

  committed
  cab401c

  Copy full SHA for cab401c

• Added comments

  Niharikadutta

  committed
  9104a8a

  Copy full SHA for 9104a8a

• Added call to Unpersist broadcast variable to test it is being removed successfully

  Niharikadutta

  committed
  6b3acbd

  Copy full SHA for 6b3acbd

• Removing extra whitespace

  Niharikadutta

  committed
  4ab4ba3

  Copy full SHA for 4ab4ba3

• Reverting comment changes as per previous state

  Niharikadutta

  committed
  13afe99

  Copy full SHA for 13afe99

• merging latest from master

  Niharikadutta

  committed
  68cb556

  Copy full SHA for 68cb556

• Removed runtime identifier added for local publishing

  Niharikadutta

  committed
  8eeb875

  Copy full SHA for 8eeb875

Commits on Feb 23, 2020

• Merge branch 'master' into nidutta/BroadcastVariableSupport

  Niharikadutta

  committed
  faab81b

  Copy full SHA for faab81b

Commits on Feb 24, 2020

• Added support for Spark 2.3.0 and 2.3.1 that uses different logic to broadcast variables

  Niharikadutta

  committed
  c228849

  Copy full SHA for c228849

• PR review comment changes

  Niharikadutta

  committed
  7255874

  Copy full SHA for 7255874

• Removing since default is false for spark.io.encryption.enabled

  Niharikadutta

  committed
  72b4c0b

  Copy full SHA for 72b4c0b

Commits on Feb 25, 2020

• PR review comment changes

  Niharikadutta

  committed
  b813bd1

  Copy full SHA for b813bd1

• Testing pipeline failure

  Niharikadutta

  committed
  4d355a5

  Copy full SHA for 4d355a5

• Refactoring test cases to be more readable

  Niharikadutta

  committed
  5eeae3c

  Copy full SHA for 5eeae3c

• retriggering stuck pipeline

  Niharikadutta

  committed
  2c56a95

  Copy full SHA for 2c56a95

Commits on Feb 27, 2020

• redoing PR review changes to locate pipeline failure

  Niharikadutta

  committed
  6991381

  Copy full SHA for 6991381

Commits on Feb 29, 2020

• PR review comments changes

  Niharikadutta

  committed
  847446a

  Copy full SHA for 847446a

Commits on Mar 2, 2020

• PR review comment changes

  Niharikadutta

  committed
  21b7ac0

  Copy full SHA for 21b7ac0

• Adding test for Broadcast.Destroy()

  Niharikadutta

  committed
  12d8641

  Copy full SHA for 12d8641

Commits on Mar 4, 2020

• PR review comment changes

  Niharikadutta

  committed
  5398e44

  Copy full SHA for 5398e44

• PR review comment changes

  Niharikadutta

  committed
  e010162

  Copy full SHA for e010162

Commits on Mar 6, 2020

• Added test case for Broadcast.Unpersist()

  Niharikadutta

  committed
  886e8ec

  Copy full SHA for 886e8ec

Commits on Mar 8, 2020

• working Broadcast.Destroy() functionality

  Niharikadutta

  committed
  04febf8

  Copy full SHA for 04febf8

• Made path as private member as it is used in Destroy()

  Niharikadutta

  committed
  7916c9f

  Copy full SHA for 7916c9f

• Adding summary for TestUnpersist()

  Niharikadutta

  committed
  bf5bce8

  Copy full SHA for bf5bce8

Commits on Mar 9, 2020

• Merge branch 'master' into nidutta/BroadcastVariableSupport

  imback82

  authored
  e7518d5

  Copy full SHA for e7518d5

• PR review update

  Niharikadutta

  committed
  82eae14

  Copy full SHA for 82eae14

• Merge branch 'nidutta/BroadcastVariableSupport' of github.com:Niharikadutta/spark into nidutta/BroadcastVariableSupport

  Niharikadutta

  committed
  5ea8f93

  Copy full SHA for 5ea8f93

• Clearing the JvmBroadcastRegistry after CreatePythonFunction is called.

  Niharikadutta

  committed
  d7ce1da

  Copy full SHA for d7ce1da

Commits on Mar 10, 2020

• Pushing UDF serialize exception repro in Basic

  Niharikadutta

  committed
  8a23912

  Copy full SHA for 8a23912

• Updating TestDestroy() to use try/catch and assert exception

  Niharikadutta

  committed
  a23c000

  Copy full SHA for a23c000

• PR review updates

  Niharikadutta

  committed
  3f51cd3

  Copy full SHA for 3f51cd3

Commits on Mar 11, 2020

• Calling JavaSpark.broadcast() to create the broadcast variable

  Niharikadutta

  committed
  533c534

  Copy full SHA for 533c534

• Testing pipeline stuck state

  Niharikadutta

  committed
  7037ed7

  Copy full SHA for 7037ed7

• Testing stuck pipeline

  Niharikadutta

  committed
  610805a

  Copy full SHA for 610805a

• Testing stuck pipeline 2

  Niharikadutta

  committed
  6cff809

  Copy full SHA for 6cff809

• testing stuck pipeline 3

  Niharikadutta

  committed
  13b05c1

  Copy full SHA for 13b05c1

• testing

  Niharikadutta

  committed
  69d8dc6

  Copy full SHA for 69d8dc6

• Fixing stuck pipeline

  Niharikadutta

  committed
  778eabe

  Copy full SHA for 778eabe

Commits on Mar 12, 2020

• Testing again

  Niharikadutta

  committed
  76fb5c8

  Copy full SHA for 76fb5c8

• Testing pipeline failure by segregating spark tests for v2_3 and v2_4

  Niharikadutta

  committed
  495387a

  Copy full SHA for 495387a

Commits on Mar 21, 2020

• Adding latest changes

  Niharikadutta

  committed
  ca6e273

  Copy full SHA for ca6e273

Commits on Mar 22, 2020

• PR comments

  Niharikadutta

  committed
  c9f1935

  Copy full SHA for c9f1935

• PR comment changes

  Niharikadutta

  committed
  69661c3

  Copy full SHA for 69661c3

• CreateTempFilePath changes

  Niharikadutta

  committed
  5ac3b3b

  Copy full SHA for 5ac3b3b

Commits on Mar 25, 2020

• Merge branch 'master' into nidutta/BroadcastVariableSupport

  imback82

  authored
  22b9a51

  Copy full SHA for 22b9a51

Commits on Mar 26, 2020

• Added guides for using UDFs and broadcast variables.

  Niharikadutta

  committed
  9200745

  Copy full SHA for 9200745

• Merge branch 'nidutta/BroadcastVariableSupport' of github.com:Niharikadutta/spark into nidutta/BroadcastVariableSupport

  Niharikadutta

  committed
  f4f122e

  Copy full SHA for f4f122e

• Adding links to official Spark docs

  Niharikadutta

  committed
  823f57a

  Copy full SHA for 823f57a

• Some formatting changes in guides

  Niharikadutta

  committed
  f5b9eae

  Copy full SHA for f5b9eae

Commits on Mar 30, 2020

• PR review comment changes, and updating the udf-guide

  Niharikadutta

  committed
  f9a537c

  Copy full SHA for f9a537c

Commits on Mar 31, 2020

• Using pythonUtils function to get sparkconf EncryptionEnabled

  Niharikadutta

  committed
  f3eb080

  Copy full SHA for f3eb080

• Removing guides from current PR

  Niharikadutta

  committed
  25a196d

  Copy full SHA for 25a196d

• Reverting change to use PythonUtils function and using SparkConf.Get() to get value of encryption.enabled spark config

  Niharikadutta

  committed
  2b56569

  Copy full SHA for 2b56569

Commits on Apr 2, 2020

• PR review comment changes

  Niharikadutta

  committed
  3bfe978

  Copy full SHA for 3bfe978

• Merge branch 'master' into nidutta/BroadcastVariableSupport

  imback82

  authored
  43dd50c

  Copy full SHA for 43dd50c