Add support for spark-submit packages and repositories #1040

mdelaney · 2020-10-09T22:12:25Z

added support for packages, exclude-packages, and repositories
spark-submit parameters
added these new parameters to the user guide documentation

google-cla · 2020-10-09T22:15:41Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

- added support for packages, exclude-packages, and repositories spark-submit parameters - added these new parameters to the user guide documentation

mdelaney · 2020-10-09T22:22:59Z

@googlebot I signed it!

google-cla · 2020-10-09T22:23:34Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

liyinan926 · 2020-10-09T22:21:58Z

pkg/apis/sparkoperator.k8s.io/v1beta2/types.go

+	// +optional
+	ExcludePackages []string `json:"excludePackages,omitempty"`
+	// Repositories is a list of additional remote repositories to search for the maven coordinate
+	// given with the "packages" option


Nit: missing period at the end.

liyinan926 · 2020-10-09T22:23:50Z

pkg/controller/sparkapplication/submission.go

+	}
+	if len(app.Spec.Deps.ExcludePackages) > 0 {
+		depsConfOptions = append(depsConfOptions,
+			"--exclude-packages", strings.Join(app.Spec.Deps.ExcludePackages, ","))


Nit: there's no need to start a new line here and below.

googlebot · 2020-10-09T22:24:59Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

google-cla · 2020-10-09T22:27:16Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

google-cla · 2020-10-09T22:33:15Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

mdelaney · 2020-10-09T22:33:58Z

Addressed comments, thanks for the super fast review! :-)

liyinan926 · 2020-10-09T22:44:34Z

pkg/apis/sparkoperator.k8s.io/v1beta2/types.go

+	// Packages is a list of maven coordinates of jars to include on the driver and executor
+	// classpaths. This will search the local maven repo, then maven central and any additional
+	// remote repositories given by the "repositories" option.
+	// Each papckage should be of the form "groupId:artifactId:version"


Nit: missing period at the end.

liyinan926 · 2020-10-09T22:44:53Z

pkg/apis/sparkoperator.k8s.io/v1beta2/types.go

+	// Each papckage should be of the form "groupId:artifactId:version"
+	// +optional
+	Packages []string `json:"packages,omitempty"`
+	// ExcludePackages is a list of groupId:artifactId, to exclude while resolving the


Double quotes around groupId:artifactId.

google-cla · 2020-10-09T22:54:10Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

* Add support for spark-submit packages and repositories - added support for packages, exclude-packages, and repositories spark-submit parameters - added these new parameters to the user guide documentation * address review comments * address review comments

Add support for spark-submit packages and repositories

eb4038d

- added support for packages, exclude-packages, and repositories spark-submit parameters - added these new parameters to the user guide documentation

liyinan926 reviewed Oct 9, 2020

View reviewed changes

mdelaney force-pushed the add-package-and-repo-support branch from 42ab867 to eb4038d Compare October 9, 2020 22:26

address review comments

4cab73f

liyinan926 approved these changes Oct 9, 2020

View reviewed changes

address review comments

7589fb9

liyinan926 merged commit cf34460 into kubeflow:master Oct 10, 2020

mdelaney deleted the add-package-and-repo-support branch October 12, 2020 18:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for spark-submit packages and repositories #1040

Add support for spark-submit packages and repositories #1040

mdelaney commented Oct 9, 2020

google-cla bot commented Oct 9, 2020

mdelaney commented Oct 9, 2020

google-cla bot commented Oct 9, 2020

liyinan926 Oct 9, 2020

liyinan926 Oct 9, 2020

googlebot commented Oct 9, 2020

google-cla bot commented Oct 9, 2020

google-cla bot commented Oct 9, 2020

mdelaney commented Oct 9, 2020

liyinan926 Oct 9, 2020

liyinan926 Oct 9, 2020

google-cla bot commented Oct 9, 2020

Add support for spark-submit packages and repositories #1040

Add support for spark-submit packages and repositories #1040

Conversation

mdelaney commented Oct 9, 2020

google-cla bot commented Oct 9, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

mdelaney commented Oct 9, 2020

google-cla bot commented Oct 9, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

liyinan926 Oct 9, 2020

Choose a reason for hiding this comment

liyinan926 Oct 9, 2020

Choose a reason for hiding this comment

googlebot commented Oct 9, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

google-cla bot commented Oct 9, 2020

google-cla bot commented Oct 9, 2020

mdelaney commented Oct 9, 2020

liyinan926 Oct 9, 2020

Choose a reason for hiding this comment

liyinan926 Oct 9, 2020

Choose a reason for hiding this comment

google-cla bot commented Oct 9, 2020