[SPARK-3943] Some scripts bin\*.cmd pollutes environment variables in Windows

[tsudukim]

Modified not to pollute environment variables.
Just moved the main logic into XXX2.cmd from XXX.cmd, and call XXX2.cmd with cmd command in XXX.cmd.
pyspark.cmd and spark-class.cmd are already using the same way, but spark-shell.cmd, spark-submit.cmd and /python/docs/make.bat are not.

[tsudukim]

Please merge this AFTER #2796 is merged, because /python/docs/make2.bat will be ignored by .gitignore in /python by mistake.

[AmplabJenkins]

Can one of the admins verify this patch?

[andrewor14]

ok to test

[andrewor14]

Is the purpose of this file such that exporting SPARK_HOME here doesn't keep it in the system environment even after the application exits?

[tsudukim]

Yes it is.

Although this script now has only one environment variable SPARK_HOME, it might be more complicated in future just like bin/spark-shell (linux version), then it might export other environment variables.

[andrewor14]

I haven't done a close review to make sure this doesn't break anything yet, but the general idea looks good to me. Just so I understand correctly, is the purpose of this to ensure we don't accidentally read environment variables from previous applications that have now exited? Were you observing any strange behavior, or is this just a safeguard for the future?

[SparkQA]

QA tests have started for PR 2797 at commit b397a7d.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2797 at commit b397a7d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21737/
Test PASSed.

[tsudukim]

@andrewor14 thank you for following this PR.
Yes that's I mean.
I'm not observing any problems, this is just a safeguard. Polluting environment might affect noy only spark scripts but also non-spark batch scripts and applications.

[andrewor14]

I see. This LGTM actually. I just did a closer look at the scripts, and it appears that we already do the same for older ones (pyspark.cmd, spark-class.cmd etc.). I'm merging this into master.