[SPARK-3140] Clarify confusing PySpark exception message #2067
Conversation
| @@ -54,12 +54,18 @@ def preexec_func(): | |||
| gateway_port = proc.stdout.readline() | |||
| gateway_port = int(gateway_port) | |||
| except ValueError: | |||
| # Grab the remaining lines of stdout | |||
| (stdout, _) = proc.communicate() | |||
| exit_code = proc.poll() | |||
| error_msg = "Launching GatewayServer failed" | |||
| error_msg += " with exit code %d! " % exit_code if exit_code else "! " | |||
There was a problem hiding this comment.
unrelated to your change but can you fix this too while you're at it? Looks like there's an extra "!" when there's no exit code
There was a problem hiding this comment.
wait, when there's no exit code it looks like this, no?
Launching GatewayServer failed! (Warning: ...)
There was a problem hiding this comment.
OH sorry misread this -- thought the "if" was for what to substitute. NVM.
There was a problem hiding this comment.
Yeah, I misread it too the first time after reading your comment :)
|
This is great...there's been quite a lot of pain associated with these lines of code and I think this will help a lot. |
|
With the latest commit, this is what it looks like with extra output: and no output: |
andrewor14
force-pushed
the
python-exception
branch
from
1c63d82 to
742f823
Compare
|
Cool this looks great! |
|
This looks great. Surprised Jenkins hasn't chimed in to test this, but I don't think it would (intentionally) exercise this code path anyways, so I think it's safe to merge this since you've tested it locally. |
|
Given the number of accidental-build-breaks that have happened recently, I think it would be good to let Jenkins have at this unless there's a rush to get it in |
|
Jenkins, test this please |
|
QA tests have started for PR 2067 at commit
|
|
QA tests have finished for PR 2067 at commit
|
|
Thanks, merged into master and 1.1 |
We read the py4j port from the stdout of the `bin/spark-submit` subprocess. If there is interference in stdout (e.g. a random echo in `spark-submit`), we throw an exception with a warning message. We do not, however, distinguish between this case from the case where no stdout is produced at all. I wasted a non-trivial amount of time being baffled by this exception in search of places where I print random whitespace (in vain, of course). A clearer exception message that distinguishes between these cases will prevent similar headaches that I have gone through. Author: Andrew Or <andrewor14@gmail.com> Closes #2067 from andrewor14/python-exception and squashes the following commits: 742f823 [Andrew Or] Further clarify warning messages e96a7a0 [Andrew Or] Distinguish between unexpected output and no output at all (cherry picked from commit ba3c730) Signed-off-by: Andrew Or <andrewor14@gmail.com>
We read the py4j port from the stdout of the `bin/spark-submit` subprocess. If there is interference in stdout (e.g. a random echo in `spark-submit`), we throw an exception with a warning message. We do not, however, distinguish between this case from the case where no stdout is produced at all. I wasted a non-trivial amount of time being baffled by this exception in search of places where I print random whitespace (in vain, of course). A clearer exception message that distinguishes between these cases will prevent similar headaches that I have gone through. Author: Andrew Or <andrewor14@gmail.com> Closes apache#2067 from andrewor14/python-exception and squashes the following commits: 742f823 [Andrew Or] Further clarify warning messages e96a7a0 [Andrew Or] Distinguish between unexpected output and no output at all
We read the py4j port from the stdout of the
bin/spark-submitsubprocess. If there is interference in stdout (e.g. a random echo inspark-submit), we throw an exception with a warning message. We do not, however, distinguish between this case from the case where no stdout is produced at all.I wasted a non-trivial amount of time being baffled by this exception in search of places where I print random whitespace (in vain, of course). A clearer exception message that distinguishes between these cases will prevent similar headaches that I have gone through.