[qob] prevent race to write to log file #13729
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This finally block is currently located after `gather.take()` but *before* we cancel (aka shutdown) all threads. As a result, it is possible for us to shudown the logging (thus flushing, closing, and destorying old appenders) then restart logging (thus opening the file in overwrite mode) and blow away whatever was there. I have verified in my namespace across 10s of thosuands of JVM Jobs that this never blows away the log.
jigold
approved these changes
Sep 27, 2023
danking
added a commit
that referenced
this pull request
Oct 16, 2023
…13715) CHANGELOG: Fixes #13697, a long standing issue with QoB, in which a failing partition job or driver job is not failed in the Batch UI. I am not sure why we did not do this this way in the first place. If a JVMJob raises an exception, Batch will mark the job as failed. Ergo, we should raise an exception when a driver or a worker fails! Here's an example: I used a simple pipeline that write to a bucket to which I have read-only access. You can see an example Batch (where every partition fails): https://batch.hail.is/batches/8046901. [1] ```python3 import hail as hl hl.utils.range_table(3, n_partitions=3).write('gs://neale-bge/foo.ht') ``` NB: I removed the `log.error` in `handleForPython` because that log is never necessary. That function converts a stack of exceptions into a triplet of the short message, the full exception with stack trace, and a Hail error id (if present). That triplet is always passed along to someone else who logs the exception. (FWIW, the error id indicates a Python source location that is associated with the error. On the Python-side, we can look up that error id and provide a better stack trace.) [1] You'll notice the logs are missing. I noticed this as well, it's a new bug. I fixed it in #13729.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #13716
This finally block is currently located after
gather.take()but before we cancel (aka shutdown) all threads. As a result, it is possible for us to shudown the logging (thus flushing, closing, and destorying old appenders) then restart logging (thus opening the file in overwrite mode) and blow away whatever was there.I have verified in my namespace across 10s of thosuands of JVM Jobs that this never blows away the log.