[batch] Don't rmtree if any errors occur while unmounting #12985
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
It would be great in a separate PR to check for sure there aren't any zombie fuse mounts before removing files. |
danking
previously requested changes
May 4, 2023
batch/batch/worker/worker.py
Outdated
| except asyncio.CancelledError: | ||
| raise | ||
| except Exception: | ||
| log.exception(f'error while unmounting cloudfuse for {self.jvm_name} for job {self.id}') |
There was a problem hiding this comment.
Let's include the same information we have above:
Suggested change
| log.exception(f'error while unmounting cloudfuse for {self.jvm_name} for job {self.id}') | |
| log.exception( | |
| f'while unmounting fuse blob storage {bucket} from {mount_path} for {self.jvm_name} for job {self.id}' | |
| ) |
batch/batch/worker/worker.py
Outdated
| except IncompleteJVMCleanupError: | ||
| assert isinstance(job, JVMJob) | ||
| await self.recreate_jvm(job.jvm) | ||
| log.exception(f'while running {job}, incomplete cleanup from {job.jvm}') |
There was a problem hiding this comment.
This results in two exceptional logs for a single exception. Let's either not log here or not log near the raise.
jigold
force-pushed
the
no-rmtree-wout-unmount-v2
branch
from
57703d3
to
86f266a
Compare
danking
reviewed
May 4, 2023
batch/batch/worker/worker.py
Outdated
| raise IncompleteJVMCleanupError | ||
| raise IncompleteJVMCleanupError( | ||
| f'while unmounting fuse blob storage {bucket} from {mount_path} for {self.jvm_name} for job {self.id}' | ||
| ) |
There was a problem hiding this comment.
Name the exception and add a from e so that we get the fulls tack trace in the log.exception later.
danking
previously requested changes
May 4, 2023
batch/batch/worker/worker.py
Outdated
| @@ -2312,6 +2320,11 @@ class JVMCreationError(Exception): | |||
| pass | |||
|
|
|||
|
|
|||
| class IncompleteJVMCleanupError(Exception): | |||
| def __init__(self, msg): | |||
| self.msg = msg | |||
There was a problem hiding this comment.
This isn't necessary:
In [1]: class IncompleteJVMCleanupError(Exception):
...: pass
...:
In [2]: raise IncompleteJVMCleanupError('foo')
---------------------------------------------------------------------------
IncompleteJVMCleanupError Traceback (most recent call last)
<ipython-input-2-a8a0c94b0ce4> in <cell line: 1>()
----> 1 raise IncompleteJVMCleanupError('foo')
IncompleteJVMCleanupError: foo
|
These changes look great |
jigold
commented
May 4, 2023
| assert isinstance(job, JVMJob) | ||
| await self.recreate_jvm(job.jvm) | ||
| log.exception(e.msg) | ||
| log.exception(f'while running {job}, ignoring') |
There was a problem hiding this comment.
@danking Can you double check the actual exception message will appear in the logs here?
There was a problem hiding this comment.
Yeah, log.exception does this.
danking
approved these changes
May 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I think this change is better than #12975