-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consolidations of shuffle files from different map tasks #635
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -76,12 +76,12 @@ private class DiskStore(blockManager: BlockManager, rootDirs: String) | |
// Revert by discarding current writes, except that if no values have been committed, | ||
// we revert by recreate the file (otherwise there are errors when reading objects from the file later on | ||
if (lastValidPosition == initialPosition) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. initialPosition is set to the size of the file when the file is opened. Isn't it problematic if we open an existing file, did some writes and then want to revert, we could delete the old file? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A shuffle file is closed when we get a shuffle request, and it should never be re-opened (and if you re-open the file and append it, you may corrupt the file as it may have a tailer). Maybe we should first delete the shuffle file if it exits when we first open it (e.g., in case we need to re-run the map tasks). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But if we re-run the map tasks, wouldn't that wipe out the shuffle outputs for all other previous map tasks? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The shuffle file is closed when all the map tasks are done (after the shuffle request is received). If we need to re-run a map task afterwards, I actually also tried throwing random exceptions during shuffle and it works fine. Again, is there a good way to include such tests? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the response. It would be useful to actually put whatever you just wrote into the code comment block. For failure testing, can we do something similar to what FailureSutie does? |
||
recerateFile() | ||
recreateFile() | ||
else | ||
discardWrites() | ||
} | ||
|
||
private def recerateFile () { | ||
private def recreateFile () { | ||
close () | ||
f.delete() | ||
f = createFile(blockId) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a comment on what the Int and Long represent.