fix s3 file cache sharing temp folder across processes #1650
Conversation
ghost
self-requested a review
Codecov Report
@@ Coverage Diff @@
## master #1650 +/- ##
==========================================
- Coverage 77.83% 77.81% -0.02%
==========================================
Files 232 232
Lines 18209 18219 +10
==========================================
+ Hits 14173 14178 +5
- Misses 4036 4041 +5
Continue to review full report at Codecov.
|
| Logger.error("Reducer Error: #{filename}:#{File.size(filename)} bytes: \n#{e.formatted}") | ||
| else | ||
| Logger.error("Reducer Error: #{filename}:(Does Not Exist): \n#{e.formatted}") | ||
| end |
There was a problem hiding this comment.
Is this what you expect is happening with the reducer error?
There was a problem hiding this comment.
Not sure what this comment means, but this error code is buggy and I need to fix it.
| FileUtils.mkdir_p(@cache_dir) | ||
| at_exit do | ||
| FileUtils.remove_dir(@cache_dir, true) | ||
| end |
There was a problem hiding this comment.
I assume the at_exit applies to the Thread you create below?
There was a problem hiding this comment.
At exit applies to the entire ruby process.
| FileUtils.mkdir_p(@cache_dir) | ||
| at_exit do | ||
| FileUtils.remove_dir(@cache_dir, true) | ||
| end | ||
|
|
||
| # Clear out local file cache | ||
| FileUtils.rm_f Dir.glob("#{@cache_dir}/*") |
There was a problem hiding this comment.
Probably don't need to do this if you're dynamically creating a new dir above
| file.retrieve | ||
| rescue | ||
| # Will be automatically retried | ||
| end |
There was a problem hiding this comment.
What's the point of raising the error and then silently rescuing and retrying?
There was a problem hiding this comment.
The error didn't use to be there, but I added it for when using retrieve by itself
close #1633