[SPARK-7315][STREAMING][TEST] Fix flaky WALBackedBlockRDDSuite #5853
Conversation
| // this dummy directory should not already exist otherwise the WAL will try to recover | ||
| // past events from the directory and throw errors. | ||
| val nonExistentDirectory = new File( | ||
| FileUtils.getTempDirectory(), UUID.randomUUID().toString).getAbsolutePath |
There was a problem hiding this comment.
I keep trying to get rid of uses of commons-io since it is not actually a Spark dependency. This can be Utils.createTempDir? The other usage in StreamingContextSuite can be replaced with Guava. Not exactly the topic of this PR but might yet be worth doing if you're potentially removing this usage.
There was a problem hiding this comment.
Actually it cannot be Utils.createTempDir because that actually created the directory, which I dont want it to. But I can very easily replace FileUtils.getTimeDirectory with System.getProperty("java.io.tmpDir")
There was a problem hiding this comment.
Got it. You can delete the result too but of course this is fine.
There was a problem hiding this comment.
Yeah, I thought the same. But then why do an additional file creation and deletion, especially when this is going to run potentially 10s of times every second.
|
Test build #31629 has finished for PR 5853 at commit
|
|
Test build #31636 has finished for PR 5853 at commit
|
`FileUtils.getTempDirectoryPath()` path may or may not exist. We want to make sure that it does not exist. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes apache#5853 from tdas/SPARK-7315 and squashes the following commits: 141afd5 [Tathagata Das] Removed use of FileUtils b08d4f1 [Tathagata Das] Fix flaky WALBackedBlockRDDSuite
`FileUtils.getTempDirectoryPath()` path may or may not exist. We want to make sure that it does not exist. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes apache#5853 from tdas/SPARK-7315 and squashes the following commits: 141afd5 [Tathagata Das] Removed use of FileUtils b08d4f1 [Tathagata Das] Fix flaky WALBackedBlockRDDSuite
`FileUtils.getTempDirectoryPath()` path may or may not exist. We want to make sure that it does not exist. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes apache#5853 from tdas/SPARK-7315 and squashes the following commits: 141afd5 [Tathagata Das] Removed use of FileUtils b08d4f1 [Tathagata Das] Fix flaky WALBackedBlockRDDSuite
FileUtils.getTempDirectoryPath()path may or may not exist. We want to make sure that it does not exist.