Harden Sharness #288
Harden Sharness #288
Conversation
Better error messages in test-lib init functions and ipfshttp Cleanup functions are now called after every test file License: MIT Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
ghost
assigned 
| @@ -31,4 +29,7 @@ test_expect_success "starting a second cluster-service process fails" ' | |||
| test_expect_code 1 ipfs-cluster-service --config "test-config" | |||
| ' | |||
|
|
|||
| test_clean_ipfs | |||
There was a problem hiding this comment.
Hmm it seems that cleanup ... does not clean when the tests have failed (even though documentation says it does? https://github.com/chriscool/sharness/blob/master/API.md#final_cleanup).
I understand this fixes that? The reason clean up does not run on that case is probably to allow the user to debug things. I'm fine with this though, since debugging implies running stuff locally and we can then potentially comment this.
There was a problem hiding this comment.
Yes. It was tough to see this because as you say the docs are a bit misleading. But it explained behavior we saw and I'm reasonably confident after looking at sharness source (see link above)
My thoughts exactly. It makes sense if you want a single failure to stop everything so you can debug, but running this on CI we don't even have a chance to investigate the left over state.
|
merge if you want @ZenGround0 |
|
@hsanjuan Will do. I'll leave the harden sharness issue open as it sounds like you have more in mind. If you are looking to offload that work make sure to write down your thoughts on the issue because right now I'm drawing a blank on what else needs to be done. |
Recent sharness failures prompted more investigation. In particular sharness
cleanupdoes not run cleanup functions on failure (see here: https://github.com/chriscool/sharness/blob/master/sharness.sh#L763). To ensure isolation among test failures we need reliable teardown of ipfs and cluster between each testfile so now the cleanup functions are run directly before test_done.Error messages are also slightly improved.
The current
test-lib.shalready ensures that there is no data folder withrm -rf ipfs-cluster-dataafter the config folder is created.Altogether this should close #181.