Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CookieStoreTest.testConcurrentLoad fails randomly #274

Closed
ato opened this issue Aug 2, 2019 · 0 comments · Fixed by #280
Closed

CookieStoreTest.testConcurrentLoad fails randomly #274

ato opened this issue Aug 2, 2019 · 0 comments · Fixed by #280
Labels

Comments

@ato
Copy link
Collaborator

ato commented Aug 2, 2019

Just hit this with a Travis on #271. It looks like someone else also hit when building from source in #175.

testConcurrentLoad(org.archive.modules.fetcher.CookieStoreTest)  Time elapsed: 1.188 sec  <<< FAILURE!
junit.framework.AssertionFailedError
	at junit.framework.Assert.fail(Assert.java:47)
	at junit.framework.Assert.assertTrue(Assert.java:20)
	at junit.framework.Assert.assertTrue(Assert.java:27)
	at org.archive.modules.fetcher.CookieStoreTest.testConcurrentLoad(CookieStoreTest.java:391)

The problem seems to have been introduced by #133:

I don't think we need the synchronization in fact. Because the 50 cookie limit doesn't have to be completely strict. It's just a cutoff to avoid problems, and if a domain has 51 cookies, or even 60, it's unlikely to be a problem. So my suggestion is to remove the synchronization, and change CookieStoreTest:350 to something like assertTrue(domainCounts.get(domain) <= BdbCookieStore.MAX_COOKIES_FOR_DOMAIN + 5);, i.e. check that it's not way over the limit.

Unfortunately I don't think this assumption is safe. Not just because it can cause the tests to randomly fail either. At the NLA we've actually hit a gotcha in several different applications where Java's HashMap goes into an infinite loop when accessed concurrently without synchronization.

https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6423457

If a HashMap is used in a concurrent setting with insufficient synchronization,
it is possible for the data structure to get corrupted in such a way that
infinite loops appear in the data structure and thus get() could loop forever.

Edit: Confused myself reading the code. The HashMap is not accessed in multiple threads.

@ato ato added the bug label Aug 2, 2019
ato added a commit that referenced this issue Aug 12, 2019
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.

I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.

Fixes #274
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant