Skip to content
This repository has been archived by the owner on Jul 7, 2020. It is now read-only.

Initial implementation of disk backed queue. #2

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

mspiegel
Copy link

Thread-safe FIFO queue that uses a muxy filesystem to
write overflow elements to disk. If memory capacity is not exceeded
then all operations occur in memory. {@link #put(Object)}
requests are designed to return quickly and write to disk
asynchronously if one or more background threads have been configured.
All requests of the {@link DiskBackedQueueInternals#get(long, TimeUnit)}
family perform synchronous reads from the disk. A {@link #close()}
will write the contents of the queue to the muxy filesystem.

Nearly all parameter are required to the Builder object.
This is a deliberate design decision to discourage a reliance of magical
default values that lead to behavior surprises for your application.

Single threaded unit tests are passing. Multithreaded test is failing
on IOException for uninitialized muxy stream. I am probably doing something
silly that is easily fixed.

Two changes to core muxy implementation. (1) Added MuxFileDirectory#sync()
method. Written with same concurrency protections as WritableMuxFile#sync()
(which are none) so presumed to be safe. (2) MuxyEventListener is described
as optional in the Javadoc but a few places assumed it was always defined.
Fixed those instances.

@mspiegel mspiegel force-pushed the spiegel/disk-backed-queue branch 3 times, most recently from db1b51d to 6a3c77d Compare March 31, 2015 13:53
Thread-safe FIFO queue that uses a muxy filesystem to
write overflow elements to disk. If memory capacity is not exceeded
then all operations occur in memory. {@link #put(Object)}
requests are designed to return quickly and write to disk
asynchronously if one or more background threads have been configured.
All requests of the {@link DiskBackedQueueInternals#get(long, TimeUnit)}
family perform synchronous reads from the disk. A {@link #close()}
will write the contents of the queue to the muxy filesystem.

Nearly all parameter are required to the Builder object.
This is a deliberate design decision to discourage a reliance of magical
default values that lead to behavior surprises for your application.

Single threaded unit tests are passing. Multithreaded test is failing
on IOException for uninitialized muxy stream. I am probably doing something
silly that is easily fixed.

Two changes to core muxy implementation. (1) Added MuxFileDirectory#sync()
method. Written with same concurrency protections as WritableMuxFile#sync()
(which are none) so presumed to be safe. (2) MuxyEventListener is described
as optional in the Javadoc but a few places assumed it was always defined.
Fixed those instances.
@mspiegel mspiegel force-pushed the spiegel/disk-backed-queue branch from 6a3c77d to 463e323 Compare April 14, 2015 20:33
Add a workaround to have the unit tests partially work.
The tests pass only if each file write is followed by
a sync() operation.
@tea-dragon
Copy link
Contributor

this is a lot to review at once, is it possible that this could be broken up into a few different sub changes?

@tea-dragon
Copy link
Contributor

that aside, please move out all the inner classes, and keep fields adjacent to each other rather than spread throughout the class.

@mspiegel
Copy link
Author

The multithreaded unit tests are only passing if a sync() is called after every write. Uncommenting https://github.com/addthis/muxy/pull/2/files#diff-12d90acbab2efef07369c1c6f30506adR179 will fail the test.

@mspiegel mspiegel force-pushed the spiegel/disk-backed-queue branch from 02618b4 to 3b817e3 Compare April 21, 2015 19:49
Pulling out inner class Page into its own class.
Performance tests are next on the TODO list.
@mspiegel mspiegel force-pushed the spiegel/disk-backed-queue branch from 3b817e3 to f09881c Compare April 21, 2015 19:50
Michael Spiegel added 3 commits April 26, 2015 15:48
Copy pages to temporary streams before reading and writing to
external storage. Incurs the overhead of an extra copy of the memory
contents. Benefit is reduced time spend holding the lock on the
external storage. In synthetic benchmarks overall performance is improved.
Also cleanup a unit test where termination condition was incorrectly
specified.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants