Add queue overflow handler in asyncsender.

[pguz]

Hi,

In order to guarantee that the logging process won't block under any circumstance the app, queue_circular flag can be used. In that case it can cause loss of events if the queue fills up completely.

However sometimes there is a need to handle such cases in some way. That's why I propose adding queue_overflow_handler parameter to asyncsender.

Please consider my pull request.

Cheers!

[coveralls]

Coverage remained the same at 100.0% when pulling 478bd02 on pguz:queue_overflow_handler into eb68481 on fluent:master.

[coveralls]

Coverage decreased (-0.7%) to 99.191% when pulling 50119c5 on pguz:queue_overflow_handler into a37f313 on fluent:master.

[arcivanov]

Please add tests to cover and see comments inline.

[arcivanov]

I don't like checks like that in the critical paths, as it's going to be repeated every call.
Can we push it up into the constructor like we do here https://github.com/fluent/fluent-logger-python/blob/master/fluent/handler.py#L86?

[arcivanov]

We don't support Python 2 anymore, there is no need for these changes.

UPD: Nevermind, we still do. :)

[pguz]

I have just added improved test.

[arcivanov]

I'm travelling and will try to get to this tonight.

[pguz]

up

[arcivanov]

This particular line concerns me a bit. If self._queue_overflow_handler raises an Empty, it's going to go into pass too.

[pguz]

I have just resolved it. I hope it does a job.

[arcivanov]

@pguz sorry, it was Pending review for two weeks now, because I always forget to actually submit the review :(

[pguz]

Locally I am not receiving these errors. Is it testing environment issue?

[arcivanov]

Nope, that's a dependency issue, i.e. your local testing environment issue :)

[jgoclawski]

Hi @arcivanov, this PR looks like a nice addition. Would you have any hints as to why one test fails only o Python 2.7 on Travis? What can be done to move this forward?

FAIL: test_simple (tests.test_asynchandler.TestHandlerWithCircularQueueHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/fluent/fluent-logger-python/tests/test_asynchandler.py", line 371, in test_simple
    log.info({'cnt': 2, 'from': 'userA', 'to': 'userB'})
AssertionError: QueueOverflowException not raised

[pguz]

Hi, I fixed this issue writing more defensive test. QueueOverflowException not raised error was not caused by python2.7 itself, but rather multithreading issues. We cannot be sure that QueueOverflowException is always thrown, that's why in the last commit I assumed that this exception may not be thrown, however I wrapped in print statement number of times this exception is excepted to be thrown.

print('Exception raised: {} (expected 3)'.format(exc_counter))

We can see that in newer series of tests for python 2.7 tests are passing and even the condition which was not passed last time is passing now. However this time the problematic condition would not pass pypy3.6-7.1.1 test. However due to defensive nature of tests pypy3.6-7.1.1 tests are passing. I am sure that running tests one more time for pypy3.6-7.1.1 tests would pass the problematic condition. It is all about multithreading.

[pguz]

up