Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logger: fix rare case where logging stops #12679

Merged
merged 1 commit into from
Aug 12, 2019
Merged

Conversation

bkueng
Copy link
Member

@bkueng bkueng commented Aug 10, 2019

The file writer thread could get into a state where it blocked infinitely
on pthread_cond_wait() (or rather until the logging stops).

This is very rare and the following conditions must be met:

  • the buffer is almost empty (<4KB filled), so that the writer thread does
    not write anything.
  • an fsync call is scheduled (happens once every second)
  • the fsync call takes a long time (several 100ms), during which time the
    complete log buffer fills up.

The main thread would then get into dropout state where it does not call
_writer.notify() anymore.

Notifying the writer thread on every loop update of the main thread fixes
that.

It does not increase resource usage.

This can go to stable as well.

Fixes #12565

The file writer thread could get into a state where it blocked infinitely
on pthread_cond_wait() (or rather until the logging stops).

This is very rare and the following conditions must be met:
- the buffer is almost empty (<4KB filled), so that the writer thread does
  not write anything.
- an fsync call is scheduled (happens once every second)
- the fsync call takes a long time (several 100ms), during which time the
  complete log buffer fills up.

The main thread would then get into dropout state where it does not call
_writer.notify() anymore.

Notifying the writer thread on every loop update of the main thread fixes
that.

It does not increase resource usage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Logger stopped completely during flight?
1 participant