Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restrict the max size of each WAL entry to the remaining size of the WAL file #14122

Merged
merged 2 commits into from
Jun 17, 2022
Merged

Restrict the max size of each WAL entry to the remaining size of the WAL file #14122

merged 2 commits into from
Jun 17, 2022

Commits on Jun 16, 2022

  1. Add FileReader and FileBufReader utilities 

    The FileReader interface is the wrapper of io.Reader. It provides
the fs.FileInfo as well. The FileBufReader struct is the wrapper of
bufio.Reader, it also provides fs.FileInfo.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
    @ahrtr
    ahrtr committed Jun 16, 2022
    Configuration menu
    Copy the full SHA
    d23917c View commit details
    Browse the repository at this point in the history

  2. restrict the max size of each WAL entry to the remaining size of the … 

    …file

Currently the max size of each WAL entry is hard coded as 10MB. If users
set a value > 10MB for the flag --max-request-bytes, then etcd may run
into a situation that it successfully processes a big request, but fails
to decode it when replaying the WAL file on startup.

On the other hand, we can't just remove the limitation, because if a
WAL entry is somehow corrupted, and its recByte is a huge value, then
etcd may run out of memory. So the solution is to restrict the max size
of each WAL entry as a dynamic value, which is the remaining size of
the WAL file.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
    @ahrtr
    ahrtr committed Jun 16, 2022
    Configuration menu
    Copy the full SHA
    ccf477d View commit details
    Browse the repository at this point in the history