Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add parameter for limiting maximum AOF files size on disk #1630

Open
wants to merge 3 commits into
base: unstable
Choose a base branch
from

Conversation

kronwerk
Copy link
Contributor

@kronwerk kronwerk commented Jan 27, 2025

[duplicate of this]

one can optionally want to limit AOF disk usage not with disk real size, but with some tailored value. this PR makes it possible.
possible use cases:

  1. filling disk up with AOF to the real limit may break something in user's environment (logs, watchdogs, anything) - setting lower value prevents that completely;
  2. setting this new limit to some low value may allow automatic solving of filling disk problem for those users who don't care too much about disk cost:
  • intense writes may overcome automatic AOF rewrite with some user's settings at some moment;
  • AOF reach lower than full disk limit (e.g. 50-60% of disk);
  • eventually automatic AOF rewrite comes in once more and finally fixes that.
    while on 100% filled disk (like nowadays) that automatic solution is impossible due to insufficient space for tmp AOF rewrite files (0 space, actually) - one should size up disk under running database which makes some people a bit nervous, as I noticed, and in some cases that's not always convenient or even possible.

Fixes #540

Signed-off-by: kronwerk <kronwerk@users.noreply.github.com>

improved aof-max-size tests

Signed-off-by: kronwerk <kronwerk@users.noreply.github.com>
Signed-off-by: kronwerk <kronwerk@users.noreply.github.com>
Signed-off-by: kronwerk <kronwerk@users.noreply.github.com>
Copy link

codecov bot commented Jan 27, 2025

Codecov Report

Attention: Patch coverage is 87.50000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 70.99%. Comparing base (a18fcdb) to head (2954eb0).
Report is 2 commits behind head on unstable.

Files with missing lines Patch % Lines
src/aof.c 87.50% 1 Missing ⚠️
src/server.c 87.50% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #1630      +/-   ##
============================================
+ Coverage     70.84%   70.99%   +0.14%     
============================================
  Files           121      121              
  Lines         65169    65180      +11     
============================================
+ Hits          46172    46274     +102     
+ Misses        18997    18906      -91     
Files with missing lines Coverage Δ
src/config.c 78.39% <ø> (ø)
src/script.c 87.69% <ø> (ø)
src/server.h 100.00% <ø> (ø)
src/aof.c 81.37% <87.50%> (+1.13%) ⬆️
src/server.c 87.64% <87.50%> (+0.02%) ⬆️

... and 11 files with indirect coverage changes

@kronwerk
Copy link
Contributor Author

@xbasel @soloestoy pls take a look, I've dumped my previous similar PR where you were reviewers

* there is an actual error condition we'll get it at the next try.
* We also check for aof-max-size limit here returning custom error on exceed. */
ssize_t aofWrite(int fd, const char *buf, size_t len, off_t aof_current_size, unsigned long long aof_max_size) {
ssize_t nwritten = 0, totwritten = 0, nonewritten = -1;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like nonewritten should be a constant/literal.
Perhaps directly returning -1/C_ERR instead of using an extra variable.

ssize_t nwritten = 0, totwritten = 0;
* there is an actual error condition we'll get it at the next try.
* We also check for aof-max-size limit here returning custom error on exceed. */
ssize_t aofWrite(int fd, const char *buf, size_t len, off_t aof_current_size, unsigned long long aof_max_size) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aof_max_size -> size_t

ssize_t nwritten = 0, totwritten = 0;
* there is an actual error condition we'll get it at the next try.
* We also check for aof-max-size limit here returning custom error on exceed. */
ssize_t aofWrite(int fd, const char *buf, size_t len, off_t aof_current_size, unsigned long long aof_max_size) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not related to this change, but do you have any idea why aof_current_size (and some other aof-related variables) are already defined as off_t instead of size_t?

@@ -4627,13 +4627,17 @@ int writeCommandsDeniedByDiskError(void) {
return DISK_ERROR_TYPE_NONE;
}

char *getAofWriteErrStr(int error_code) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this function is strictly for AOF-related and always uses server.aof_last_write_errno, passing error_code is misleading. I'd remove the parameter and read server.aof_last_write_errno directly to avoid confusion and unintended wider use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[NEW] Limit maximum size on disk of AOF files. Avoid disk full, long load times.
2 participants