Copy WAL frames through temp file to shadow #474
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implementation replicates what the new behavior in the legacy branch did using a temporary file instead of memory to buffer WAL frames between commits.
Fixes excess memory use if the WAL includes massive commits due to big inserts or migrations.
A small refactor would remove the requirement of a temporary file that was already part of the legacy branch where we keep the cached valid position of the shadow WAL in the DB struct and seek the real WAL to the last valid position instead of expecting the file to be completely intact which isn't guaranteed. After that we can directly write the pages from real WAL to shadow WAL and not worry about partial pages being treated as part of the WAL.