Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle a graceful shutdown on request #594

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kathia-barahona
Copy link
Contributor

@kathia-barahona kathia-barahona commented Jul 4, 2023

About this change - What it does

Handles graceful shutdowns by making sure all compressed files get uploaded (considers also partial files)
Edit: renaming partial files is dangerous, after testing I didn't got the expected results (duplicated segments on different timelines)

Why this way

Shutting down pghoard can be dangerous, leading to missed segments if not all information was uploaded. To avoid this, its better to make sure all tasks/events were completed.

@kathia-barahona kathia-barahona self-assigned this Jul 4, 2023
# wait for all queues to be emptied
self._wait_for_queue_to_be_emptied(self.compression_queue, "compression")
self._wait_for_queue_to_be_emptied(self.transfer_queue, "transfer")
self._wait_for_queue_to_be_emptied(self.wal_file_deletion_queue, "wal_file_deletion")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment wal_file_deletion_queue is not super high prio, but makes sure we do not re-compress or re-upload files when pghoard is restarted (startup_walk)

@kathia-barahona kathia-barahona marked this pull request as ready for review July 5, 2023 09:20
@kathia-barahona kathia-barahona requested review from a team as code owners July 5, 2023 09:20
@kathia-barahona
Copy link
Contributor Author

:/ Build failing, seems rohmu got updated again (using Protocol, not supported on 3.7)

@Mulugruntz
Copy link
Contributor

:/ Build failing, seems rohmu got updated again (using Protocol, not supported on 3.7)

To be noted that Python 3.7 reached EOL a week ago.
That raises the question on whether we want to support a Python version on pghoard that is no longer officially supported and will no longer receive security patches.
I'd be in favour of removing 3.7 from the CI here.

test/test_pghoard.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants