Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CDC] Reset WAL retention time when a table doesn't have any more CDC streams #2159

Open
hectorgcr opened this issue Aug 26, 2019 · 3 comments
Assignees
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue

Comments

@hectorgcr
Copy link
Contributor

hectorgcr commented Aug 26, 2019

Jira Link: DB-1895
When we create a CDC stream, we are modifying the WAL retention time for all the tablets in a table, but we need to revert this once a table doesn't have a stream.

The default WAL retention time is 5 minutes when there is no CDC stream, and 4 hours when there is a CDC stream, so we should avoid keeping logs for 4 hours when they are not needed (no CDC stream on the table).

This is related to #2050

@hectorgcr hectorgcr self-assigned this Aug 26, 2019
@hectorgcr hectorgcr added the area/cdc Change Data Capture label Aug 26, 2019
@hectorgcr
Copy link
Contributor Author

hectorgcr commented Sep 29, 2020

Since we now support log retention by min replicated id, we should remove this automatic increase of the log retention time. The only thing I need to confirm is that as soon as a stream is created for a specific tablet, we start retaining the log files.

@hectorgcr
Copy link
Contributor Author

Related to #3317

@bmatican bmatican assigned nspiegelberg and unassigned hectorgcr Oct 2, 2020
@bmatican
Copy link
Contributor

This is how we setup 2DC now: https://docs.yugabyte.com/latest/deploy/multi-dc/2dc-deployment/#root

We can also use yb-ctl with a different start_id, so we create 2 clusters locally, and then link them as such.

If I understand correctly, we should be able to completely remove the master driven WAL retention time, if we automatically turn on op id retention, when we turn on a stream. @nspiegelberg can you confirm this and perhaps share some code pointers?

@bmatican bmatican assigned sanketkedia and unassigned nspiegelberg Nov 17, 2020
@bmatican bmatican added the area/docdb YugabyteDB core features label Jul 20, 2021
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 9, 2022
@yugabyte-ci yugabyte-ci removed the area/cdc Change Data Capture label Jul 9, 2022
@yugabyte-ci yugabyte-ci added kind/enhancement This is an enhancement of an existing feature and removed kind/bug This issue is a bug labels Jul 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
None yet
Development

No branches or pull requests

6 participants