Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Per-timeline metrics for whether branch is from within PITR interval, and its history size #8108

Closed
jcsp opened this issue Jun 19, 2024 · 0 comments · Fixed by #8131
Closed

Per-timeline metrics for whether branch is from within PITR interval, and its history size #8108

jcsp opened this issue Jun 19, 2024 · 0 comments · Fixed by #8131
Assignees
@jcsp
Labels
c/storage/pageserver Component: storage: pageserver t/feature Issue type: feature, for new features or requests

Comments

@jcsp
Copy link
Collaborator

jcsp commented Jun 19, 2024

This will allow analysis of storage usage in alternative ways to synthetic size, that attribute each branch's cost in a simpler way:

  • It's logical size (we already have this)
  • Whether its ancestor_lsn is within the parent's PITR interval (i.e. whether its ancestor data is something the parent would keep anyway, or if it's being kept just for this branch's benefit).
  • The LSN delta between its branch point and its tip (currently to get that one would have to hit HTTP API).

Collectively this would be enough information to describe a branch's high level storage cost, e.g.

  • "This is a branch within PITR window, and it has 10MiB of writes on top of that"
  • "This is a branch outside the PITR window, it has a logical size of 100MiB and 20MiB of history".
@jcsp jcsp added t/feature Issue type: feature, for new features or requests c/storage/pageserver Component: storage: pageserver labels Jun 19, 2024
@jcsp jcsp self-assigned this Jun 19, 2024
@jcsp jcsp mentioned this issue Jun 21, 2024
Merged
5 tasks
@jcsp jcsp closed this as completed in #8131 Jul 3, 2024
@jcsp jcsp closed this as completed in 778787d Jul 3, 2024
VladLazar pushed a commit that referenced this issue Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
754e5d3 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this issue Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
841ef39 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this issue Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
f3c21f4 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this issue Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
cb1b35b 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this issue Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
dad88b0 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this issue Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
7b7d16f 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jcsp jcsp

Labels
c/storage/pageserver Component: storage: pageserver t/feature Issue type: feature, for new features or requests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pageserver: add supplementary branch usage stats neondatabase/neon
1 participant
@jcsp