Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(scrubber): add unified command suitable for cron job #8635

Merged
merged 3 commits into from
Aug 17, 2024

Conversation

yliang412
Copy link
Contributor

@yliang412 yliang412 commented Aug 7, 2024

Part of #8128.

Description

This PR creates a unified command to run both physical gc and metadata health check as a cron job. This also enables us to add additional tasks to the cron job in the future.

cargo run --bin storage_scrubber -- cron-job --help
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.32s
     Running `target/debug/storage_scrubber cron-job --help`
Usage: storage_scrubber cron-job [OPTIONS] --min-age <GC_MIN_AGE>

Options:
      --min-age <GC_MIN_AGE>  
  -g, --gc-mode <GC_MODE>     [default: indices-only] [possible values: dry-run, indices-only, full]
      --post                  
  -h, --help                  Print help

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

Signed-off-by: Yuchen Liang <yuchen@neon.tech>
@yliang412 yliang412 self-assigned this Aug 7, 2024
@yliang412 yliang412 added the c/storage/scrubber Component: s3_scrubber label Aug 7, 2024
@yliang412 yliang412 changed the title refactor(scrubber): add command suitable for cronjob refactor(scrubber): add command suitable for cron job Aug 7, 2024
@yliang412 yliang412 requested a review from skyzh August 7, 2024 15:48
@yliang412 yliang412 marked this pull request as ready for review August 7, 2024 15:49
Copy link

github-actions bot commented Aug 7, 2024

2172 tests run: 2103 passed, 0 failed, 69 skipped (full report)


Flaky tests (1)

Postgres 16

  • test_scrubber_physical_gc_ancestors_split: debug

Code coverage* (full report)

  • functions: 32.4% (7231 of 22306 functions)
  • lines: 50.4% (58488 of 116119 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
cae5641 at 2024-08-17T15:56:33.181Z :recycle:

@yliang412 yliang412 changed the title refactor(scrubber): add command suitable for cron job refactor(scrubber): add unified command suitable for cron job Aug 12, 2024
@yliang412 yliang412 requested a review from jcsp August 14, 2024 03:12
@yliang412 yliang412 enabled auto-merge (squash) August 17, 2024 15:05
@yliang412 yliang412 merged commit 7131ac4 into main Aug 17, 2024
59 checks passed
@yliang412 yliang412 deleted the yuchen/scrubber-cronjob-command branch August 17, 2024 15:48
VladLazar pushed a commit that referenced this pull request Aug 20, 2024
Part of #8128.

## Description

This PR creates a unified command to run both physical gc and metadata
health check as a cron job. This also enables us to add additional tasks
to the cron job in the future.

Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/scrubber Component: s3_scrubber
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants