-
Notifications
You must be signed in to change notification settings - Fork 526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add compactor HTTP API for uploading TSDB blocks #1694
Merged
Merged
Changes from all commits
Commits
Show all changes
82 commits
Select commit
Hold shift + click to select a range
7635314
Compactor: Add API for uploading blocks
aknuds1 9587978
Fix tests
aknuds1 3051479
Close reader
aknuds1 4893a7f
Return 404 when temporary meta.json missing
aknuds1 de5ac22
Only log when failing to delete temporary meta.json
aknuds1 45c00ec
Remove stale code
aknuds1 cbee725
Verify that block is not complete
aknuds1 b7fe4df
Add comment
aknuds1 f2d1270
validates uploaded chunk metadata times and retention period
aldernero 0fe64a5
validates uploaded chunk metadata times and retention period
aldernero 6b913da
addressed lint / Sprintf formatting error
aldernero 7ddfd0d
addressed lint / Sprintf formatting error
aldernero f763df7
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 45b3552
Fix formatting/comment
aknuds1 46be255
Fix formatting
aknuds1 22f7d19
Add missing returns
aknuds1 1bb3674
Fix test
aknuds1 3741e30
Fix checking of block retention
aknuds1 43a7f8c
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 25acc52
Simplify meta.json validations
aknuds1 94b2990
Reduce code duplication in sanitizeMeta
aknuds1 6945fa3
Rewrite tests to be data driven
aknuds1 dd919bf
Add tests
aknuds1 f65a4f6
Add tests
aknuds1 eae1378
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 25bc5b6
Update pkg/compactor/block_upload.go
aknuds1 b517fcb
Handle case of retention period <= 0
aknuds1 40587af
Improve error logging
aknuds1 b071784
Refactor error checking
aknuds1 7cd6133
Remove unnecessary logging
aknuds1 234011c
Remove check for missing block ID
aknuds1 3dcf196
Simplify tests
aknuds1 88fa7e4
Remove unnecessary logging
aknuds1 ad09ebe
Simplify tests
aknuds1 abe0845
Formatting fix
aknuds1 6c1a3d9
Update pkg/compactor/block_upload_test.go
aknuds1 b9a3bac
Update pkg/compactor/block_upload_test.go
aknuds1 034a921
Simplify tests
aknuds1 78332e5
Add negative retention period test
aknuds1 21f79f2
Get tenant ID from context
aknuds1 b85c39c
Verify block version, sanitize some more
aknuds1 f49984a
Make tests data driven
aknuds1 8f58d41
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 7920764
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 6abb0cc
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 d156236
Reject invalid shard ID label
aknuds1 d2394c4
Use in-memory object store for valid cases
aknuds1 513f039
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 36f9810
Use mock object store also for valid cases
aknuds1 0e7ec52
Fix UploadBlockFile tests
aknuds1 25e47e8
Fix linting
aknuds1 b15e254
Make block upload completion tests data driven
aknuds1 fdfdef3
Use code unprocessable entity for block outside of retention
aknuds1 cdfa31e
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 0b38895
Simplify log messages in sanitizeMeta
aknuds1 23bd24b
Update pkg/compactor/block_upload.go
aknuds1 266117a
Use sharding.ParseShardIDLabelValue
aknuds1 320860b
Rename function
aknuds1 0748099
Rename/document constant
aknuds1 c7b4bae
Use url.QueryEscape
aknuds1 502461a
Move retention check outside of sanitizeMeta
aknuds1 a799f5d
Add test for trying to upload in-flight meta file
aknuds1 3283c13
Add test for different block ID in meta file
aknuds1 41f0801
Refactor
aknuds1 61139bd
Fix log message
aknuds1 371fe80
Improve comment
aknuds1 bd16911
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 f2606bd
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 6384d99
Add tests
aknuds1 bc8df77
Reduce code duplication
aknuds1 b2bf100
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 106ba2a
Add test case
aknuds1 6f07e73
Add tests
aknuds1 0d71074
Add changelog entry
aknuds1 5609beb
Add tests
aknuds1 2aa0cb4
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 9feae38
Remove unused parameters
aknuds1 cf8ed71
Reduce code duplication
aknuds1 9c7650b
Document experimental feature
aknuds1 d6dd163
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 4446ca1
Fix linting issue
aknuds1 c23c91e
Simplify, fix comment
aknuds1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's also mention that the feature is experimental (here: docs/sources/operators-guide/configuring/about-versioning.md:35) for now, as API may change a bit still (I expect that validation will modify "finish block" call to return progress updates).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the heads up @pstibrany, done. PTAL.