Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compactor HTTP API for uploading TSDB blocks #1694

Merged
merged 82 commits into from
Jun 17, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
7635314
Compactor: Add API for uploading blocks
aknuds1 Jun 7, 2022
9587978
Fix tests
aknuds1 Jun 7, 2022
3051479
Close reader
aknuds1 Jun 7, 2022
4893a7f
Return 404 when temporary meta.json missing
aknuds1 Jun 7, 2022
de5ac22
Only log when failing to delete temporary meta.json
aknuds1 Jun 7, 2022
45c00ec
Remove stale code
aknuds1 Jun 7, 2022
cbee725
Verify that block is not complete
aknuds1 Jun 7, 2022
b7fe4df
Add comment
aknuds1 Jun 7, 2022
f2d1270
validates uploaded chunk metadata times and retention period
aldernero Jun 8, 2022
0fe64a5
validates uploaded chunk metadata times and retention period
aldernero Jun 8, 2022
6b913da
addressed lint / Sprintf formatting error
aldernero Jun 8, 2022
7ddfd0d
addressed lint / Sprintf formatting error
aldernero Jun 8, 2022
f763df7
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 8, 2022
45b3552
Fix formatting/comment
aknuds1 Jun 8, 2022
46be255
Fix formatting
aknuds1 Jun 8, 2022
22f7d19
Add missing returns
aknuds1 Jun 8, 2022
1bb3674
Fix test
aknuds1 Jun 8, 2022
3741e30
Fix checking of block retention
aknuds1 Jun 8, 2022
43a7f8c
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 8, 2022
25acc52
Simplify meta.json validations
aknuds1 Jun 9, 2022
94b2990
Reduce code duplication in sanitizeMeta
aknuds1 Jun 9, 2022
6945fa3
Rewrite tests to be data driven
aknuds1 Jun 9, 2022
dd919bf
Add tests
aknuds1 Jun 9, 2022
f65a4f6
Add tests
aknuds1 Jun 9, 2022
eae1378
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 9, 2022
25bc5b6
Update pkg/compactor/block_upload.go
aknuds1 Jun 9, 2022
b517fcb
Handle case of retention period <= 0
aknuds1 Jun 9, 2022
40587af
Improve error logging
aknuds1 Jun 9, 2022
b071784
Refactor error checking
aknuds1 Jun 9, 2022
7cd6133
Remove unnecessary logging
aknuds1 Jun 9, 2022
234011c
Remove check for missing block ID
aknuds1 Jun 9, 2022
3dcf196
Simplify tests
aknuds1 Jun 9, 2022
88fa7e4
Remove unnecessary logging
aknuds1 Jun 9, 2022
ad09ebe
Simplify tests
aknuds1 Jun 9, 2022
abe0845
Formatting fix
aknuds1 Jun 9, 2022
6c1a3d9
Update pkg/compactor/block_upload_test.go
aknuds1 Jun 9, 2022
b9a3bac
Update pkg/compactor/block_upload_test.go
aknuds1 Jun 9, 2022
034a921
Simplify tests
aknuds1 Jun 9, 2022
78332e5
Add negative retention period test
aknuds1 Jun 9, 2022
21f79f2
Get tenant ID from context
aknuds1 Jun 9, 2022
b85c39c
Verify block version, sanitize some more
aknuds1 Jun 9, 2022
f49984a
Make tests data driven
aknuds1 Jun 9, 2022
8f58d41
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 13, 2022
7920764
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 13, 2022
6abb0cc
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 13, 2022
d156236
Reject invalid shard ID label
aknuds1 Jun 13, 2022
d2394c4
Use in-memory object store for valid cases
aknuds1 Jun 13, 2022
513f039
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 14, 2022
36f9810
Use mock object store also for valid cases
aknuds1 Jun 14, 2022
0e7ec52
Fix UploadBlockFile tests
aknuds1 Jun 14, 2022
25e47e8
Fix linting
aknuds1 Jun 14, 2022
b15e254
Make block upload completion tests data driven
aknuds1 Jun 14, 2022
fdfdef3
Use code unprocessable entity for block outside of retention
aknuds1 Jun 14, 2022
cdfa31e
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 14, 2022
0b38895
Simplify log messages in sanitizeMeta
aknuds1 Jun 14, 2022
23bd24b
Update pkg/compactor/block_upload.go
aknuds1 Jun 14, 2022
266117a
Use sharding.ParseShardIDLabelValue
aknuds1 Jun 14, 2022
320860b
Rename function
aknuds1 Jun 14, 2022
0748099
Rename/document constant
aknuds1 Jun 14, 2022
c7b4bae
Use url.QueryEscape
aknuds1 Jun 14, 2022
502461a
Move retention check outside of sanitizeMeta
aknuds1 Jun 14, 2022
a799f5d
Add test for trying to upload in-flight meta file
aknuds1 Jun 14, 2022
3283c13
Add test for different block ID in meta file
aknuds1 Jun 14, 2022
41f0801
Refactor
aknuds1 Jun 14, 2022
61139bd
Fix log message
aknuds1 Jun 14, 2022
371fe80
Improve comment
aknuds1 Jun 14, 2022
bd16911
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 14, 2022
f2606bd
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 15, 2022
6384d99
Add tests
aknuds1 Jun 15, 2022
bc8df77
Reduce code duplication
aknuds1 Jun 15, 2022
b2bf100
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 15, 2022
106ba2a
Add test case
aknuds1 Jun 15, 2022
6f07e73
Add tests
aknuds1 Jun 16, 2022
0d71074
Add changelog entry
aknuds1 Jun 16, 2022
5609beb
Add tests
aknuds1 Jun 16, 2022
2aa0cb4
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 16, 2022
9feae38
Remove unused parameters
aknuds1 Jun 16, 2022
cf8ed71
Reduce code duplication
aknuds1 Jun 16, 2022
9c7650b
Document experimental feature
aknuds1 Jun 16, 2022
d6dd163
Merge remote-tracking branch 'origin/main' into feat/backfill
aknuds1 Jun 16, 2022
4446ca1
Fix linting issue
aknuds1 Jun 16, 2022
c23c91e
Simplify, fix comment
aknuds1 Jun 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
* [ENHANCEMENT] Chunk Mapper: reduce memory usage of async chunk mapper. #2043
* [ENHANCEMENT] Ingesters: Added new configuration option that makes it possible for mimir ingesters to perform queries on overlapping blocks in the filesystem. Enabled with `-blocks-storage.tsdb.allow-overlapping-queries`. #2091
* [ENHANCEMENT] Ingester: reduce sleep time when reading WAL. #2098
* [ENHANCEMENT] Compactor: Add HTTP API for uploading TSDB blocks. #1694
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also mention that the feature is experimental (here: docs/sources/operators-guide/configuring/about-versioning.md:35) for now, as API may change a bit still (I expect that validation will modify "finish block" call to return progress updates).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the heads up @pstibrany, done. PTAL.

* [BUGFIX] Fix regexp parsing panic for regexp label matchers with start/end quantifiers. #1883
* [BUGFIX] Ingester: fixed deceiving error log "failed to update cached shipped blocks after shipper initialisation", occurring for each new tenant in the ingester. #1893
* [BUGFIX] Ring: fix bug where instances may appear unhealthy in the hash ring web UI even though they are not. #1933
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ The following features are currently experimental:
- `-alertmanager-storage.storage-prefix`
- `-blocks-storage.storage-prefix`
- `-ruler-storage.storage-prefix`
- Compactor
- HTTP API for uploading TSDB blocks

## Deprecated features

Expand Down
6 changes: 5 additions & 1 deletion pkg/api/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -354,12 +354,16 @@ func (a *API) RegisterStoreGateway(s *storegateway.StoreGateway) {
a.RegisterRoute("/store-gateway/tenant/{tenant}/blocks", http.HandlerFunc(s.BlocksHandler), false, true, "GET")
}

// RegisterCompactor registers the ring UI page associated with the compactor.
// RegisterCompactor registers routes associated with the compactor.
func (a *API) RegisterCompactor(c *compactor.MultitenantCompactor) {
a.indexPage.AddLinks(defaultWeight, "Compactor", []IndexPageLink{
{Desc: "Ring status", Path: "/compactor/ring"},
})
a.RegisterRoute("/compactor/ring", http.HandlerFunc(c.RingHandler), false, true, "GET", "POST")
a.RegisterRoute("/api/v1/upload/block/{block}", http.HandlerFunc(c.HandleBlockUpload), true,
false, http.MethodPost)
a.RegisterRoute("/api/v1/upload/block/{block}/files", http.HandlerFunc(c.UploadBlockFile),
true, false, http.MethodPost)
}

type Distributor interface {
Expand Down
Loading