Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

safekeeper: use protobuf for sending compressed records to pageserver #9821

Merged
merged 14 commits into from
Nov 27, 2024
Merged

safekeeper: use protobuf for sending compressed records to pageserver #9821

merged 14 commits into from
Nov 27, 2024

Conversation

VladLazar
Copy link
Contributor

@VladLazar VladLazar commented Nov 20, 2024 •
edited
Loading

Problem

#9746 lifted decoding and interpretation of WAL to the safekeeper.
This reduced the ingested amount on the pageservers by around 10x for a tenant with 8 shards, but doubled
the ingested amount for single sharded tenants.

Also, #9746 uses bincode which doesn't support schema evolution.
Technically the schema can be evolved, but it's very cumbersome.

Summary of changes

This patch set addresses both problems by adding protobuf support for the interpreted wal records and adding compression support. Compressed protobuf reduced the ingested amount by 100x on the 32 shards test_sharded_ingest case (compared to non-interpreted proto). For the 1 shard case the reduction is 5x.

Sister change to rust-postgres is here.

Links

Related: #9336
Epic: #9329

@VladLazar VladLazar mentioned this pull request Nov 20, 2024
Merged
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 20, 2024 •
edited
Loading

6941 tests run: 6633 passed, 0 failed, 308 skipped (full report)

Flaky tests (1)

Postgres 17

Code coverage* (full report)

  • functions: 30.6% (7973 of 26014 functions)
  • lines: 48.5% (63306 of 130404 lines)

* collected from Rust tests only

The comment gets automatically updated with the latest test results
caf475c at 2024-11-27T11:51:36.353Z :recycle:

@VladLazar VladLazar force-pushed the vlad/sk-ps-protobuf branch 2 times, most recently from a6413a3 to 8b13a09 Compare November 20, 2024 15:37
Base automatically changed from vlad/safekeeper-interpret-wal to main November 25, 2024 17:31
@VladLazar
wal_decoder: abstract wire format
7b0f160 
We want to add new wire formats, but the current code has bincode
hard-codeded.

To this end:
1. Rework the wal_receiver_protocol PS config to include a format
   modifier for interpreted protocol type.
2. Abstract wire format encoding and decoding to a separate module
   in wal_decoder
3. Glue things back together
@VladLazar
wal_decoder: define records batch type
8ffb1c1 
Instead of serializing a vector of records over the wire,
define a type for it. Also include next_record_lsn into that type
to get varint encoding.
@VladLazar
wal_decoder: generate protobuf types
b32c6a8
@VladLazar
wal_decoder: add protobuf support
6009d9f 
The bulk of this commit is glue code to convert between protobuf
representation and in-memory representation.
@VladLazar
wal_decoder: add compression support
0a13a06
@VladLazar
safekeeper: pipeline WAL sending
cbe0042 
With this patch we can serialize one batch while waiting for the
read for the next one.
@VladLazar
tests: update sharded ingest test to use compression
1db16e2
@VladLazar VladLazar marked this pull request as ready for review November 25, 2024 18:57
@VladLazar VladLazar requested a review from a team as a code owner November 25, 2024 18:57
erikgrinaker
erikgrinaker approved these changes Nov 25, 2024
View reviewed changes
libs/wal_decoder/src/models.rs Outdated Show resolved Hide resolved
libs/wal_decoder/src/wire_format.rs Show resolved Hide resolved
libs/wal_decoder/src/wire_format.rs Outdated Show resolved Hide resolved
libs/wal_decoder/src/wire_format.rs Show resolved Hide resolved
safekeeper/src/send_interpreted_wal.rs Outdated Show resolved Hide resolved
Cargo.toml Outdated Show resolved Hide resolved
test_runner/performance/test_sharded_ingest.py Show resolved Hide resolved
@VladLazar
review: fix next_record_lsn comment
c8eaec8
@VladLazar
review: inline decoding into the select body
d0eaf20
@VladLazar
review: use proto::... instead of Proto...
8cc0143
@VladLazar
benchmarking: test_sharded_ingest results
48b0943 
test_sharded_ingest[release-pg16-vanilla-1].wal_ingest: 124.027 s
test_sharded_ingest[release-pg16-vanilla-1].pageserver_ingest: 124.038 s
test_sharded_ingest[release-pg16-vanilla-1].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-vanilla-1].wal_ingested: 6,911 MB
test_sharded_ingest[release-pg16-vanilla-1].records_received: 100442685 records
test_sharded_ingest[release-pg16-vanilla-8].wal_ingest: 121.476 s
test_sharded_ingest[release-pg16-vanilla-8].pageserver_ingest: 121.493 s
test_sharded_ingest[release-pg16-vanilla-8].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-vanilla-8].wal_ingested: 55,284 MB
test_sharded_ingest[release-pg16-vanilla-8].records_received: 803541480 records
test_sharded_ingest[release-pg16-vanilla-32].wal_ingest: 145.895 s
test_sharded_ingest[release-pg16-vanilla-32].pageserver_ingest: 145.938 s
test_sharded_ingest[release-pg16-vanilla-32].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-vanilla-32].wal_ingested: 221,137 MB
test_sharded_ingest[release-pg16-vanilla-32].records_received: 3214165952 records
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-1].wal_ingest: 121.892 s
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-1].pageserver_ingest: 121.903 s
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-1].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-1].wal_ingested: 1,557 MB
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-1].records_received: 100442669 records
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-8].wal_ingest: 123.056 s
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-8].pageserver_ingest: 123.073 s
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-8].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-8].wal_ingested: 2,037 MB
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-8].records_received: 188730015 records
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-32].wal_ingest: 142.646 s
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-32].pageserver_ingest: 142.687 s
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-32].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-32].wal_ingested: 2,256 MB
test_sharded_ingest[release-pg16-interpreted-bincode-compressed-32].records_received: 214164479 records
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-1].wal_ingest: 122.775 s
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-1].pageserver_ingest: 122.787 s
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-1].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-1].wal_ingested: 1,540 MB
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-1].records_received: 100442669 records
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-8].wal_ingest: 121.436 s
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-8].pageserver_ingest: 121.453 s
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-8].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-8].wal_ingested: 1,870 MB
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-8].records_received: 188730015 records
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-32].wal_ingest: 143.335 s
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-32].pageserver_ingest: 143.380 s
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-32].wal_written: 6,911 MB
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-32].wal_ingested: 2,073 MB
test_sharded_ingest[release-pg16-interpreted-protobuf-compressed-32].records_received: 214164479 records
@VladLazar VladLazar added this pull request to the merge queue Nov 27, 2024
Merged via the queue into main with commit 9e0148d Nov 27, 2024
80 checks passed
@VladLazar VladLazar deleted the vlad/sk-ps-protobuf branch November 27, 2024 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erikgrinaker erikgrinaker Awaiting requested review from erikgrinaker

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@VladLazar @erikgrinaker