Skip to content

Commit

Permalink
rfc: update aux file rfc to reflect latest optimizations (#9681)
Browse files Browse the repository at this point in the history
Reflects #9631 in the RFC.

Signed-off-by: Alex Chi Z <chi@neon.tech>
  • Loading branch information
skyzh authored Nov 11, 2024
1 parent 48c06d9 commit 54a1676
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/rfcs/038-aux-file-v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ generating the basebackup by scanning the `REPL_ORIGIN_KEY_PREFIX` keyspace.
There are two places we need to read the aux files from the pageserver:

* On the write path, when the compute node adds an aux file to the pageserver, we will retrieve the key from the storage, append the file to the hashed key, and write it back. The current `get` API already supports that.
* We use the vectored get API to retrieve all aux files during generating the basebackup. Because we need to scan a sparse keyspace, we slightly modified the vectored get path. The vectorized API will attempt to retrieve every single key within the requested key range, and therefore, we modified it in a way that keys within `NON_INHERITED_SPARSE_RANGE` will not trigger missing key error.
* We use the vectored get API to retrieve all aux files during generating the basebackup. Because we need to scan a sparse keyspace, we slightly modified the vectored get path. The vectorized API used to always attempt to retrieve every single key within the requested key range, and therefore, we modified it in a way that keys within `NON_INHERITED_SPARSE_RANGE` will not trigger missing key error. Furthermore, as aux file reads usually need all layer files intersecting with that key range within the branch and cover a big keyspace, it incurs large overhead for tracking keyspaces that have not been read. Therefore, for sparse keyspaces, we [do not track](https://github.com/neondatabase/neon/pull/9631) `ummapped_keyspace`.

## Compaction and Image Layer Generation

Expand Down

1 comment on commit 54a1676

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5446 tests run: 5213 passed, 1 failed, 232 skipped (full report)


Failures on Postgres 16

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_sharded_ingest[release-pg16-github-actions-selfhosted-1]"
Flaky tests (1)

Postgres 17

Code coverage* (full report)

  • functions: 31.7% (7869 of 24806 functions)
  • lines: 49.4% (62262 of 126036 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
54a1676 at 2024-11-11T16:04:44.436Z :recycle:

Please sign in to comment.