Code

Proxy release 2024-07-11#8351

Code

Merged

conradludgate merged 102 commits intorelease-proxyneondatabase/neon:release-proxyfrom rc/proxy/2024-07-11neondatabase/neon:rc/proxy/2024-07-11Copy head branch name to clipboardJul 11, 2024

+9,465-3,398

Commits

Commits on Jun 27, 2024

• strocon: don't overcommit when making node fill plan (#8171)

  Show description for d557002
  VladLazar

  authored
  d557002

  Copy full SHA for d557002

• Allow to change compute safekeeper list without restart.

  Show description for 6f20a18
  arssher

  committed
  6f20a18

  Copy full SHA for 6f20a18

• CI: Use runner.arch in cache keys along with runner.os (#8175)

  Show description for 54a06de
  bayandin

  authored
  54a06de

  Copy full SHA for 54a06de

• stocon: bump number of concurrent reconciles per operation (#8179)

  Show description for 89cf8df
  VladLazar

  authored
  89cf8df

  Copy full SHA for 89cf8df

• fix: shutdown does not kill walredo processes (#8150)

  Show description for 66b0bf4
  problame

  authored
  66b0bf4

  Copy full SHA for 66b0bf4

• feat(pageserver): add delta layer iterator (#8064)

  Show description for 23827c6
  skyzh

  authored
  23827c6

  Copy full SHA for 23827c6

• Improve slow operations observability in safekeepers (#8188)

  Show description for 1d66ca7
  petuhovskiy

  authored
  1d66ca7

  Copy full SHA for 1d66ca7

• Add application_name to compute activity monitor connection string

  Show description for 5700233
  tristan957

  committed
  5700233

  Copy full SHA for 5700233

Commits on Jun 28, 2024

• pageserver: remove tenant create API (#8135)

  Show description for 063553a
  jcsp

  authored
  063553a

  Copy full SHA for 063553a

• virtual_file: take a Slice in the read APIs, eliminate read_exact_at_n, fix UB for engine std-fs (#8186)

  Show description for deec3bc
  problame

  authored
  deec3bc

  Copy full SHA for deec3bc

• Add buckets to safekeeper ops metrics (#8194)

  Show description for c22c6a6
  petuhovskiy

  authored
  c22c6a6

  Copy full SHA for c22c6a6

• Cherry-pick upstream fix for TruncateMultiXact assertion (#8195)

  Show description for ca2f7d0
  hlinnaka

  authored
  ca2f7d0

  Copy full SHA for ca2f7d0

• pageserver: drop out of secondary download if iteration time has passed (#8198)

  Show description for babbe12
  jcsp

  authored
  babbe12

  Copy full SHA for babbe12

• Add rate limiter for partial uploads (#8203)

  Show description for e1a06b4
  petuhovskiy

  authored
  e1a06b4

  Copy full SHA for e1a06b4

• storage controller: fix heatmaps getting disabled during shard split (#8197)

  Show description for b8bbaaf
  jcsp

  authored
  b8bbaaf

  Copy full SHA for b8bbaaf

• fix(pageserver): ensure tenant harness has different names (#8205)

  Show description for bc70491
  skyzh

  authored
  bc70491

  Copy full SHA for bc70491

Commits on Jun 30, 2024

• Fix tracking of the nextMulti in the pageserver's copy of CheckPoint (#6528)

  Show description for 30027d9
  

  hlinnaka

  and

  Konstantin Knizhnik

  committed
  30027d9

  Copy full SHA for 30027d9

Commits on Jul 1, 2024

• tests: remove a leftover 'running' flag (#8216)

  Show description for 57535c0
  hlinnaka

  authored
  57535c0

  Copy full SHA for 57535c0

• tests: Make neon_xlogflush() flush all WAL, if you omit the LSN arg

  Show description for 75c84c8
  hlinnaka

  committed
  75c84c8

  Copy full SHA for 75c84c8

• Restore running xacts from CLOG on replica startup (#7288)

  Show description for 9ce1930
  

  hlinnaka

  and

  Konstantin Knizhnik

  committed
  9ce1930

  Copy full SHA for 9ce1930

• pageserver: add metric pageserver_secondary_resident_physical_size (#8204)

  Show description for aea5cfe
  jcsp

  authored
  aea5cfe

  Copy full SHA for aea5cfe

• CI(build-tools): Remove libpq from build image (#8206)

  Show description for e823b92
  bayandin

  authored
  e823b92

  Copy full SHA for e823b92

• fix(pageserver): include aux file in basebackup only once (#8207)

  Show description for b02aafd
  

  skyzh

  and

  Konstantin Knizhnik

  authored
  b02aafd

  Copy full SHA for b02aafd

• CI(gather-rust-build-stats): fix build with libpq (#8219)

  Show description for 9c32604
  bayandin

  authored
  9c32604

  Copy full SHA for 9c32604

• tests: Make neon_xlogflush() flush all WAL, if you omit the LSN arg (#8215)

  Show description for 0789160
  hlinnaka

  authored
  0789160

  Copy full SHA for 0789160

• docs: Graceful storage controller cluster restarts RFC (#7704)

  Show description for 9882ac8
  VladLazar

  authored
  9882ac8

  Copy full SHA for 9882ac8

Commits on Jul 2, 2024

• Check status of connection after PQconnectStartParams (#8210)

  Show description for 0497b99
  

  knizhnik

  and

  Konstantin Knizhnik

  authored
  0497b99

  Copy full SHA for 0497b99

• remote_storage config: move handling of empty inline table {} to callers (#8193)

  Show description for 7dcdbaa
  problame

  authored
  7dcdbaa

  Copy full SHA for 7dcdbaa

• pageserver: simpler, stricter config error handling (#8177)

  Show description for 1a0f545
  jcsp

  authored
  1a0f545

  Copy full SHA for 1a0f545

• fix: noisy logging when download gets cancelled during shutdown (#8224)

  Show description for 9b4b4bb
  problame

  authored
  9b4b4bb

  Copy full SHA for 9b4b4bb

• pageserver: rate limit log for loads of layers visited (#8228)

  Show description for 28929d9
  VladLazar

  authored
  28929d9

  Copy full SHA for 28929d9

• Add support for reading and writing compressed blobs (#8106)

  Show description for 25eefde
  arpad-m

  authored
  25eefde

  Copy full SHA for 25eefde

• L0 flush: opt-in mechanism to bypass PageCache reads and writes (#8190)

  Show description for 5de896e
  problame

  authored
  5de896e

  Copy full SHA for 5de896e

• CI(benchmarking): move psql queries to actions/run-python-test-set (#8230)

  Show description for 6216df7
  bayandin

  authored
  6216df7

  Copy full SHA for 6216df7

• tense of errors (#8234)

  Show description for f583232
  jcsp

  authored
  f583232

  Copy full SHA for f583232

• fix(pageserver): comments about metadata key range (#8236)

  Show description for 891cb5a
  skyzh

  authored
  891cb5a

  Copy full SHA for 891cb5a

• Add test for proper handling of connection failure to avoid 'cannot wait on socket event without a socket' error (#8231)

  Show description for 4a0c2ae
  

  knizhnik

  and

  Konstantin Knizhnik

  authored
  4a0c2ae

  Copy full SHA for 4a0c2ae

Commits on Jul 3, 2024

• docker: add storage_scrubber into the docker image (#8239)

  Show description for 4273309
  skyzh

  authored
  4273309

  Copy full SHA for 4273309

• Simplify test_wal_page_boundary_start test (#8214)

  Show description for dae55ba
  hlinnaka

  authored
  dae55ba

  Copy full SHA for dae55ba

• CI: update docker/* actions to latest versions (#7694)

  Show description for aae3876
  bayandin

  authored
  aae3876

  Copy full SHA for aae3876

• pageserver: don't try to flush if shutdown during attach (#8235)

  Show description for 97f7188
  jcsp

  authored
  97f7188

  Copy full SHA for 97f7188

• Only support compressed reads if the compression setting is present (#8238)

  Show description for e0891ec
  arpad-m

  authored
  e0891ec

  Copy full SHA for e0891ec

• add pagebench test cases for periodic pagebench on dedicated hardware (#8233)

  Show description for 392a58b
  Bodobolero

  authored
  392a58b

  Copy full SHA for 392a58b

• pageserver: reduce ops tracked at per-timeline detail (#8245)

  Show description for ea0b22a
  jcsp

  authored
  ea0b22a

  Copy full SHA for ea0b22a

• Fix outdated comment (#8149)

  Show description for cdaed4d
  japinli

  authored
  cdaed4d

  Copy full SHA for cdaed4d

• page_service: stop exposing get_last_record_rlsn (#8244)

  Show description for a85aa03
  problame

  authored
  a85aa03

  Copy full SHA for a85aa03

• fix(pageserver): ensure test creates valid layer map (#8191)

  Show description for 90b51dc
  

  skyzh

  and

  problame

  authored
  90b51dc

  Copy full SHA for 90b51dc

• pageserver: add supplementary branch usage stats (#8131)

  Show description for 778787d
  jcsp

  authored
  778787d

  Copy full SHA for 778787d

Commits on Jul 4, 2024

• tests: perform graceful rolling restarts in storcon scale test (#8173)

  Show description for bbb2fa7
  VladLazar

  authored
  bbb2fa7

  Copy full SHA for bbb2fa7

• proxy: cache certain non-retriable console errors for a short time (#8201)

  Show description for e03c3c9
  conradludgate

  authored
  e03c3c9

  Copy full SHA for e03c3c9

• CI(build-and-test): add conclusion job (#8246)

  Show description for 5b69b32
  bayandin

  authored
  5b69b32

  Copy full SHA for 5b69b32

• pageserver: increase rate limit duration for layer visit log (#8263)

  Show description for a462537
  VladLazar

  authored
  a462537

  Copy full SHA for a462537

• Use bool param for round_trip_test_compressed (#8252)

  Show description for a004d27
  arpad-m

  authored
  a004d27

  Copy full SHA for a004d27

• CI(pg-clients): unify workflow with build-and-test (#8160)

  Show description for bf9fc77
  bayandin

  authored
  bf9fc77

  Copy full SHA for bf9fc77

• pageserver: downgrade stale generation messages to INFO (#8256)

  Show description for c9e6dd4
  jcsp

  authored
  c9e6dd4

  Copy full SHA for c9e6dd4

• Add find-large-objects subcommand to scrubber (#8257)

  Show description for e579bc0
  arpad-m

  authored
  e579bc0

  Copy full SHA for e579bc0

• feat(pageserver): integrate lsn lease into synthetic size (#8220)

  Show description for 19accfe
  

  3 peopleauthored
  19accfe

  Copy full SHA for 19accfe

• Flatten compression algorithm setting (#8265)

  Show description for adde0ec
  arpad-m

  authored
  adde0ec

  Copy full SHA for adde0ec

• implement rolling hyper-log-log algorithm (#8068)

  Show description for 88b13d4
  

  3 peopleauthored
  88b13d4

  Copy full SHA for 88b13d4

• add checkout depth1 to workflow to access local github actions like generate allure report (#8259)

  Show description for 711716c
  

  Bodobolero

  and

  bayandin

  authored
  711716c

  Copy full SHA for 711716c

Commits on Jul 5, 2024

• tests: extend allow list in deletion test (#8268)

  Show description for e25ac31
  jcsp

  authored
  e25ac31

  Copy full SHA for e25ac31

• correct error handling for periodic pagebench runner status (#8274)

  Show description for 6876f0d
  

  Bodobolero

  and

  bayandin

  authored
  6876f0d

  Copy full SHA for 6876f0d

• tests: make location_conf_churn more robust (#8271)

  Show description for 5aae806
  jcsp

  authored
  5aae806

  Copy full SHA for 5aae806

• safekeeper: add separate tombstones map for deleted timelines (#8253)

  Show description for 6849ae4
  jcsp

  authored
  6849ae4

  Copy full SHA for 6849ae4

• pageserver: add time based image layer creation check (#8247)

  Show description for 7dd2e44
  VladLazar

  authored
  7dd2e44

  Copy full SHA for 7dd2e44

• SELECT 💣(); (#8270)

  Show description for c9fd8d7
  bayandin

  authored
  c9fd8d7

  Copy full SHA for c9fd8d7

• Increase timeout for wating subscriber caught-up (#8118)

  Show description for 13522fb
  

  knizhnik

  and

  Konstantin Knizhnik

  authored
  13522fb

  Copy full SHA for 13522fb

• pageserver_live_connections: track as counter pair (#8227)

  Show description for f0d29a0
  problame

  authored
  f0d29a0

  Copy full SHA for f0d29a0

• Improve parsing of ImageCompressionAlgorithm (#8281)

  Show description for b8d031c
  arpad-m

  authored
  b8d031c

  Copy full SHA for b8d031c

• Add concurrency to the find-large-objects scrubber subcommand (#8291)

  Show description for 0a937b7
  arpad-m

  authored
  0a937b7

  Copy full SHA for 0a937b7

Commits on Jul 6, 2024

• build(deps): bump certifi from 2023.7.22 to 2024.7.4 (#8301)

  dependabot[bot]

  authored
  27fe7f8

  Copy full SHA for 27fe7f8

Commits on Jul 8, 2024

• fix(pageserver): ensure sparse keyspace is ordered (#8285)

  Show description for 154ba5e
  skyzh

  authored
  154ba5e

  Copy full SHA for 154ba5e

• pageserver: switch to jemalloc (#8307)

  Show description for 1121a1c
  jcsp

  authored
  1121a1c

  Copy full SHA for 1121a1c

• Hide import behind TYPE_CHECKING

  Show description for 2a3410d
  tristan957

  committed
  2a3410d

  Copy full SHA for 2a3410d

• Add Neon HTTP API test fixture

  Show description for f2ec542
  

  tristan957

  and

  Sasha Krassovsky

  committed
  f2ec542

  Copy full SHA for f2ec542

• Log PG environment variables when a PgBin runs

  Show description for 118847c
  

  tristan957

  and

  Sasha Krassovsky

  committed
  118847c

  Copy full SHA for 118847c

• Add PgBin.run_nonblocking()

  Show description for b54dd9a
  

  tristan957

  and

  Sasha Krassovsky

  committed
  b54dd9a

  Copy full SHA for b54dd9a

• Add long running replication tests

  Show description for 1c57f6b
  

  tristan957

  and

  Sasha Krassovsky

  committed
  1c57f6b

  Copy full SHA for 1c57f6b

• pageserver: respect has_relmap_file in collect_keyspace (#8276)

  Show description for fcdf060
  jcsp

  authored
  fcdf060

  Copy full SHA for fcdf060

• refactor: move part of sharding API from pageserver_api to utils (#8254)

  Show description for a68edad
  yliang412

  authored
  a68edad

  Copy full SHA for a68edad

• compute_ctl: Use 'fast' shutdown for Postgres termination (#8289)

  Show description for 84b039e
  ololobus

  authored
  84b039e

  Copy full SHA for 84b039e

• tests: use smaller layers in test_pg_regress (#8232)

  Show description for daea26a
  jcsp

  authored
  daea26a

  Copy full SHA for daea26a

• fix(pageserver): write to both v1+v2 for aux tenant import (#8316)

  Show description for df3dc6e
  skyzh

  authored
  df3dc6e

  Copy full SHA for df3dc6e

• tests: stabilize test_timeline_size_quota_on_startup (#8255)

  Show description for 811eb88
  jcsp

  authored
  811eb88

  Copy full SHA for 811eb88

• tests: increase approx size equal threshold to avoid test_lsn_lease_size flakiness (#8282)

  Show description for d9c1068
  yliang412

  authored
  d9c1068

  Copy full SHA for d9c1068

Commits on Jul 9, 2024

• CI(promote-compatibility-data): take into account commit sha (#8283)

  Show description for 8b15864
  bayandin

  authored
  8b15864

  Copy full SHA for 8b15864

• proxy/http: avoid spurious vector reallocations

  Show description for c196cf6
  

  lucab

  authored and

  conradludgate

  committed
  c196cf6

  Copy full SHA for c196cf6

• chore(storage-scrubber): allow disable file logging (#8297)

  Show description for 73fa3c0
  skyzh

  authored
  73fa3c0

  Copy full SHA for 73fa3c0

• chore: fix nightly build (#8142)

  Show description for 4a5b55c
  conradludgate

  authored
  4a5b55c

  Copy full SHA for 4a5b55c

• fix(storage-scrubber): use default AWS authentication (#8299)

  Show description for b1fe825
  

  skyzh

  and

  koivunej

  authored
  b1fe825

  Copy full SHA for b1fe825

• Refactor how migrations are ran

  Show description for 6d3cb22
  tristan957

  committed
  6d3cb22

  Copy full SHA for 6d3cb22

• Add an application_name to more Neon connections

  Show description for abc330e
  tristan957

  committed
  abc330e

  Copy full SHA for abc330e

• refactor: postgres_backend: replace abstract shutdown_watcher with CancellationToken (#8295)

  Show description for 3f7aebb
  problame

  authored
  3f7aebb

  Copy full SHA for 3f7aebb

• fix(l0_flush): drops permit before fsync, potential cause for OOMs (#8327)

  Show description for 9bb16c8
  problame

  authored
  9bb16c8

  Copy full SHA for 9bb16c8

• pageserver: move page_service's import basebackup / import wal to mgmt API (#8292)

  Show description for 1a49f1c
  problame

  authored
  1a49f1c

  Copy full SHA for 1a49f1c

Commits on Jul 10, 2024

• proxy: pg17 fixes (#8321)

  Show description for fe13fcc
  conradludgate

  authored
  fe13fcc

  Copy full SHA for fe13fcc

• tests: stabilize test_sharding_split_compaction (#8318)

  Show description for e89ec55
  jcsp

  authored
  e89ec55

  Copy full SHA for e89ec55

• proxy: remove some trace logs (#8334)

  conradludgate

  authored
  1afab13

  Copy full SHA for 1afab13

• build(deps-dev): bump zipp from 3.8.1 to 3.19.1

  Show description for 98387d6
  

  dependabot[bot]

  authored and

  conradludgate

  committed
  98387d6

  Copy full SHA for 98387d6

• Remove ImageCompressionAlgorithm::DisabledNoDecompress (#8300)

  Show description for e78341e
  arpad-m

  authored
  e78341e

  Copy full SHA for e78341e

• feat(pageserver): add k-merge layer iterator with lazy loading (#8053)

  Show description for 9f4511c
  

  skyzh

  and

  problame

  authored
  9f4511c

  Copy full SHA for 9f4511c

• safekeeper: add eviction_min_resident to stop evictions thrashing (#8335)

  Show description for 24f8133
  jcsp

  authored
  24f8133

  Copy full SHA for 24f8133