Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

👑 Node Broker Delta Protocol #11064

Open
13 tasks
pixcc opened this issue Oct 29, 2024 · 0 comments
Open
13 tasks

👑 Node Broker Delta Protocol #11064

pixcc opened this issue Oct 29, 2024 · 0 comments
Assignees
@pixcc
Labels
area/core YDB Core issues epic

Comments

@pixcc
Copy link
Member

pixcc commented Oct 29, 2024 •
edited by CyberROFL
Loading

Description

Currently Node Broker sends out full node list information to all nodes when the epoch ends (usually every hour). This causes a periodic spike in network traffic. It is necessary to change the protocol to send out only node list changes (deltas).

Internal design doc.

Steps

Dynamic Nameserver

  • Refactor cache miss case to use only one pipe to Node Broker
  • Determine which protocol to use for Node Broker
  • Apply change deltas to a cached list of nodes
  • Synchronize node list with Node Broker (client-side)

Node Broker

  • Refactor internal transactions
  • Calculate and broadcast change deltas
  • Synchronize node list with Dynamic Nameserver (server-side)
  • Ratelimit change deltas broadcasting
  • Migration to new schema

Deploy milestones

  • Refactor cache miss case in Dynamic Nameserver to use only one pipe to Node Broker
  • Refactor internal transactions in Node Broker
  • Support new delta protocol in Node Broker & Dynamic Nameserver
  • Ratelimit change deltas broadcasting
@pixcc pixcc self-assigned this Oct 29, 2024
@snaury snaury mentioned this issue Nov 6, 2024
Closed
8 tasks
@pixcc pixcc added the area/core YDB Core issues label Nov 11, 2024
@CyberROFL CyberROFL added the epic label Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pixcc pixcc

Labels
area/core YDB Core issues epic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@CyberROFL @pixcc