Implement Phoenix Network Coordinates #854

XAMPPRocky · 2023-11-15T08:55:15Z

This commit uses the previous commit of datacentre discovery with an
implementation of Phoenix Network Coordinates1, to build a
decentralised network coordinate map of all the datacentres that the
proxies know about, using that information, we expose a new HTTP server
at the same QCMP port that a client can use to query what datacentres
are the closest to the proxy.

For identifying datacentres we use ICAO codes, this allows us to
colnsildate multiple logical datacentres into a single location, for
example if you have multiple zones allocated in the same physical
datacentre, in order to reduce the amount of options a client has to
pick from in that case they're consolidated under their ICAO code.

This allows for clients to get their full latency to datacentres which
allows operators to a allocate datacentres dynamically, and in the
closest location available to players.

github-actions · 2023-11-15T08:55:36Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-15T08:59:17Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-15T09:01:02Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-15T09:05:13Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-16T08:51:30Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-16T16:51:02Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-17T09:36:16Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-17T12:08:38Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-18T10:10:20Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-22T17:13:10Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-22T17:50:09Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-22T18:22:11Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-22T18:22:15Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-22T18:22:22Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-11-22T18:22:28Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

markmandel

Awesome - passing tests!

Would love some docs - this looks very cool 😄

github-actions · 2023-11-23T21:00:11Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2024-02-27T13:29:07Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

XAMPPRocky · 2024-02-27T13:59:19Z

Well, works as in, all tests pass locally for me, I have no idea why relay_routing fails in CI.

Seems like that is still spurious. I'll add you as the code owner, and add the docs for this feature, and then we can get it in.

Jake-Shadle · 2024-02-27T14:23:46Z

Yes, I just increased the timeout to get it to pass. Honestly I would like to find a path to getting all tests to work without any timeouts via some kind of sidechannel only enabled when running tests because the amount of just "random" timeouts throughout the tests just feels very fragile.

XAMPPRocky · 2024-02-27T14:29:14Z

because the amount of just "random" timeouts throughout the tests just feels very fragile.

Agreed, though it would require a lot of refactoring that I'm not sure is worth it right now. Since it would require making a lot of initialisation and watch change notification channels and making those available to tests in a way that's also not hacky.

markmandel · 2024-02-28T07:47:26Z

Rather than sleeps, one pattern I've always appreciated in Go was:
https://pkg.go.dev/github.com/stretchr/testify/assert#Eventually

Which is essentially an assertion that polls as a set interval until a timeout - and I've found it super handy for test cases where things are eventually consistent (and don't require the aforementioned refactoring).

I've never managed to find an equivalent in Rust, short a loop with a sleep in it and a continue on the condition check? Maybe that's just the best way?

This commit adds the concept of "datacenter" to our xDS protocol. Unlike other resources, there's slightly different behaviour between proxies to relays and relays to agents. When agent queries for a datacenter from the agent, it returns its own QCMP port and ICAO location. The host then adds the address of remote agent to identify the datacenter. The proxy then queries for datacenters, it gets all of agents addresses, ports, and ICAO codes. This will be used with a future commit to measure the distance between proxies and agents.

This commit uses the previous commit of datacentre discovery with an implementation of Phoenix Network Coordinates[1], to build a decentralised network coordinate map of all the datacentres that the proxies know about, using that information, we expose a new HTTP server at the same QCMP port that a client can use to query what datacentres are the closest to the proxy. For identifying datacentres we use ICAO codes, this allows us to colnsildate multiple logical datacentres into a single location, for example if you have multiple zones allocated in the same physical datacentre, in order to reduce the amount of options a client has to pick from in that case they're consolidated under their ICAO code. This allows for clients to get their full latency to datacentres which allows operators to a allocate datacentres dynamically, and in the closest location available to players. [1]: https://www.researchgate.net/publication/221198962_Phoenix_Towards_an_Accurate_Practical_and_Decentralized_Network_Coordinate_System

github-actions · 2024-02-29T08:33:44Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

quilkin-bot · 2024-02-29T08:38:47Z

Build Failed 😭

Build Id: 1c410a77-71ec-41ad-b912-206d177686e5

Status: FAILURE

To get permission to view the Cloud Build view, join the quilkin-discuss Google Group.

Filter with the Git Commit 6b706e7 within us-docker.pkg.dev/quilkin/ci//quilkin to see if a development image is available from this build, and can be used for debugging purposes.

Development images are retained for at least 30 days.

quilkin-bot · 2024-02-29T09:02:09Z

Build Succeeded 🥳

Build Id: c6805cdf-9d46-46e0-8aef-44cb07288baa

The following development images have been built, and will exist for the next 30 days:

us-docker.pkg.dev/quilkin/ci/quilkin:0.8.0-dev-6b706e7

To build this version:

git fetch git@github.com:googleforgames/quilkin.git pull/854/head:pr_854 && git checkout pr_854
cargo build

XAMPPRocky requested review from markmandel and koslib as code owners November 15, 2023 08:55

github-actions bot added the size/xl label Nov 15, 2023

XAMPPRocky force-pushed the ep/phoenix branch from a43b7c9 to ca2fd38 Compare November 15, 2023 08:59

XAMPPRocky force-pushed the ep/phoenix branch from ca2fd38 to 5cb0e86 Compare November 15, 2023 09:00

XAMPPRocky force-pushed the ep/phoenix branch from 5cb0e86 to c565a71 Compare November 15, 2023 09:04

XAMPPRocky force-pushed the ep/phoenix branch from c565a71 to 18b1b33 Compare November 16, 2023 08:51

XAMPPRocky force-pushed the ep/phoenix branch from 18b1b33 to 526daaf Compare November 16, 2023 16:50

XAMPPRocky force-pushed the ep/phoenix branch from 526daaf to e2a45ef Compare November 17, 2023 09:35

XAMPPRocky force-pushed the ep/phoenix branch from 1573dab to 90f24de Compare November 18, 2023 10:10

XAMPPRocky force-pushed the ep/phoenix branch from 90f24de to b8e4486 Compare November 22, 2023 17:12

markmandel reviewed Nov 22, 2023

View reviewed changes

XAMPPRocky force-pushed the ep/phoenix branch from f62bbfe to 45fe007 Compare November 23, 2023 20:59

XAMPPRocky mentioned this pull request Nov 25, 2023

Add datacenter discovery #851

Closed

XAMPPRocky and others added 14 commits February 29, 2024 09:32

Use bounded channels to prevent backpressure

b21f9ca

Spawn phoenix task on load balancers

7305580

add update watcher to phoenix

daed714

Add shutdown handler

4840279

formatting

f07c7c5

Slight adjustments

4d64810

ubounded

aec2959

Fix compilation

2a7794b

Oops

d8d7563

Ugly, but it works now

9dcc920

Increase timeout

58a11c4

Add documentation

6b706e7

XAMPPRocky force-pushed the ep/phoenix branch from 9a248c9 to 6b706e7 Compare February 29, 2024 08:33

XAMPPRocky requested a review from Jake-Shadle as a code owner February 29, 2024 08:33

Jake-Shadle approved these changes Feb 29, 2024

View reviewed changes

XAMPPRocky merged commit d22a444 into main Feb 29, 2024
5 checks passed

markmandel deleted the ep/phoenix branch March 11, 2024 06:00

markmandel added kind/feature New feature or request area/operations Installation, updating, metrics etc area/user-experience Pertaining to developers trying to use Quilkin, e.g. cli interface, configuration, etc labels Mar 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Phoenix Network Coordinates #854

Implement Phoenix Network Coordinates #854

XAMPPRocky commented Nov 15, 2023 •

edited

Loading

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Nov 18, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

markmandel left a comment •

edited

Loading

github-actions bot commented Nov 23, 2023

github-actions bot commented Feb 27, 2024

XAMPPRocky commented Feb 27, 2024

Jake-Shadle commented Feb 27, 2024

XAMPPRocky commented Feb 27, 2024

markmandel commented Feb 28, 2024

github-actions bot commented Feb 29, 2024

quilkin-bot commented Feb 29, 2024

quilkin-bot commented Feb 29, 2024

Implement Phoenix Network Coordinates #854

Implement Phoenix Network Coordinates #854

Conversation

XAMPPRocky commented Nov 15, 2023 • edited Loading

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Nov 18, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

markmandel left a comment • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Nov 23, 2023

github-actions bot commented Feb 27, 2024

XAMPPRocky commented Feb 27, 2024

Jake-Shadle commented Feb 27, 2024

XAMPPRocky commented Feb 27, 2024

markmandel commented Feb 28, 2024

github-actions bot commented Feb 29, 2024

quilkin-bot commented Feb 29, 2024

quilkin-bot commented Feb 29, 2024

XAMPPRocky commented Nov 15, 2023 •

edited

Loading

markmandel left a comment •

edited

Loading