Load balancing does not return first successful match immediately (or seems not to be parallelized)

[peperunas]

Output of the following commands:

./dnscrypt-proxy -version

$ dnscrypt-proxy -version                                                                                                                                                                                                                                                                                                                    
2.0.45

./dnscrypt-proxy -check

[2021-06-21 13:10:44] [NOTICE] dnscrypt-proxy 2.0.45
[2021-06-21 13:10:44] [NOTICE] Source [relays] loaded
[2021-06-21 13:10:44] [NOTICE] Source [public-resolvers] loaded
[2021-06-21 13:10:44] [NOTICE] Configuration successfully checked

./dnscrypt-proxy -resolve example.com

Resolving [example.com] using 127.0.0.1 port 5353

Resolver      : 78.129.140.65

Canonical name: example.com.

IPv4 addresses: 93.184.216.34
IPv6 addresses: 2606:2800:220:1:248:1893:25c8:1946

Name servers  : a.iana-servers.net., b.iana-servers.net.
DNSSEC signed : yes
Mail servers  : 1 mail servers found

HTTPS alias   : -
HTTPS info    : -

Host info     : -
TXT records   : 8j5nfqld20zpcyr8xjw0ydcfq9rk8hgm, v=spf1 -all

What is affected by this bug?

The load balancer (with different strategies) seems to be waiting for all queries to be executed first, without returning the first match. This makes the resolver very slow.

When does this occur?

When resolving any query. Used all the load balancing strategies to check if the issue was related to a specific one but it is not.

Where does it happen?

On any domain that has not been cached before.

How do we replicate the issue?

1. Setup a list of resolvers
2. Setup one of the load balancing strategies (in the example here, p2)
3. Execute a query

Expected behavior (i.e. solution)

Return the first (fastest) response from one of the resolvers.

Other Comments

It seems that the load balancer does not resolve in parallel and/or waits for the requests to either fail or timeout before returning a previous successful query.

[lifenjoiner]

dnscrypt-proxy/dnscrypt-proxy/example-dnscrypt-proxy.toml

Lines 142 to 144 in 4caa7b6

	## Load-balancing strategy: 'p2' (default), 'ph', 'p<n>', 'first' or 'random'
	## Randomly choose 1 of the fastest 2, half, n, 1 or all live servers by latency.
	## The response quality still depends on the server itself.

Randomly choose 1 of ... and then query to it.

[peperunas]

I think this is exactly the current behaviour. I believe that parallelizing and returning the first match would improve performance significantly. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Monday, June 21st, 2021 at 1:14 PM, lifenjoiner ***@***.***> wrote: https://github.com/DNSCrypt/dnscrypt-proxy/blob/4caa7b6d64f4126ec89021a6ef13ffc62a215d6e/dnscrypt-proxy/example-dnscrypt-proxy.toml#L142-L144 Randomly choose 1 of ... and then query to it. — You are receiving this because you authored the thread. Reply to this email directly, [view it on GitHub](#1757 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABOA72ZLMDJVG25SMK5FGB3TT4UJFANCNFSM47BMGR4Q).

[lifenjoiner]

If you are really interested in parallelizing, try https://github.com/lifenjoiner/dnsforwarder with one more tier 😜

[jedisct1]

Querying all resolvers simultaneously can cause congestion and produce irrelevant measurements. It would also send a large amount of connections to relays at the same time, which would also return wrong results, or trigger rate limits.

The current behavior is, like any benchmark, to test one thing at a time. However, as soon as at least one resolver is reachable, queries are handled. So, at startup time, initial connections may not be using the most optimal resolver, but there's very little delay before the configuration is usable.

Once all resolvers have been benchmarked, the ones with the lowest latencies are used instead.

[peperunas]

This is an example of a query that took around 16 seconds to be executed with the p2 strategy.

[2021-06-21 14:27:18]        127.0.0.1        zkp.science        A        NETWORK_ERROR        151ms        nextdns
[2021-06-21 14:27:18]        127.0.0.1        zkp.science        AAAA        NETWORK_ERROR        152ms        nextdns
[2021-06-21 14:27:23]        127.0.0.1        zkp.science        A        PASS        103ms        nextdns
[2021-06-21 14:27:23]        127.0.0.1        zkp.science        AAAA        NETWORK_ERROR        151ms        dns.digitale-gesellschaft.ch-2
[2021-06-21 14:27:28]        127.0.0.1        zkp.science        AAAA        NETWORK_ERROR        151ms        dns.digitale-gesellschaft.ch-2
[2021-06-21 14:27:33]        127.0.0.1        zkp.science        AAAA        PASS        118ms        dns.digitale-gesellschaft.ch-2

You can see that after a NETWORK_ERROR, it takes around 5 seconds to try again with another server. Is there a way to tune this timeout?

Thanks.

[jedisct1]

## How long a DNS query will wait for a response, in milliseconds.
## If you have a network with *a lot* of latency, you may need to
## increase this. Startup may be slower if you do so.
## Don't increase it too much. 10000 is the highest reasonable value.

timeout = 5000

Latency of DNS queries has a very high variance, so it takes time to learn a reasonable sorting order for these resolvers. Every time there's a network error, the resolver gets a penalty, which decreases its probability of being chosen for further queries.

But if some servers are slow or unreliable, I'd suggest disabling them instead (disabled_server_names=[...])

[peperunas]

I am already handling that option, I thought other timeouts may have been used throughout the resolving process. I will try to disable some of the servers then. Thank you very much. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Monday, June 21st, 2021 at 1:49 PM, Frank Denis ***@***.***> wrote: # # How long a DNS query will wait for a response, in milliseconds. # # If you have a network with *a lot* of latency, you may need to # # increase this. Startup may be slower if you do so. # # Don't increase it too much. 10000 is the highest reasonable value. timeout = 5000 Latency of DNS queries has a very high variance, so it takes time to learn a reasonable sorting order for these resolvers. Every time there's a network error, the resolver gets a penalty, which decreases its probability of being chosen for further queries. But if some servers are slow or unreliable, I'd suggest disabling them instead (disabled_server_names=[...]) — You are receiving this because you authored the thread. Reply to this email directly, [view it on GitHub](#1758 (reply in thread)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABOA725U6ESSQEVQDZV2SRDTT4YPBANCNFSM47BP2LFA).