This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
client/authority-discovery: Throttle DHT requests #7018
client/authority-discovery: Throttle DHT requests #7018
Changes from all commits
d959cdb
ebbe005
077d2a9
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't it better to split this between two counters
requests_started_total
andrequests_finished_total
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure I understand your question correctly. Let me try.
With this pull request authority discovery publishes (among others) the following 3 metrics:
authority_discovery_authority_address_requests_pending
authority_discovery_authority_addresses_requested_total
authority_discovery_dht_event_received
Your suggested
requests_started_total
metric already exists withauthority_discovery_authority_addresses_requested_total
. Your suggestedrequests_finished_total
already exists withsum(authority_discovery_dht_event_received{name=~"value_found|value_not_found") by (name)
.We could split
authority_discovery_authority_address_requests_pending
into two metrics in order to not miss spikes in between Prometheus scrapes, but I doubt that is worth the complexity tracking the two causes.Does that make sense @tomaka?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the value of
..._authority_address_requests_pending
is always equal to..._authority_addresses_requested_total - sum(..._dht_event_received{name=~"value_found|value_not_found") by (name)
, then what is the reason for having it?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, now I get the confusion.
There is a difference between pending, in-flight and finished lookups.
Every 10 minutes the authority discovery module requests the current and next authority set from the runtime and overrides
pending_lookups
accordingly. It then takesMAX_IN_FLIGHT_LOOKUPS
lookups frompending_lookups
calls theNetworkService::get_record
and tracks those in-flight lookups inin_flight_lookups
. Whenever a lookup succeeds another lookup is popped off frompending_lookups
and passed to the network and inserted intoin_flight_lookups
.authority_discovery_authority_address_requests_pending
tracks the amount of pending, not yet started lookups...._authority_addresses_requested_total - sum(..._dht_event_received{name=~"value_found|value_not_found") by (name)
tracks the amount of in-flight, started, not yet finished lookups.Does that make sense @tomaka?