-
Notifications
You must be signed in to change notification settings - Fork 618
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refacor(indexer): Add local cache for delayed local receipts. Metrics…
…, log clean up and improvements (#10798) Today on testnet, we encountered very weird behavior from the indexers. Indexing BPS dropped to 0.004 blocks per second. This speed is unacceptable, especially when indexing the blocks already synced by the node. After an investigation, we found the suspect. The most obvious suspect was the process of looking for the local delayed receipt. This kind of receipt is treated differently by the indexer. On the indexer side, we tend to include them in the `StreamerMessage` while `nearcore` avoids storing them to save some space. During some special periods (e.g., congestion), the number of local delayed receipts grows, leading to extra work by the indexer. This PR introduces a local (in-memory) cache for the indexer instance to keep such receipts until they are needed. This should prevent the example from having to walk back through history. When we miss the receipt, we try to find it in previous blocks. Not knowing where to look exactly, we go back block by block until we find it. However, we limit this walk up to 1000 blocks because otherwise, the indexer node would be stuck forever. By the way, that's exactly what it looks like. * Add local cache to keep local receipts if we don't observe `ExecutionOutcome` for them in the same block (Local Delayed Receipts) * Add a metric to keep track of how far in history we go back when we do * Clean up and improve some debug messages. I was stupid and didn't know what I was doing when I coded them three years ago 🤦♂️ * Add extra debug message when they were missing
- Loading branch information
Showing
8 changed files
with
1,486 additions
and
1,324 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.