Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf(rpc): optimistically retrieve block if near the tip on eth_getLogs #11582

Merged
merged 8 commits into from
Oct 9, 2024
Merged

perf(rpc): optimistically retrieve block if near the tip on eth_getLogs #11582

merged 8 commits into from
Oct 9, 2024

Conversation

joshieDo
Copy link
Collaborator

@joshieDo joshieDo commented Oct 8, 2024 •
edited
Loading

closes #11553

If the block is near the tip (4 blocks), then query it from cache instead of requesting every block transaction from disk

@joshieDo joshieDo added C-perf A change motivated by improving speed, memory usage or disk footprint A-rpc Related to the RPC implementation labels Oct 8, 2024
@github-actions github-actions bot added the C-enhancement New feature or request label Oct 8, 2024
mattsse
mattsse requested changes Oct 9, 2024
View reviewed changes
Copy link
Collaborator

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a pretty clever solution

only have some nits

crates/rpc/rpc-eth-types/src/logs_utils.rs Outdated Show resolved Hide resolved
crates/rpc/rpc-eth-types/src/logs_utils.rs Outdated Show resolved Hide resolved
crates/rpc/rpc-eth-types/src/logs_utils.rs Outdated Show resolved Hide resolved
crates/rpc/rpc/src/eth/filter.rs Outdated
Comment on lines 533 to 547
async fn receipts_and_maybe_block(
&self,
block_num_hash: &BlockNumHash,
best_number: u64,
) -> Result<Option<(Arc<Vec<Receipt>>, Option<SealedBlock>)>, EthFilterError> {
let receipts_block = if (best_number - 4..=best_number).contains(&block_num_hash.number) {
self.eth_cache
.get_block_and_receipts(block_num_hash.hash)
.await?
.map(|(b, r)| (r, Some(b)))
} else {
self.eth_cache.get_receipts(block_num_hash.hash).await?.map(|r| (r, None))
};
Ok(receipts_block)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can even make this a function of the cache itself, that only fetches the receipts, and returns None for the block if not cached.

but can do that in a good first issue

crates/rpc/rpc/src/eth/filter.rs Outdated Show resolved Hide resolved
@joshieDo joshieDo mentioned this pull request Oct 9, 2024
Open
@joshieDo joshieDo added this pull request to the merge queue Oct 9, 2024
Merged via the queue into main with commit b787d9e Oct 9, 2024
37 checks passed
@joshieDo joshieDo deleted the joshie/append-logs branch October 9, 2024 14:01
Thegaram added a commit to scroll-tech/reth that referenced this pull request Oct 10, 2024
@Thegaram @joshieDo
@shekhirin @mattsse @0x00101010 @emhane @klkvr @ewoolsey @tcoratger @fgimenez @varun-doshi @garwahl @greged93 @jsvisa @Parikalp-Bhardwaj @github-actions @github-merge-queue @Rjected @rkrasiuk @crazykissshout @edisontim @No0key @stevencartavia @debjit-bw @onbjerg @loocapro @devthejohn
chore: merge upstream b787d9e (#18)
a6ca484 
* chore(provider): use `get_in_memory_or_storage` on `transactions_by_block_range` (paradigmxyz#11453)

* chore(exex): adjust WAL gauge metric names (paradigmxyz#11454)

* chore(provider): use `get_in_memory_or_storage_by_block` on `fn block_body_indices` (paradigmxyz#11452)

* chore(provider): find `last_database_block_number` with `BlockState` anchor instead (paradigmxyz#11455)

* feat: add metrics for failed deliveries (paradigmxyz#11456)

* chore: release 1.0.8 (paradigmxyz#11457)

* chore(provider): use `block_ref` instead on `BlockState` (paradigmxyz#11458)

* feat(rpc): Add codes in execution witness return (paradigmxyz#11443)

* fix: ensure the request's gas limit does not exceed the target gas limit (paradigmxyz#11462)

* feat(grafana): ExEx WAL (paradigmxyz#11461)

* fix: use correct rpc errors (paradigmxyz#11463)

* chore(provider): clone after filtering on `sealed_headers_while`  (paradigmxyz#11459)

      Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* Map `TransferKind::EofCreate` => `OperationType::OpEofCreate` (paradigmxyz#11090)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* fix: windows build (paradigmxyz#11465)

* chore: use `block_ref` on `CanonicalInMemoryState`  (paradigmxyz#11467)

* chore(rpc): remove include_preimage param on debug_execution_witness (paradigmxyz#11466)

* feat: make addons stateful (paradigmxyz#11204)

Co-authored-by: Arsenii Kulikov <klkvrr@gmail.com>

* Relax Trait Bounds on TransactionPool::Transaction and EthPoolTransaction (paradigmxyz#11079)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* test: add unit tests for `CanonicalChain` (paradigmxyz#11472)

* feat(perf): integrate OnStateHook in executor (paradigmxyz#11345)

* feat: cleaned up prepare_call_env() (paradigmxyz#11469)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* chore: use block.body directly (paradigmxyz#11474)

* feat: Add metrics to track transactions by type in txpool  (paradigmxyz#11403)

Co-authored-by: garwah <garwah@garwah>
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* chore: op chainspec (paradigmxyz#11415)

* chore(exex): more backfill debug logs (paradigmxyz#11476)

* fix(exex): use thresholds in stream backfill (paradigmxyz#11478)

* chore(db): capture tx opening backtrace in debug mode (paradigmxyz#11477)

* chore(sdk): `SealedHeader` generic over header (paradigmxyz#11429)

* chore(lint): fix lint primitives (paradigmxyz#11487)

* chore(provider): add more test coverage on `BlockchainProvider::*by_tx_range` queries (paradigmxyz#11480)

* test: ensure default hash matches (paradigmxyz#11486)

* chore: rm deposit contract config for op (paradigmxyz#11479)

* chore(lint): fix lint storage (paradigmxyz#11485)

* chore(provider): add more test coverage on `BlockchainProvider::*by_block_range` queries (paradigmxyz#11488)

* fix(rpc-eth-types): incorrect error msg(; -> :) (paradigmxyz#11503)

Signed-off-by: jsvisa <delweng@gmail.com>

* Reexport optimism specific crates from `op-reth` (paradigmxyz#11499)

* feat: rpc replace function created (paradigmxyz#11501)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* Add metrics for failed deliveries to Grafana dashboard (paradigmxyz#11481)

* chore: Remove duplicate EthereumChainSpecParser in favor of existing EthChainSpecParser  (paradigmxyz#11412)

Co-authored-by: garwah <garwah@garwah>
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* chore(lint): fix `clippy::needles_lifetimes` (paradigmxyz#11496)

* chore: rm from genesis impl (paradigmxyz#11509)

* fix: cap gas limit properly (paradigmxyz#11505)

* feat: add PoolBuilderConfigOverrides (paradigmxyz#11507)

* feat: expose Op node network_config helper (paradigmxyz#11506)

* chore(deps): weekly `cargo update` (paradigmxyz#11518)

Co-authored-by: github-merge-queue <118344674+github-merge-queue@users.noreply.github.com>

* test: add unit tests for `PruneLimiter` (paradigmxyz#11517)

* feat(provider): add `test_race` to `BlockchainProvider2` tests (paradigmxyz#11523)

* fix(tree): make state methods work for historical blocks (paradigmxyz#11265)

Co-authored-by: Roman Krasiuk <rokrassyuk@gmail.com>
Co-authored-by: Federico Gimenez <fgimenez@users.noreply.github.com>

* rpc: use `eth_api()` method (paradigmxyz#11516)

* chore: delete rpc-types (paradigmxyz#11528)

* feat: add get_highest_tx_by_sender to pools (paradigmxyz#11514)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* ci: add `windows` cargo check  (paradigmxyz#11468)

* fix: acquire permit first (paradigmxyz#11537)

* chore: dont fail on ttd (paradigmxyz#11539)

* grafana: add metrics of all transactions in pool by type  (paradigmxyz#11515)

Co-authored-by: Emilia Hane <emiliaha95@gmail.com>
Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>

* feat(exex): subscribe to notifications with head using `ExExContext` (paradigmxyz#11500)

* chore: enforce window (paradigmxyz#11540)

* Refactor get_payload_bodies_by_hash_with to be non-blocking (paradigmxyz#11511)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* Introduce Eth PayloadTypes Impl (paradigmxyz#11519)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* fix(op-reth): add jemalloc feature to optimism-cli for version (paradigmxyz#11543)

* fix(grafana): remove rate function from panel "Transactions by Type in Pool" (paradigmxyz#11542)

* chore: move ethfiltererror (paradigmxyz#11552)

* chore: rm redundant type hint (paradigmxyz#11557)

* Introduce Op PayloadTypes Impl (paradigmxyz#11558)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* chore: chain manual serialisation implementation (paradigmxyz#11538)

* chore: rm unused optimism feature (paradigmxyz#11559)

* feat(trie): expose storage proofs (paradigmxyz#11550)

* chore: rm unused optimism feature from compat (paradigmxyz#11560)

* Added InternalBlockExecutionError to execute.rs exports (paradigmxyz#11525)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* docs(exex): include code for ExEx book from real files (paradigmxyz#11545)

* fix: actually configure the custom gas limit (paradigmxyz#11565)

* chore: relax trait bound for `EthTransactions` (paradigmxyz#11571)

* fix(provider): fix sub overflow on `tx_range` queries for empty blocks (paradigmxyz#11568)

* chore(provider): add more test coverage on `BlockchainProvider` non-range queries (paradigmxyz#11564)

* feat: adding a new method to network config builder (paradigmxyz#11569)

* chore: rm unused optimism feature from engine api (paradigmxyz#11577)

* chore: replace some revm deps (paradigmxyz#11579)

* chore: rm bad cap function (paradigmxyz#11562)

* feat: impl `Encodable2718` and `Decodable2718` for `PooledTransactionsElement` (paradigmxyz#11482)

* fix(exex): exhaust backfill job when using a stream (paradigmxyz#11578)

* fix: in-memory trie updates pruning (paradigmxyz#11580)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* chore(providers): test race condition on all `BlockchainProvider2` macro tests (paradigmxyz#11574)

* chore: also derive arb for test (paradigmxyz#11588)

* chore: bump alloy primitives 0 8 7 (paradigmxyz#11586)

* chore(ci): remove expected failures related to checksummed addresses (paradigmxyz#11589)

* chore(rpc): use `block_hash` instead on fetching `debug_trace_block` block (paradigmxyz#11587)

* feat: add mul support for SubPoolLimit (paradigmxyz#11591)

Co-authored-by: Dan Cline <6798349+Rjected@users.noreply.github.com>

* docs: delete missing part path (paradigmxyz#11590)

Co-authored-by: Dan Cline <6798349+Rjected@users.noreply.github.com>

* fix: simplify reorg handling (paradigmxyz#11592)

* fix: use original bytes for codes (paradigmxyz#11593)

* perf(rpc): use `Arc<BlockWithSenders` on `full_block_cache` (paradigmxyz#11585)

* fix: active inflight count (paradigmxyz#11598)

* fix(net): max inflight tx reqs default (paradigmxyz#11602)

* fix: set deposit gasprice correctly (paradigmxyz#11603)

* fix: set system tx correctly (paradigmxyz#11601)

* fix(trie): prefix set extension (paradigmxyz#11605)

* feat: add tx propagation mode (paradigmxyz#11594)

* feat: add helper function to provde the tx manager config (paradigmxyz#11608)

* fix(grafana): set instance variable from `reth_info` metric (paradigmxyz#11607)

* fix(net): add concurrency param from config to `TransactionFetcherInfo` (paradigmxyz#11600)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

* perf(rpc): optimistically retrieve block if near the tip on `eth_getLogs` (paradigmxyz#11582)

* update fork base commit

* remove new book tests

---------

Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: joshieDo <93316087+joshieDo@users.noreply.github.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
Co-authored-by: Francis Li <francis.li@coinbase.com>
Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
Co-authored-by: Arsenii Kulikov <klkvrr@gmail.com>
Co-authored-by: Eric Woolsey <ewoolsey@ualberta.ca>
Co-authored-by: Thomas Coratger <60488569+tcoratger@users.noreply.github.com>
Co-authored-by: Federico Gimenez <fgimenez@users.noreply.github.com>
Co-authored-by: Varun Doshi <61531351+varun-doshi@users.noreply.github.com>
Co-authored-by: garwah <14845405+garwahl@users.noreply.github.com>
Co-authored-by: garwah <garwah@garwah>
Co-authored-by: greged93 <82421016+greged93@users.noreply.github.com>
Co-authored-by: Delweng <delweng@gmail.com>
Co-authored-by: Parikalp Bhardwaj <53660958+Parikalp-Bhardwaj@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-merge-queue <118344674+github-merge-queue@users.noreply.github.com>
Co-authored-by: Dan Cline <6798349+Rjected@users.noreply.github.com>
Co-authored-by: Roman Krasiuk <rokrassyuk@gmail.com>
Co-authored-by: crazykissshout <crazykisss111@gmail.com>
Co-authored-by: tedison <76473430+edisontim@users.noreply.github.com>
Co-authored-by: David <39963997+No0key@users.noreply.github.com>
Co-authored-by: Emilia Hane <emiliaha95@gmail.com>
Co-authored-by: Steven <112043913+stevencartavia@users.noreply.github.com>
Co-authored-by: Debjit Bhowal <debjitbhowal.db@gmail.com>
Co-authored-by: Oliver <onbjerg@users.noreply.github.com>
Co-authored-by: Luca Provini <luca.provini@usemerkle.com>
Co-authored-by: John <devthejohn@gmail.com>
vandenbogart pushed a commit to testmachine-ai/reth that referenced this pull request Oct 14, 2024
@joshieDo
perf(rpc): optimistically retrieve block if near the tip on `eth_getL…
c25cc57 
…ogs` (paradigmxyz#11582)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mattsse mattsse mattsse approved these changes

@Rjected Rjected Awaiting requested review from Rjected Rjected is a code owner

@emhane emhane Awaiting requested review from emhane emhane is a code owner

Assignees
No one assigned
Labels
A-rpc Related to the RPC implementation C-enhancement New feature or request C-perf A change motivated by improving speed, memory usage or disk footprint
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Specialize append_matching_block_logs for when we the block is available
2 participants
@joshieDo @mattsse