feat: run inserts in batch and in parallel when processing new block #1818
Conversation
Signed-off-by: bestmike007 <i@bestmike007.com>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #1818 +/- ##
==========================================
+ Coverage 71.61% 71.76% +0.14%
==========================================
Files 91 91
Lines 11771 11817 +46
Branches 2612 2616 +4
==========================================
+ Hits 8430 8480 +50
+ Misses 3188 3184 -4
Partials 153 153 ☔ View full report in Codecov by Sentry. |
Signed-off-by: bestmike007 <i@bestmike007.com>
Signed-off-by: bestmike007 <i@bestmike007.com>
… min value Signed-off-by: bestmike007 <i@bestmike007.com>
Signed-off-by: bestmike007 <i@bestmike007.com>
Signed-off-by: bestmike007 <i@bestmike007.com>
There was a problem hiding this comment.
This looks good to me. We'd like to cut a pre-release Docker image with this PR and run a few tests so we can measure how it performs in the kubernetes pods, especially when interacting with postgres over a network connection.
I'll create a beta branch and target your PR into it so we can merge there and we can run tests on our end @bestmike007 , sound good? cc @zone117x
|
It'd be better if you run with different concurrency values, and then decide the default value. |
## [7.8.0-beta.1](v7.7.1...v7.8.0-beta.1) (2024-01-11) ### Features * run inserts in batch and in parallel when processing new block ([#1818](#1818)) ([86dfdb5](86dfdb5)) ### Bug Fixes * log block event counts after processing ([#1820](#1820)) ([9c39743](9c39743)), closes [#1819](#1819) [#1819](#1819)
|
🎉 This PR is included in version 7.8.0-beta.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
## [7.8.0](v7.7.2...v7.8.0) (2024-01-23) ### Features * add `/extended/v2/smart-contracts/status` endpoint ([#1833](#1833)) ([3535c11](3535c11)) * run inserts in batch and in parallel when processing new block ([#1818](#1818)) ([86dfdb5](86dfdb5)) * upgrade to node 20, use bookworm-slim image ([#1832](#1832)) ([0a42109](0a42109)) ### Bug Fixes * change all HASH indexes to BTREE to optimize writes ([#1825](#1825)) ([234936b](234936b)) * log block event counts after processing ([#1820](#1820)) ([9c39743](9c39743)), closes [#1819](#1819) [#1819](#1819) * optimize re-org queries and indexes ([#1821](#1821)) ([5505d35](5505d35)) * parallelize re-org update queries ([#1835](#1835)) ([340a304](340a304))
|
🎉 This PR is included in version 7.8.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Description
This PR improves the new block process improvements by means of running inserts in batch and in parallel.
Previously when the event server process a new block, it roughly process less than 10 transactions per second.
When tested with concurrency=100, it processes more than 30 transactions per second.
Type of Change
Performance improvements.
Does this introduce a breaking change?
No.
Are documentation updates required?
No.
Testing information
No changes required.
Checklist
npm run testpasses