🔷 [ProjectTracking] Implement support for optimistic block finality
#186
Assignees
Labels
Comments
This was referenced Mar 1, 2024
github-merge-queue bot
pushed a commit
to near/nearcore
that referenced
this issue
Mar 1, 2024
…ng Optimistic Blocks (#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
gmilescu
changed the title
optimistic block finalityoptimistic block finality
|
UPDATE 2022-03-11 For the last week, we have made good progress. The PR that introduces support for the optimistic block finalities is under review. Ref: #197 We aim to wrap up the PR earlier this week and release version 0.2.0 of the ReadRPC for deployment on the testnet as soon as possible. We will require help from the SRE team since we need to spin up a bit different Some changes to the build pipeline might be involved, but I expect them to be insignificant. Once assured that the support of optimistic finalities works on testnet we will start a roll-up for the mainnet. We aim to finish it until the end of this week, though it is an optimistic estimate 😅 |
Update 2024-03-19Even though last week was very busy with the resharding release of the nearcore 1.37.0 (and this one will be fun with 1.38.0 resharding), we managed to progress with the infra on our own while the SRE team was busy. Anyway, we had to ask @ecp88 for help when my experience wasn't enough. Today, we enabled the optimistic block finalities feature on the traffic mirroring instance 🎉 The planThe plan is to monitor the feature on the traffic mirroring instance a bit, fix any mismatches, if any, and then proceed to enable the feature for the We wanted to finish this sooner, but the release train caught us off guard. Obviously, we were blocked without the SRE team's help. We seem unblocked right now and want to test the feature before rolling it out to production. |
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
Mar 20, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
khorolets
pushed a commit
to near/nearcore
that referenced
this issue
Apr 9, 2024
…ng Optimistic Blocks (#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
Apr 9, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
Apr 23, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
khorolets
pushed a commit
to near/nearcore
that referenced
this issue
Apr 23, 2024
…ng Optimistic Blocks (#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
Update 2024-04-24When we got the traffic on the staging instance, we observed a major performance degradation, which caused the node to become stuck. It is a blocker for us, see #236 We will be able to get back to delivering the optimistic block finalities support after the blocker resolution. |
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
May 13, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
May 13, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
May 13, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
May 13, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
May 14, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
May 14, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
May 14, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
kobayurii
added a commit
to kobayurii/nearcore
that referenced
this issue
May 14, 2024
…ng Optimistic Blocks (near#10692) Hello everyone! This pull request addresses two urgent tasks related to our project's [Read RPC](https://github.com/near/read-rpc) functionality. [Issue 1](near/read-rpc#188): Making our state-indexer compatible with the near-indexer to eliminate the 3-second delay caused by [Lake-framework](https://github.com/near/near-lake-framework-rs). This compatibility enhancement ensures smoother and more efficient operation of our system by removing unnecessary delays. [Issue 2](near/read-rpc#186): Enhancing our service to handle requests for optimistic blocks directly, without proxying to nodes. This optimization streamlines the process, improving response times and overall system performance. To achieve these tasks, modifications have been made to the near-indexer. Specifically, the `build_streamer_message` method has been made public to facilitate the creation of `StreamerMessage` for optimistic blocks. This pull request aims to enhance the reliability, efficiency, and performance of our Read RPC functionality, addressing critical needs within our project.
|
The feature is done. It can work, deployed to production, but was turned off, because of high load on ScyllaDB leading to increase of latency. We are investigating it. See #236 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Goals
Background
Recently we've added additional metrics to measure the number of proxy requests from the ReadRPC to native one. Additionally, we have a metric that counts how many of the requests are for the blocks with optimistic finality. This finality is not yet supported in the ReadRPC. The metrics revealed that a significant portion of the requests are for the optimistic blocks - 85% of all requests. This is a significant portion of the requests that could be served by the ReadRPC if it supported optimistic blocks. Without the optimistic finality support, the ReadRPC is not showing it's potential as a regular RPC for the NEAR blockchain.
I haven't found any documentation about optimistic finality except for this short mention in the NEAR RPC documentation:
Why should NEAR One work on this
Enabling optimistic block support in ReadRPC presents a significant opportunity to enhance dApp responsiveness and reduce the load on NEAR's native RPC. This is crucial for applications that require the most up-to-date blockchain data to ensure dynamic and responsive user interactions.
What needs to be accomplished
Main use case
Improving data responsiveness for dApps by providing quicker access to blockchain data through optimistic blocks, thereby enhancing user experience and reducing the load on native RPC services.
Links to external documentation and discussions
Estimated effort
Engineers assigned to this initiative: @khorolets and @kobayurii
The initial effort estimate is about 1-1.5 PM (person months)
Assumptions
This project does not make specific assumptions.
Pre-requisites
N/A
Out of scope
N/A
Task list:
Tasks
state-indexercompatibility with NEAR Lake and NEAR Indexer Framework #188near-state-indexer#199The text was updated successfully, but these errors were encountered: