Implement Dynamic RPC pick handler

[0x4007]

We want to use Chainlist's list of RPC endpoints. Create a class that will handle this:

• We should import the Chainlist repository as a submodule to stay in sync.
• The URLs should be tested upon initialization (i.e. page load) via Promise.race
• The class should rank sort based on latency.
• The class should drop failed endpoints.
• The class should have a wrapper method to make a network request using the optimal RPC endpoint, and to drop it/automatically rotate to the next best from the list if a network request fails.

We will use this across all of our products that require RPC calls so:

• Make a standalone and polymorphic (frontend and backend compatible) npm module.
• Make sure to have unit tests in Jest.

Original Spec

The code should drop bad RPCs dynamically. I had a prototype implemented where it would delete from local storage if there was an error for example. Later in the code it would only select from local storage options.

This blacklist idea doesn't seem like a robust approach. We can merge it in but it should be handled dynamically. I think I implemented this in an unstable manner on my branch, I vaguely recall.

Originally posted by @pavlovcik in ubiquity/pay.ubq.fi#169 (comment)

[molecula451]

/start

[ubiquibot]

Deadline	Tue, Feb 27, 6:56 AM UTC
Registered Wallet	0x4D0704f400D57Ba93eEa88765C3FcDBD826dCFc4

Tips:

• Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
• Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
• Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.

[molecula451]

looking at the committer solution indeed this blacklist it's not a good idea because it's manually blacklisting each RPC it's senseless

[ubiquibot]

# No linked pull requests to close

[Keyrxng]

this is resolved in ubiquity/pay.ubq.fi#177, the temporary fix was preventing the claim button from flickering on the live portal (since only gnosis is used right now I only blacklisted those that were failing), senseless but effective

We can't know if an rpc call is going to fail until it's attempted, those that fail return no data through const { data } = await API.post("", RPC_BODY);. In logging status and statusText you only get 200 responses. Although in testing mainnet (49 rpcs on sync-erc-permits) some fail but may succeed later whereas the two I added for gnosis makeup three failed calls which I haven't saw succeed once.

So we can't dynamically pop RPCs from the list until they've been tried at least once and even then with no data returned there are no bad latency entries to pop either

I think filtering out the tracking: "yes" category should be considered at least. none-only takes mainnet down to 18 and gnosis down to 7. Limited adds 4 more to gnosis and 7 for mainnet.

chainlist/extraRpcs.js

      {
        url: "https://eth.nodeconnect.org/",
        tracking: "yes",
        trackingDetails: privacyStatement.nodeconnect,
      },
      {
        url: "https://rpc.polysplit.cloud/v1/chain/1",
        tracking: "none",
        trackingDetails: privacyStatement.polysplit,
      },
       {
      	url: "https://eth-mainnet.diamondswap.org/rpc",
      	tracking: "limited",
      	trackingDetails: privacyStatement.diamondswap,
      },

[molecula451]

senseless but effective

it's senseless to send each time a PR to just blacklist a RPC

[0x4007]

this is resolved in ubiquity/pay.ubq.fi#177, the temporary fix was preventing the claim button from flickering on the live portal (since only gnosis is used right now I only blacklisted those that were failing), senseless but effective

We can't know if an rpc call is going to fail until it's attempted, those that fail return no data through const { data } = await API.post("", RPC_BODY);. In logging status and statusText you only get 200 responses. Although in testing mainnet (49 rpcs on sync-erc-permits) some fail but may succeed later whereas the two I added for gnosis makeup three failed calls which I haven't saw succeed once.

So we can't dynamically pop RPCs from the list until they've been tried at least once and even then with no data returned there are no bad latency entries to pop either

I think filtering out the tracking: "yes" category should be considered at least. none-only takes mainnet down to 18 and gnosis down to 7. Limited adds 4 more to gnosis and 7 for mainnet.

chainlist/extraRpcs.js

      {
        url: "https://eth.nodeconnect.org/",
        tracking: "yes",
        trackingDetails: privacyStatement.nodeconnect,
      },
      {
        url: "https://rpc.polysplit.cloud/v1/chain/1",
        tracking: "none",
        trackingDetails: privacyStatement.polysplit,
      },
       {
      	url: "https://eth-mainnet.diamondswap.org/rpc",
      	tracking: "limited",
      	trackingDetails: privacyStatement.diamondswap,
      },

I handled all of those cases in my unstable branch. It starts with a Promise.race and then continues with the fastest functional RPC basically.

Tracking I didn't care about but we can eventually get around to it.

I think what makes the most sense is to actually create a standalone npm module that handles this in any context (browser or node) then we can use it across all of our codebases.

[molecula451]

GPT
Manually dropping and PRing to blacklist a list of RPC (Remote Procedure Call) endpoints that aren't functioning well in a TypeScript environment can be useful initially, especially if there's a specific set of endpoints that consistently cause issues. However, relying solely on manual intervention might become cumbersome and less efficient as the project grows or as new issues arise.

Implementing dynamic handling, where the system automatically detects and responds to faulty endpoints, could indeed be more practical in the long run.

Context: https://chat.openai.com/share/3c64624c-f8dc-4066-ba69-e86c08a75ee0

[Keyrxng]

/start

[ubiquibot]

Deadline	Thu, Mar 7, 7:26 AM UTC
Registered Wallet	0xAe5D1F192013db889b1e2115A370aB133f359765

Tips:

• Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
• Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
• Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.

[Keyrxng]

Should I work from sync-erc-permits, main or development?

[molecula451]

Should I work from sync-erc-permits, main or development?

If you're proficient with sync-erc-permits, consider forking another branch and merging changes from main or development. Then, ensure all changes align with your intended outcome."

[0x4007]

I think it might be easier from scratch although you are definitely welcome to reference my logic. I realized that my implementation was overly complicated when it didn't need to be for our business goals. I only realized after taking a step back and writing this spec lol

[molecula451]

moving this to RPC handler repo

[molecula451]

congrats @Keyrxng even tho i linked your PR, github did not automatically closed perhaps, tho this was targeted to another branch, closing as completed

[ubiquibot]

+ Evaluating results. Please wait...

[rndquu]

The permit was not generated.

@pavlovcik Do we need a manual payout here?

@FernandVEYRIER Could you take a look at the root cause of why the permit was not generated and perhaps create a new issue for this case?

[molecula451]

The permit was not generated.

permit was not generated due to an already known issue

https://github.com/ubiquibot/comment-incentives/actions/runs/8276258518/job/22645535224#step:5:798

[ubiquibot]

+ Evaluating results. Please wait...

[ubiquibot]

[ 84 WXDAI ]

@pavlovcik

Contributions Overview

View	Contribution	Count	Reward
Issue	Specification	1	42.2
Issue	Comment	2	34.4
Review	Comment	2	7.4

Conversation Incentives

Comment	Formatting	Relevance	Reward
We want to use Chainlist's list of RPC endpoints. Create a class...	42.2 h3: count: 1 score: "1" words: 2 li: count: 7 score: "7" words: 97 code: count: 1 score: "1" words: 2	1	42.2
> this is resolved in ubiquity/pay.ubq.fi#177, the temporary fix...	24.6 code: count: 10 score: "10" words: 21	0.685	24.6
I think it might be easier from scratch although you are definit...	9.8	0.715	9.8
> https://rpc-pay.ubq.fi/v1/mainnet I just deleted this while...	5.2	-	5.2
Just make a GitHub action to prove its not env related?...	2.2	-	2.2

[ 30.3 WXDAI ]

@molecula451

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	7	18.1
Review	Comment	7	12.2

Conversation Incentives

Comment	Formatting	Relevance	Reward
looking at the committer solution indeed this blacklist it's not...	2.4	0.75	2.4
> senseless but effective it's senseless to send each time a ...	1.4	0.78	1.4
> GPT Manually dropping and PRing to blacklist a list of RPC ...	1.1	0.765	1.1
> Should I work from sync-erc-permits, main or development...	5.8 code: count: 3 score: "3" words: 5	0.705	5.8
moving this to RPC handler repo...	0.6	0.755	0.6
congrats @Keyrxng even tho i linked your [PR](https://github.com...	3.4 a: count: 1 score: "1" words: 1	0.645	3.4
> The permit was not generated. permit was not generated due ...	3.4 a: count: 1 score: "1" words: 1	0.76	3.4
build still fails @Keyrxng ...	0.4	-	0.4
> and I think Knip is still not fully set up yet, is that right?...	0.6	-	0.6
> Is the run failing maybe because the workflow isn't on the bra...	0.5	-	0.5
> The kebab case workflow, the debug run was a big help yeah ...	0.9	-	0.9
What am I doing wrong? extraRpcs it's part of chainlist s...	1.7	-	1.7
Tested, working good, it looks like https://gnosis-pokt.nodies.a...	4.7	-	4.7
Code looks good and well structured, the package it's working we...	3.4	-	3.4

[ 1330.7 WXDAI ]

@Keyrxng

Contributions Overview

View	Contribution	Count	Reward
Issue	Task	1	1200
Issue	Comment	2	0
Review	Comment	14	130.7

Conversation Incentives

Comment	Formatting	Relevance	Reward
this is resolved in ubiquity/pay.ubq.fi#177, the temporary fix w...	- code: count: 9 score: "0" words: 20	0.81	-
Should I work from ``sync-erc-permits``, ``main`` or ``developme...	- code: count: 3 score: "0" words: 5	0.825	-
dont hold back @molecula451 ...	0.4	-	0.4
idk what the official name will be but I just went with rpc-hand...	1.4	-	1.4
> build still fails @Keyrxng Added the kebab scripts (forgot ...	4.6	-	4.6
Is the run failing maybe because the workflow isn't on the branc...	4.1	-	4.1
> > Is the run failing maybe because the workflow isn't on the b...	1.1	-	1.1
You are importing the unbuilt version, import from /dist/cjs it ...	1.3	-	1.3
> > https://rpc-pay.ubq.fi/v1/mainnet I just deleted this...	8.6	-	8.6
> This code returns the same provider all the time (`https://rpc...	12.7 code: count: 3 score: "3" words: 14	-	12.7
The networkID !== 31337 and includes("localhost") was a fix for ...	4.4	-	4.4
I see that you are importing from ``dist/cjs/rpc-handler`` and n...	23.1 code: count: 4 score: "4" words: 10	-	23.1
Your screenshot made me realize two things: type.d out was ...	24.8 li: count: 7 score: "7" words: 114 code: count: 4 score: "4" words: 7	-	24.8
Your build dir should look like this after running ``yarn build`...	11.8 code: count: 5 score: "5" words: 13	-	11.8
Happy days we are on the same page with it now tho good stuff ...	28.6 code: count: 3 score: "3" words: 3	-	28.6
> > It's failing because I have the request timeout at 500ms > ...	3.8 code: count: 2 score: "2" words: 6	-	3.8

[ 113.6 WXDAI ]

@rndquu

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	1	3.9
Review	Comment	8	109.7

Conversation Incentives

Comment	Formatting	Relevance	Reward
The permit was not generated. @pavlovcik Do we need a manual...	3.9	0.805	3.9
@Keyrxng Trying to run this code: import { RPCHandler ...</a></h6></td><td><details><summary>22</summary> <pre>code: count: 2 score: "2" words: 0 </pre> </details></td><td>-</td><td>22</td></tr><tr><td><h6><a href="https://github.com/ubiquity/rpc-handler/pull/1#issuecomment-1978108689">> You are importing the unbuilt version, import from /dist/cjs i...</a></h6></td><td>0.4</td><td>-</td><td>0.4</td></tr><tr><td><h6><a href="https://github.com/ubiquity/rpc-handler/pull/1#issuecomment-1978119431">@Keyrxng I've installed git submodules via `git submodule up...</a></h6></td><td><details><summary>21.5</summary> <pre>code: count: 3 score: "3" words: 5 </pre> </details></td><td>-</td><td>21.5</td></tr><tr><td><h6><a href="https://github.com/ubiquity/rpc-handler/pull/1#issuecomment-1978136945">@Keyrxng This code returns the same provider all the time (`...</a></h6></td><td><details><summary>8.4</summary> <pre>code: count: 3 score: "3" words: 14 </pre> </details></td><td>-</td><td>8.4</td></tr><tr><td><h6><a href="https://github.com/ubiquity/rpc-handler/pull/1#issuecomment-1980225934">@Keyrxng Check this code: import { HandlerConstructorConfi...	16.6 li: count: 2 score: "2" words: 56 code: count: 5 score: "5" words: 11	-	16.6
> I assume you are just trying to run the script with tsx or som...	22 li: count: 5 score: "5" words: 29 code: count: 6 score: "6" words: 25	-	22
@Keyrxng I haven't touched anything in the code since yesterday ...	14.5 code: count: 2 score: "2" words: 0	-	14.5
> It's failing because I have the request timeout at 500ms > >...	4.3 code: count: 2 score: "2" words: 6	-	4.3

[Keyrxng]

My biggest bounty yet, ty team

[0x4007]

@Keyrxng you think you can swap out the logic in the repos using the old version with your package? I'm not sure which all have it off hand but I assume onboard, audit, and pay have this logic.