[feature] [BIT 578] speed up metagraph storage query #933
Conversation
camfairchild
requested review from
unconst,
Eugene-hu and
shibshib
and removed request for
Eugene-hu
| """ | ||
| We expect a JSON array of: | ||
| { | ||
| "uid": int, | ||
| "ip": str, | ||
| "ip_type": int, | ||
| "port": int, | ||
| "stake": str(int), | ||
| "rank": str(int), | ||
| "emission": str(int), | ||
| "incentive": str(int), | ||
| "consensus": str(int), | ||
| "trust": str(int), | ||
| "dividends": str(int), | ||
| "modality": int, | ||
| "last_update": str(int), | ||
| "version": int, | ||
| "priority": str(int), | ||
| "last_update": int, | ||
| "weights": [ | ||
| [int, int], | ||
| ], | ||
| "bonds": [ | ||
| [int, str(int)], | ||
| ], | ||
| } | ||
| """ |
There was a problem hiding this comment.
This is helpful given that you are including an external binary.
What about moving this specification to a document in the bin file and translate this comment also to a set of unit tests?
There was a problem hiding this comment.
Both documents; the specification and the tests could be referenced here with links
There was a problem hiding this comment.
Adding unit tests for the binary file is good to:
- localize files and ensure that you have all of them in the repo
- write a specification
There was a problem hiding this comment.
Agree, @camfairchild this should be moved to a proper documentation as well. Nicely done!
If we are getting binaries from there I would suggest to comment it somewhere so we can wire the components and generate the knowledge @camfairchild very nice job!! |
Good idea. I'll add it in the |
|
Perhaps I misunderstood your graphs, but looks like the ifs cache is still a little faster despite being centralized, is that right? Also, you have conflicts ;) |
Nope, you're right. The IPFS is faster, but only when it's up ;) |
ouch. 🗡️ |
|
@camfairchild why is this still marked do not merge? any blockers? |
|
Going to refactor into an external pypi package so the binary can be distributed easier |
This PR speeds up the storage call made by
subtensor.neuronsin thesubtensor.use_neurons_fastfunction.This feature works by bundling a nodejs binary with the polkadotjs API.
This binary is a CLI that implements the
sync_and_save --filename <default:~/.bittensor/metagraph.json> --block_hash <default:latest>command.This syncs the metagraph at the blockhash and saves it to a json file.
The speed-up is quite significant, below is a test run of the manual without the fix, with the ipfs cache, and with the fix.
And below is the IPFS cache sync versus the manual sync (with fix)
A pro of this is that it removes the need for a centralized IPFS cache of the metagraph.
A downside of this fix is that the binaries with nodejs bundled use ~50MB each (one linux, one macos).
There is currently no binary for windows, but I'm not certain this should be included anyway, as we only support linux/macos.
Another pro of this fix is it works on both nobunaga and nakamoto, and can be adapted to any network.
This also leaves room for adding other large substrate queries and working further with the polkadot js api.