Analytics service

[hawyar]

• The crawler crawls historical data starting from the head of the chain at the time of starting the crawler till the genesis block.
• The streamer starts from the current head + 1 (because the crawler is inclusive it will process the current head)
• The exchange service provides historical and current exchange price for a given coin and currency using third party APIs.

The analytics service introspects AWS Glue to find the bucket and the table provisioned by the CDK. This allows using the analytics service for different versions. @shanejearley we just have to make sure we preserve the resources for some time if we want to keep previous runs.

Added a go.work file at the root and the crawler (with the streamer) as a module. This allows running the crawler from the root

Run

go run github.com/consensusnetworks/crawler

Test

go test github.com/consensusnetworks/crawler

TODO:

• Update the crawler run scripts to run from root.
• Update the required env variables to be available.
• Check gas fee calc logic with @elizyoung0011

This PR is in progress, I will merge it after resolving some questions with Liz.

[shanejearley]

Right now, .env and ETHERUEM_RPC (typo) is required.

Maybe we can set a default RPC (local 8545), default network (Goerli), or default stage (dev). The local dev script goes about a version of this.

Would just be nice to have no extra config/env needed for test runs.

[shanejearley]

Near-term, we can set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the env. Longer-term, we can tighten things up with a Fargate service deployment of the crawler (so we can make the analytics resources like buckets and tables private) and just add the service security group to those resources. The users service has a similar approach now.

[shanejearley]

The go.work did fix the VSCode extension issue, beautiful.

[hawyar]

Right now, .env and ETHERUEM_RPC (typo) is required.

Maybe we can set a default RPC (local 8545), default network (Goerli), or default stage (dev). The local dev script goes about a version of this.

Would just be nice to have no extra config/env needed for test runs.

I will remove the .env as i only needed it when developing. The crawler expects a url and it will determine the network so we can make the crawler use dev resource for goerli and prod for mainnet (now its hardcoded to dev resource)