Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IoTeX support to crawler and push events to test bucket for Athena sandbox #55

Closed
shanejearley opened this issue Jul 5, 2022 · 4 comments
Assignees

Comments

@shanejearley
Copy link
Contributor

No description provided.

@shanejearley
Copy link
Contributor Author

shanejearley commented Jul 14, 2022

@hawyar further subtask breakdown, apologies yet again for starting with such a large scope

  • Upload events.json to the base directory as you have them (stake create)

@shanejearley shanejearley changed the title Add IoTeX essentials to parma crawler and push events to test bucket for Athena sandbox Add IoTeX support to crawler and push events to test bucket for Athena sandbox Jul 25, 2022
@shanejearley
Copy link
Contributor Author

shanejearley commented Jul 25, 2022

@hawyar let's use block filenames to start, so a flat directory with {blockid}-events.json files. The event count per block and data amount per event will be small and our query patterns will be more date/time focused – so we won't benefit much from the block partitions.

Instead, we'll implement partitions of our events by year, month, and day. We can also define these ahead of time in the Glue Metastore using the CDK. But we have no need to implement that just yet.

@shanejearley
Copy link
Contributor Author

@hawyar let's further narrow this issue in on completing the event schema (1st draft to support all IoTeX Actions) and then adding those events to the offline, non-streaming version of the crawler. Our goal is to get a full dataset (to date, not real-time) in the event bucket.

Then we'll have you turn your focus back to block-by-block streaming. Then we'll attack Ethereum with a solid foundation.

@shanejearley
Copy link
Contributor Author

Ok, my bad, there is already an issue dedicated to finishing the IoTeX action support. This one is complete 🍾

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants