Prefect #115
Prefect #115
Conversation
|
New dependencies detected. Learn more about Socket for GitHub ↗︎
|
Dockerfile.prefect
Outdated
|
|
||
| RUN apt-get update \ | ||
| && apt-get -y install libpq-dev gcc \ | ||
| && pip install psycopg2 |
There was a problem hiding this comment.
I don't use the Dockerfile anymore so it's not needed anymore. These are needed if you're using sqlalchemy inside a Docker container.
.gitignore
Outdated
| @@ -129,3 +129,6 @@ dmypy.json | |||
|
|
|||
| # Pyre type checker | |||
| .pyre/ | |||
|
|
|||
| # Vim swap files | |||
| **/*.swp | |||
There was a problem hiding this comment.
This is fine, I guess, but my personal preference would be to have a small .gitignore with project specific content. For local setup, I use .git/info/exclude.
Is this approach with a huge .gitignore better for, say, using dev containers?
There was a problem hiding this comment.
Or a global .gitignore
In ~/.gitconfig:
[core]
excludesfile = /Users/bh2smith/.gitignore_global
which contains things like:
TODO.md
.venv/
.env.bkp
env/
venv/
.DS_Store
.idea/
.vscode/
out/
There was a problem hiding this comment.
Is this approach with a huge .gitignore better for, say, using dev containers?
Not that I know of, I didn't know you could add it to .git/info/exclude. Thanks!
| @@ -0,0 +1,4 @@ | |||
| prefect==3.0.2 | |||
There was a problem hiding this comment.
Alternatively, we can start with a prefect.in and generated prefect.txt right away here.
We could also make that a separate change, but I do not see much momentum for any type of refactoring.
Running it locally requires changing the way we serve the deployment. I'll make a local_deploy.py file to make this easier so you can just run |
There was a problem hiding this comment.
I like the structure of the order_rewards flow. The logic is really clear there.
I do not fully understand the logic behing the batching in time. Some clarifications would help there.
There are a few unused variables here and there.
Otherwise it looks good for finally using prefect systematically.
prefect/deployment.py
Outdated
|
|
||
| load_dotenv() | ||
|
|
||
| def get_last_monday_midnight_utc(): |
There was a problem hiding this comment.
I would have expected to see data uploaded aligned with accounting periods, so Tuesday 00:00:00.
In principle, that should not have an effect on the aggregated table. Uploading new values for an accounting week before payments should be easier with alignment.
prefect/deployment.py
Outdated
|
|
||
|
|
||
| @task | ||
| def get_block_range() -> BlockRange: |
There was a problem hiding this comment.
I have to double check this. It looks as if this approach would not be able to handle the transition between weeks. Or is it uploading data twice for one day of the week?
In either case, some comments would be helpful to explain the behavior.
There was a problem hiding this comment.
At the moment it executes the job once every 30 minutes, but it's very easy to change it. I think Haris recommended 3 hours so I can set it to that. Every time this job runs it re-uploads the entire data for that week, overwriting the current table (for that week). When we get to another week the start block will change and it will instead upload to a new table with a new name.
So on a transition between weeks it will run the query multiple times for that day, and re-upload each time, changing the table when the startblock moves up a week.
prefect/deployment.py
Outdated
| blockrange = get_block_range() | ||
| orderbook = fetch_orderbook(blockrange) | ||
| data = cast_orderbook_to_dune_string(orderbook) | ||
| table_name = upload_data_to_dune(data, blockrange.block_from, blockrange.block_to) |
There was a problem hiding this comment.
One could also use the block range object here. Would look cleaner here but less clean in upload_data_to_dune. Since the latter only uses it for creating a string, it should be fine to look less clean there.
I made some updates, now you should be able to run the prefect server with One note on running the flow locally: the dune update query command requires an enterprise API key, so I don't use that step on the local deploy. It will just run the commands and upload the data to your dune account. |
This PR creates a Prefect deployment to run the dune-sync order rewards job once every 3 hours.