This repository has been archived by the owner on Dec 20, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 1
Prefect #115
Merged
Merged
Prefect #115
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
60bd52d
Add prefect deployment
bram-vdberg eaba666
Add Dune upload for current week
bram-vdberg 5fd1266
Update prefect deployment
bram-vdberg ff21a1f
Bug fixes
bram-vdberg abcdd82
Update prefect deploy
bram-vdberg 0fac301
Update prefect local config
bram-vdberg 8714923
Bug fixes
bram-vdberg fafe1a9
Bug fixes
bram-vdberg 24a84f8
Apply PR feedback
bram-vdberg db15729
Fix typo
bram-vdberg e17a897
Pylint fixes
bram-vdberg 1df688d
Fix pylinting
bram-vdberg 6b989bd
Fix pylinting
bram-vdberg 9624eb9
Fix pylinting
bram-vdberg 617c006
Pylint fixes
bram-vdberg 9c154ab
Reformat with black
bram-vdberg 8580b91
Mypy fixes
bram-vdberg 83a63a0
Mypy fixes
bram-vdberg 5610b25
Fix linting
bram-vdberg 87cc9d5
Run black
bram-vdberg 8e6203d
Formatting changes
bram-vdberg d532f3f
Mypy fixes
bram-vdberg d7a619e
Linting
bram-vdberg 0fabc19
Linting
bram-vdberg File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Update prefect deployment
- Loading branch information
commit 5fd12666a4e20ecd5e8dd7dd29dc067f3dd70b7e
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -129,3 +129,6 @@ dmypy.json | |
|
||
# Pyre type checker | ||
.pyre/ | ||
|
||
# Vim swap files | ||
**/*.swp | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,14 +1,18 @@ | ||
import os | ||
import re | ||
import sys | ||
import logging | ||
import pandas as pd | ||
from io import StringIO | ||
from dotenv import load_dotenv | ||
from datetime import datetime, timedelta, timezone | ||
from dune_client.client import DuneClient | ||
|
||
""" | ||
deployments_path = os.path.abspath("/deployments") | ||
if deployments_path not in sys.path: | ||
sys.path.insert(0, deployments_path) | ||
""" | ||
|
||
from typing import Any | ||
from prefect import flow, task, get_run_logger | ||
|
@@ -19,6 +23,8 @@ | |
from src.fetch.orderbook import OrderbookFetcher | ||
from src.models.order_rewards_schema import OrderRewards | ||
|
||
load_dotenv() | ||
|
||
def get_last_monday_midnight_utc(): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would have expected to see data uploaded aligned with accounting periods, so Tuesday 00:00:00. In principle, that should not have an effect on the aggregated table. Uploading new values for an accounting week before payments should be easier with alignment. |
||
now = datetime.now(timezone.utc) | ||
current_weekday = now.weekday() | ||
|
@@ -85,7 +91,34 @@ def upload_data_to_dune(data: str, block_start: int, block_end: int): | |
|
||
@task | ||
def update_aggregate_query(table_name: str): | ||
pass | ||
""" | ||
Query example: | ||
WITH aggregate AS ( | ||
SELECT * FROM dune.cowswapbram.dataset_order_rewards_20921069_20921169 | ||
bram-vdberg marked this conversation as resolved.
Show resolved
Hide resolved
|
||
UNION ALL | ||
SELECT * FROM dune.cowswapbram.dataset_testtable | ||
) | ||
|
||
SELECT DISTINCT * FROM aggregate; | ||
""" | ||
|
||
logger = get_run_logger() | ||
dune = DuneClient.from_env() | ||
query_id = os.environ['AGGREGATE_QUERY_ID'] | ||
query = dune.get_query(query_id) | ||
sql_query = query.sql | ||
|
||
if table_name not in sql_query: | ||
logger.info(f"Table name not found, updating table with {table_name}") | ||
insertion_point = insertion_point = sql_query.rfind(")") | ||
updated_sql_query = ( | ||
sql_query[:insertion_point].strip() + | ||
f"\n UNION ALL\n SELECT * FROM {table_name}\n" + | ||
sql_query[insertion_point:] | ||
) | ||
dune.update_query(query_sql=updated_sql_query) | ||
else: | ||
logger.info(f"Table already in query, not updating query") | ||
|
||
|
||
@flow(retries=3, retry_delay_seconds=60, log_prints=True) | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is fine, I guess, but my personal preference would be to have a small
.gitignore
with project specific content. For local setup, I use.git/info/exclude
.Is this approach with a huge .gitignore better for, say, using dev containers?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or a global .gitignore
In
~/.gitconfig
:which contains things like:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not that I know of, I didn't know you could add it to .git/info/exclude. Thanks!