Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support the use case where the user wants to include asserts in the slice. #449

Open
marov opened this issue Dec 9, 2021 · 5 comments
Open
Labels
enhancement New feature or request large takes longer than 1HD question Further information is requested

Comments

@marov
Copy link
Contributor

marov commented Dec 9, 2021

What:
We need to include support assert statements in slices. This is a "cheap" way to add sanity checks to the "productionalized" pipelines

Why:
Just seen this cell in one of my PR reviews:

# check whether all last trading days is in date column of equity dataframe
set(df_monthly_last_trading_day["trading_day"]).issubset(set(df_equity["date"]))

The output is "True" on the test sample, but this does not help, as a) it will be omitted in the extracted DAG and b) without assert it does not stop execution or raise alerts on the upcoming data ...

Possible Approaches:
Ideal scenario: Linea catches cells or print statements that produce boolean output and converts them to try/catch blocks (better code practice than asserts)
Middle scenario: Linea adds "assert" in such cases
MVP: Users write assert themself, but linea includes them in output

@yifanwu
Copy link
Contributor

yifanwu commented Dec 9, 2021

Hi @marov, just to be sure, you want a way to include all the assert ... statements in the slice right?

Thanks!

@marov
Copy link
Contributor Author

marov commented Dec 10, 2021

The ones connected to the artifact we slice

@yifanwu yifanwu added enhancement New feature or request large takes longer than 1HD question Further information is requested labels Dec 10, 2021
@yifanwu
Copy link
Contributor

yifanwu commented Dec 10, 2021

Thanks Mike for the input! For next steps, we should consider putting this on the roadmap for Jan. Needs to solicit feedback from a few other DEs to help with prioritization.

@yifanwu
Copy link
Contributor

yifanwu commented Dec 10, 2021

The ones connected to the artifact we slice

What if there is an assertion on an upstream dependency? E.g.,

import lineapy
a = foo()
assert a > 10
b = bar(a)
lineapy.save(b, 'b val')

I assume you want assert a > 10 to also be included right?

@marov
Copy link
Contributor Author

marov commented Dec 11, 2021

Yes if it fails - we shouldn't continue.

@yifanwu yifanwu changed the title Support assert Support the use case where the user wants to include asserts in the slice. Jan 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request large takes longer than 1HD question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants