Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updating core.py to address dask-dataframe types #56

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

demousersrccode
Copy link

Updated the core.py script to accept data-frames created using the dask library as opposed pandas; update can be found on the d3_to_s3 function

Dask data frames are an alternative to pandas as it allows for parallel processing of data

@agawronski
Copy link
Owner

@BrightEmah123 I wonder if it makes more sense for you to just fork the whole project, and make it only for dask, literally just make dask-redshift.

Thinking about it more, most people looking for this package are using pandas, esp given that it's in the name, so forcing them to have dask installed to use it seems a bit odd. what do you think @yaojiach ?

@yaojiach
Copy link
Collaborator

yaojiach commented Jul 24, 2021

I agree. Or put dask into extras_require as optional dependency (doc).

@demousersrccode
Copy link
Author

oh ok, I agree with you on this; I think it makes more sense doing that considering the main purpose of this project was to cater for just Pandas Dataframe Types. I'll do what you requested - fork it and rename it as that. Then I can leave it for you to review.
Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants