Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pass Kwargs to read_csv, read_parquet etc #1519

Closed
utkarsharma2 opened this issue Jan 4, 2023 · 1 comment · Fixed by #1466
Closed

Pass Kwargs to read_csv, read_parquet etc #1519

utkarsharma2 opened this issue Jan 4, 2023 · 1 comment · Fixed by #1466
Assignees
@pankajastro
Labels
feature New feature or request product/python-sdk Label describing products
Milestone
1.4.0

Comments

@utkarsharma2
Copy link
Collaborator

Please describe the feature you'd like to see
Currently, there is no way for the user to pass all the params listed in the list below to Load_file() operator.
https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html
Which can be useful for a variety of reasons. Example the inferred schema is not correct we can specify our own dtype to it.

Describe the solution you'd like
We can pass kwargs/pandas_option to those methods.

ref: https://astronomer.slack.com/archives/C02B8SPT93K/p1672826587741709
@utkarsharma2 utkarsharma2 added the feature New feature or request label Jan 4, 2023
@utkarsharma2 utkarsharma2 added this to the 1.4.0 milestone Jan 4, 2023
@pankajastro pankajastro added the product/python-sdk Label describing products label Jan 5, 2023
@pankajastro
Copy link
Contributor

pankajastro commented Jan 5, 2023 •
edited
Loading

PR #1466 should take of this and some more pandas-related kwargs

@pankajastro pankajastro mentioned this issue Jan 6, 2023
Merged
2 tasks
@pankajastro pankajastro self-assigned this Jan 6, 2023
pankajastro added a commit that referenced this issue Jan 9, 2023
@pankajastro @pre-commit-ci
Add add pandas_options in load_file (#1466)
38809fa 
# Description
closes: #1519

## What is the current behavior?
currently, load_file do not have an option to pass the pandas-related
param while reading file

## What is the new behavior?
use `load_options` and pass the given values while reading files using
the pandas path

## Does this introduce a breaking change?
No

### Checklist
- [ ] Created tests which fail without the change (if possible)
- [ ] Extended the README / documentation, if necessary

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
utkarsharma2 pushed a commit that referenced this issue Jan 17, 2023
@pankajastro @pre-commit-ci @utkarsharma2
Add add pandas_options in load_file (#1466)
6850b44 
# Description
closes: #1519

## What is the current behavior?
currently, load_file do not have an option to pass the pandas-related
param while reading file

## What is the new behavior?
use `load_options` and pass the given values while reading files using
the pandas path

## Does this introduce a breaking change?
No

### Checklist
- [ ] Created tests which fail without the change (if possible)
- [ ] Extended the README / documentation, if necessary

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pankajastro pankajastro

Labels
feature New feature or request product/python-sdk Label describing products
Projects
None yet
Milestone
1.4.0
Development

Successfully merging a pull request may close this issue.

Use load_options param to pass pandasOptions in load_file astronomer/astro-sdk
2 participants
@utkarsharma2 @pankajastro