Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Format Type Options (formatTypeOptions) for Snowflake #1513

Closed
9 tasks
sunank200 opened this issue Jan 3, 2023 · 0 comments · Fixed by #1516
Closed
9 tasks

Add Format Type Options (formatTypeOptions) for Snowflake #1513

sunank200 opened this issue Jan 3, 2023 · 0 comments · Fixed by #1516
Labels
feature New feature or request product/python-sdk Label describing products
Milestone

Comments

@sunank200
Copy link
Contributor

Please describe the feature you'd like to see
Currently while creating stage, the Loadfile implementation for snowflake only takes integration storage details from native_support_kwargs . This is how COPY INTO command isbeing used: https://github.com/astronomer/astro-sdk/blob/main/python-sdk/src/astro/databases/snowflake.py#L608

It should support all format options for all file types.

Describe the solution you'd like

  • Add format option supports for all filetypes for Snowflake integration

User should be able to run something like the following using load file

copy into flight_data
  from 's3://jf-ml-data/flight_data/On_Time_Reporting_Carrier_On_Time_Performance_1987_present_2022_3.csv' credentials=(aws_key_id='***' aws_secret_key='***')
  file_format = (  
    type = 'CSV'
    field_delimiter = ','
    skip_header = 1
    FIELD_OPTIONALLY_ENCLOSED_BY = '"'
    NULL_IF = ('NULL','null','')
  );

Slack thread: https://astronomer.slack.com/archives/C02B8SPT93K/p1672756338230149

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)
@sunank200 sunank200 added the feature New feature or request label Jan 3, 2023
@dimberman dimberman linked a pull request Jan 3, 2023 that will close this issue
2 tasks
@phanikumv phanikumv added this to the 1.6.0 milestone Jan 4, 2023
@pankajastro pankajastro added the product/python-sdk Label describing products label Jan 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request product/python-sdk Label describing products
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants