Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for GCS #21

Merged
merged 10 commits into from
Nov 11, 2023
Merged

feat: add support for GCS #21

merged 10 commits into from
Nov 11, 2023

Conversation

nawfel-bacha
Copy link
Contributor

@nawfel-bacha nawfel-bacha commented Nov 3, 2023

Added support for loading manifest file from a GCS bucket.

Example of dbt_loom.config.yml:

Screenshot 2023-11-03 at 12 42 20

Example of a dbt run:

Screenshot 2023-11-03 at 12 41 28

Resolves: #20

@nicholasyager nicholasyager self-assigned this Nov 3, 2023
@nicholasyager nicholasyager added the enhancement New feature or request label Nov 3, 2023
Copy link
Owner

@nicholasyager nicholasyager left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the pull request, @nawfel-bacha! Adding GCS support will be a boon for folks using GCP, and I'll be able to use this as a baseline for S3 support.

I left a few comments on this PR around types and terminology. As always, this is intended to spark discussion more than anything else. Please let me know if the requested changes make sense and align with your use case.

Lastly, note that this repo is configured with pre-commit to enforce format, linting, and type checks. Please verify that your development environment is leveraging these pre-commit checks, so our changes are consistent.

README.md Outdated Show resolved Hide resolved
pyproject.toml Show resolved Hide resolved
dbt_loom/__init__.py Outdated Show resolved Hide resolved
dbt_loom/__init__.py Outdated Show resolved Hide resolved
@nawfel-bacha
Copy link
Contributor Author

@nicholasyager Thanks for the feedback, I made the necessary changes, feel free to check it out!

identifier=node.get("relation_name").split(".")[-1].replace('"', ""),
identifier=node.get("relation_name").split(".")[-1].replace('"', "").replace('`',""),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed a bug where the identifier would have double backticks.

Screenshot 2023-11-08 at 10 08 49

Copy link
Owner

@nicholasyager nicholasyager left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nawfel-bacha This is very nearly there! I left a couple comments around documentation. Once those are resolved, we'll be good to merge this in and cut a new release!

dbt_loom/__init__.py Show resolved Hide resolved
dbt_loom/__init__.py Show resolved Hide resolved
README.md Show resolved Hide resolved
@nicholasyager nicholasyager added this to the v0.3.0 milestone Nov 11, 2023
Copy link
Owner

@nicholasyager nicholasyager left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nawfel-bacha Thanks for the changes! This PR is good to go 🚀

@nicholasyager nicholasyager merged commit 3c394b4 into nicholasyager:main Nov 11, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support GCS bucket as a manifest source
2 participants