Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Path argument for local files does not work on Windows Machines #99

Open
Zeulni opened this issue Nov 27, 2024 · 2 comments
Open

Path argument for local files does not work on Windows Machines #99

Zeulni opened this issue Nov 27, 2024 · 2 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@Zeulni
Copy link

Zeulni commented Nov 27, 2024

Describe the bug
We are using dbt loom in our project on both MacOS and Windows computers. For the MacOS computers, everything works perfectly. However, for the Windows machines, we couldn't perform "dbt deps" or "dbt parse", as we always encountered a runtime error. After debugging for a while, we found the error.
Our dbt loom config looks the following:

manifests:
  - name: dbt_project_x
    type: file
    config:
      path: ../../dbt_project_x/target/manifest.json
  1. In the config, we provide a relative local path to another dbt package we want to import
  2. Then, by DBT Loom, it gets converted to an absolute path via the Pathlib
  3. This Path again is converted to a string
  4. The string is split into the individual parts by "urlsplit" -> here, normally path should end up in the path attribute in. However, for windows machines it ends up in the the netloc attribute. The reason for that is that on Windows, when converting the absolute path to a string, the path starts with two 2 forward slashes, which lets urlsplit parse it into the netloc attribute.

To Reproduce
Testing a dbt package import with a dbt loom config like above and on Windows 11.

Expected behavior
For now, we fixed it by patching the function "load_from_local_filesystem" in manifests.py the following way (if the path attribute is empty, it should look for it in the netloc attribute):

    @staticmethod
    def load_from_local_filesystem(config: FileReferenceConfig) -> Dict:
        """Load a manifest dictionary from a local file"""

        path = config.path.path
        if not path:
            path = config.path.netloc

        if not path:
            raise InvalidManifestPath()

        file_path = Path(path)

        if not file_path.exists():
            raise LoomConfigurationError(f"The path `{file_path}` does not exist.")

        if file_path.suffix == ".gz":
            with gzip.open(file_path, "rt") as file:
                return json.load(file)

        return json.load(open(file_path))

This was just a hotfix for us. Ideally, there is a better solution for that.

Screenshots
See code above.

  • OS: Windows 11
  • dbt-loom Version: 0.6.0
  • dbt-core Version: 1.8.7

Let me know if you have any further questions, happy to help/support! :)

@Zeulni Zeulni added bug Something isn't working triage This issue is being investigated labels Nov 27, 2024
@nicholasyager
Copy link
Owner

Thanks for the detailed bug report @Zeulni!

Alas, Windows is going to be a bit of a gap for the moment -- I don't have any Windows machines to test with! 😅 I'll see if I can get a VM going or something equivalent, so I can test out the hotfix and devise a more robust solution.

@nicholasyager nicholasyager removed the triage This issue is being investigated label Nov 27, 2024
@nicholasyager nicholasyager added this to the 0.7.1 milestone Nov 27, 2024
@Zeulni
Copy link
Author

Zeulni commented Nov 28, 2024

Thank you very much, highly appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants