Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to find dependency solution for airflow + dbt #8372

Closed
4 tasks done
yucer opened this issue Aug 28, 2023 · 2 comments
Closed
4 tasks done

Unable to find dependency solution for airflow + dbt #8372

yucer opened this issue Aug 28, 2023 · 2 comments
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged

Comments

@yucer
Copy link

yucer commented Aug 28, 2023

  • Poetry version: 1.6.1
  • Python version: 3.10.12
  • OS version and name: Linux Mint 21.2
  • pyproject.toml: failing pyproject.toml (it is the one resulting from poetry init --no-interaction)
  • I am on the latest stable Poetry version, installed using a recommended method.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have consulted the FAQ and blog for any relevant entries or release notes.
  • No exception occurs when executing a command.

Issue

I was following the initialization steps for this article:

## create project directory
mkdir airflow-dbt
cd airflow-dbt

## use poetry to initialize the project and manage our dependencies
pip3 install poetry
poetry init --no-interaction

## install our dependencies (Create our virtualenv through poetry)
poetry add apache-airflow apache-airflow-providers-postgres dbt-postgres pandas astronomer-cosmos

and poetry failed to find a solution for the dependencies:


Using version ^2.7.0 for apache-airflow
Using version ^5.6.0 for apache-airflow-providers-postgres
Using version ^1.6.1 for dbt-postgres
Using version ^2.0.3 for pandas
Using version ^1.0.5 for astronomer-cosmos

Updating dependencies
Resolving dependencies... (9.7s)

Because no versions of dbt-postgres match >1.6.1,<1.7.0b1 || >1.7.0b1,<2.0.0
 and dbt-postgres (1.7.0b1) depends on dbt-core (1.7.0b1), dbt-postgres (>1.6.1,<2.0.0) requires dbt-core (1.7.0b1).
And because dbt-core (1.7.0b1) depends on dbt-semantic-interfaces (>=0.2.0,<0.3.0), dbt-postgres (>1.6.1,<2.0.0) requires dbt-semantic-interfaces (>=0.2.0,<0.3.0).
And because dbt-postgres (1.6.1) depends on dbt-core (1.6.1)
 and dbt-core (1.6.1) depends on dbt-semantic-interfaces (>=0.2.0,<0.3.0), dbt-postgres (>=1.6.1,<2.0.0) requires dbt-semantic-interfaces (>=0.2.0,<0.3.0).
Because dbt-semantic-interfaces (0.2.0) depends on jsonschema (>=3.0,<4.0)
 and no versions of dbt-semantic-interfaces match >0.2.0,<0.3.0, dbt-semantic-interfaces (>=0.2.0,<0.3.0) requires jsonschema (>=3.0,<4.0).
Thus, dbt-postgres (>=1.6.1,<2.0.0) requires jsonschema (>=3.0,<4.0).
And because apache-airflow (2.7.0) depends on jsonschema (>=4.18.0)
 and no versions of apache-airflow match >2.7.0,<3.0.0, dbt-postgres (>=1.6.1,<2.0.0) is incompatible with apache-airflow (>=2.7.0,<3.0.0).
So, because airflow-dbt2 depends on both apache-airflow (^2.7.0) and dbt-postgres (^1.6.1), version solving failed.

but it works using the restrictions added by the pyproject.toml of the git repository used in the article with:

[tool.poetry.dependencies]
python = "^3.10"
apache-airflow = "^2.6.2"
apache-airflow-providers-postgres = "^5.5.0"
dbt-postgres = "^1.5.1"
pandas = "^2.0.2"
astronomer-cosmos = "^0.7.3"
black = "^23.3.0"

The description of the project says:

Poetry comes with an exhaustive dependency resolver, which will always find a solution if it exists.
And get a detailed explanation if no solution exists.

In this case the solution exist and was not found.

@yucer yucer added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Aug 28, 2023
@dimbleby
Copy link
Contributor

dimbleby commented Aug 28, 2023

poetry add without further constraints always selects the latest version of the dependencies listed. Your solution uses not-the-latest eg of apache-airflow.

Duplicate #707, please close

@yucer yucer closed this as completed Aug 28, 2023
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged
Projects
None yet
Development

No branches or pull requests

2 participants