Get rid of requests as core dependency

[potiuk]

Get rid of requests as core dependency

This change gets rid of requests as core dependency. We have to
change requests to become an optional dependency because it
(so far) pulls in chardet as dependency and chardet is
LGPL, which is not allowed to be mandatory dependency by
ASF policies.

More info here:

https://issues.apache.org/jira/browse/LEGAL-572

The changes:

• connexion is vendored-in (and requests usage is replaced with httpx)
• Http Provider is turned into optional provider (not preinstalled)
• Few places where requests were used in core and in cloud_sql provider
  which did not cause compatibility problem, it was replaced by httpx.
• new extra added for deprecated experimental API (which is disabled
  by default and optional)
• tests are fixed (using pytest-httpx fixture package)
• The providers: http, airbyte, apache.livy, opsgenie, slack (all depend
  on http) now explicitely depend on requirements.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

A little context here.

1. I used pytest-httpx fixture here which is very convenient way of mocking httpx, however it is pytest-fixture only - which means that we have to mix pytest and unittest type of tests. Howver I think this is justified (and I believe we've already agreed that it might make sense to have both kind of tests at least for a transition period. So this might be a good start.

2. I also might still take a look at slighly less-backwards-compatible change in HttpHook, however I think we will not avoid to also having to upgrade Airbyte, Livy and OpsgenieAlert providers - they all depend on HttpHook and I think we cannot make them backwrds-compatible.

3. If anyone thinks that makes sense I can split the PR, however getting rid of the requests as core dependency is common among all the changes.

4. I will have to add another change - add capability to add version dependencies between providers when they are generated. I planned to do it anyway and it was inevitable, but I want to do it as a separate PR.

[potiuk]

Also. What I tested:

• system test for http works (the example_http DAG succeeds communicates with real world)
• http provider tests (including the system test) work fine with uninstalled requests library
• airflow starts and processes example DAGS with uninstalled requests library. UI works.
• the API works with uninstalled requests library

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[turbaszek]

Hey @potiuk would you mind splitting this change into two commits (code vendor + changes)? It would improve reviewing the required changes 🙏

Plus it may be useful to add note about this in our code style guide - requests are popular library, WDYT?

[potiuk]

Yeah. I am happy to do it.... But there is still a chance we will not have to do it psf/requests#5797 - working on it, so maybe we will be able to simply delete that PR :)

[ad-m]

we will be able to simply delete that PR

I have the impression that httpx is a better library in the long term as it is under active development. If the change is accepted, maybe we want to merge at least some of the changes and set a new long-term goal for the transition to httpx?

[potiuk]

we will be able to simply delete that PR

I have the impression that httpx is a better library in the long term as it is under active development. If the change is accepted, maybe we want to merge at least some of the changes and set a new long-term goal for the transition to httpx?

That might be an option as well. I think a lot depends on requests maintainer's responses now. I think it will be extremely difficult to get rid of requests. This is the 3rd most popular library in PIP and there are 100s if not thousands of packages using it. So we will have it anyway at least due to transitive dependencies.

I think once we start moving more torwards asyncio, httpx will definitely be the way to go.

[potiuk]

@turbaszek @kaxil @ashb - as discussed, I separated out connexion vendor-in + core requests replacement with httpx and made http provider optional.

I will add http change to httpx as separate - draft PR in case it turns out requests will not merge our PR shortly.

[ashb]

Please add a note in updating about the new deprecated_api extra.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.