HOTFIX: Convert encoded HTTP connection variables to http type not https #3624
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
With the merge of #3591 and #3572, we started to encounter errors with the data refresh's HTTP steps. When run locally, the
HttpOperator
steps would raise an exception about how they were unable to find a provider for the connection typehttps
. After looking through the Airflow source code for deserializing providers, we realized that the HTTP Operator is expecting anhttp
prefix (whereas previouslyhttps
was an allowed prefix when performing the lookup for which connection type to use).We have code in our entrypoint which transforms connection URIs into URL-encoded
https
connection types (see: WordPress/openverse-catalog#480), and this was ultimately where we identified a fix. Instead of prefixing those values withhttps
, we prefix them withhttp
, and Airflow behaves as expected with the connections. Since the Airflow documentation for the hook describes that you can usehttp
andhttps
prefixes, we'll still file an issue upstream even though we have this addressed locally.Testing Instructions
j build
j api/init && j c
main
it did not.Checklist
Update index.md
).main
) or a parent feature branch.Developer Certificate of Origin
Developer Certificate of Origin