Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create ResponseUrlProvider #87

Merged
merged 4 commits into from
Sep 12, 2022
Merged

create ResponseUrlProvider #87

merged 4 commits into from
Sep 12, 2022

Conversation

BurnzZ
Copy link
Contributor

@BurnzZ BurnzZ commented Sep 6, 2022

Partially resolves #83.

The only thing missing here is ensuring that the providers are powerful enough such that having a dependency of HttpResponse and BrowserHtml would not trigger downloading the page twice (using the ResponseUrl from the BrowserHtml is preferred if available). However, web-poet needs to be updated to add a url component to BrowserHtml.

Checklist:

  • Basic implementation
  • Changelog

@codecov
Copy link

codecov bot commented Sep 6, 2022

Codecov Report

Merging #87 (4055381) into master (7ac5fbe) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #87      +/-   ##
==========================================
+ Coverage   97.86%   97.88%   +0.02%     
==========================================
  Files          12       12              
  Lines         468      473       +5     
==========================================
+ Hits          458      463       +5     
  Misses         10       10              
Impacted Files Coverage Δ
scrapy_poet/downloadermiddlewares.py 100.00% <ø> (ø)
scrapy_poet/page_input_providers.py 95.65% <100.00%> (+0.33%) ⬆️

@BurnzZ BurnzZ marked this pull request as ready for review September 7, 2022 07:17
@BurnzZ BurnzZ requested review from Gallaecio and kmike September 7, 2022 07:17
@kmike kmike merged commit 83a3cd8 into master Sep 12, 2022
@kmike
Copy link
Member

kmike commented Sep 12, 2022

Thanks @BurnzZ!

@BurnzZ BurnzZ deleted the response-url-provider branch September 13, 2022 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add ResponseUrl provider
3 participants