Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Blank/empty HTML comments result in loss of information in that element #45

Closed
dgtlmoon opened this issue Jul 18, 2022 · 9 comments · Fixed by #46
Closed

Blank/empty HTML comments result in loss of information in that element #45

dgtlmoon opened this issue Jul 18, 2022 · 9 comments · Fixed by #46

Comments

@dgtlmoon
Copy link

version 2.2.0

from inscriptis import get_text

x = get_text('<html><body><span class="price-detailed__unit-price"><span>$<!-- -->90<!-- -->.<!-- -->74</span></span></body></html>')
print (x)
# $

x = get_text('<html><body><span class="price-detailed__unit-price"><span>$90.74</span></span></body></html>')
print (x)
# 90.74

I have no idea why this is the case :/ I noticed some sites may use this to stop scrapers
@dgtlmoon
Copy link
Author

For now I'm just regexing this out https://github.com/dgtlmoon/changedetection.io/pull/764/files#diff-97e35d8550009a723110a24b826e4329175e4d159ebe1f2cca6b684fc45197c5R216

@AlbertWeichselbraun
Copy link
Contributor

thank you for reporting this issue.

i did a first analysis and see two ways of addressing this:

  1. filter out comments with regexps (very easy to do for the general case, but tricky if we want to consider more sophisticated settings such as comments in pre-text, nested comments, etc.).
  2. parse comment nodes for text (this seems to be the sustainable solution).

I will add your example to the unittests cases and do some more additional testing before commiting to a solution.

@dgtlmoon
Copy link
Author

@AlbertWeichselbraun anytime :) Thanks for the great project!

Here's an example in the wild https://www.homedepot.com/p/Home-Decorators-Collection-Hand-Scraped-Strand-Woven-Sahara-3-8-in-T-x-5-1-5-in-W-x-36-02-in-L-Engineered-Click-Bamboo-Flooring-HL655H/300011064

@AlbertWeichselbraun
Copy link
Contributor

I have created a fix that draws upon parsing node comments (i.e., should be considerably more robust than the regex) in the branch fix/html-comment-ofuscation
https://github.com/weblyzard/inscriptis/tree/fix/html-comment-ofuscation

may I ask you to check, whether this solves the problem (so far I can confirm that it works with both examples you provided).

Many thanks, Albert

@AlbertWeichselbraun AlbertWeichselbraun mentioned this issue Aug 2, 2022
Merged
@dgtlmoon
Copy link
Author

dgtlmoon commented Feb 2, 2024

@AlbertWeichselbraun apologies, i only just saw that! I cant find that branch

@dgtlmoon
Copy link
Author

dgtlmoon commented Feb 2, 2024

with version 2.4.0.1 there is a whitespace added

>>> from inscriptis import get_text
>>> 
>>> x = get_text('<html><body><span class="price-detailed__unit-price"><span>$<!-- -->90<!-- -->.<!-- -->74</span></span></body></html>')
>>> print(x)
$ 90.74

@dgtlmoon
Copy link
Author

dgtlmoon commented Feb 2, 2024

Maybe i'm missing some setting that enforces a whitespace where there is a comment block?

>>> x = get_text('<html><body><span class="price-detailed__unit-price"><span>$90.<!-- silent -->74</span></span></body></html>')
>>> x
'$90. 74'

@dgtlmoon dgtlmoon mentioned this issue Feb 2, 2024
Closed
@AlbertWeichselbraun
Copy link
Contributor

Using strict indentation handling fixes this (see #83).

@dgtlmoon
Copy link
Author

dgtlmoon commented Feb 2, 2024

@AlbertWeichselbraun thanks so much :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix/html comment ofuscation weblyzard/inscriptis
2 participants
@dgtlmoon @AlbertWeichselbraun