Introduce normalize and normalization schema. #130

mgautierfr · 2023-12-08T11:07:55Z

This is a first PR with "real" work on warc2zim2.

Introduce url normalization (in place of "canonicalize")
Don't store headers
Remove all service worker stuff (sw.js itself, index.html, ...)

This PR should works with website which don't need any content or url rewriting. Meaning:

Local resource only.
All links relative
No js or "simple" js (following the above rules)
No url being reduced (or else, with link pointed already to the reduced url)

benoit74

Mainly one "stupid" question.
I cannot really review the rest, it is too hard for me to follow where we are going. At least I don't see things which looks weird to me.

src/warc2zim/converter.py

benoit74 · 2023-12-08T11:57:07Z

it looks like you did like I mostly always do, you requested the review before checking the CI status ... and it is failing ^^

mgautierfr · 2023-12-08T15:26:19Z

We now use the addAlias method, which need the (really) recent version of libzim (PR openzim/python-libzim#175 not yet merged)

And we need to update https://github.com/openzim/python-scraperlib/ as it is through this project that we get libzim deps.

benoit74 · 2023-12-08T17:29:35Z

As we said last monday, we should probably release all this stuff so that the CI and us can test this correctly.

And again, we should not merge this to main but to a temporary branch.

kelson42 · 2023-12-10T16:33:12Z

@mgautierfr Please create a zimit2 branch to gather all your (post PR) work.

rgaudin

Thank you ; given we're very close to a libzim release, I didn't go through the trouble of building a special pylibzim to run the tests.

Only a few minor comments on my side 👍

src/warc2zim/converter.py

src/warc2zim/url_rewriting.py

mgautierfr · 2023-12-11T14:34:39Z

Two "fixup" commits added in the git history.

rgaudin

All good now 👍

kelson42 · 2023-12-16T17:17:37Z

Can we rely on new’y released pythin-libzim 3.4.0 and merge?

rgaudin · 2023-12-16T17:19:16Z

No it depends on scraperlib

kelson42 · 2023-12-16T17:35:38Z

@rgaudin Then openzim/python-scraperlib#116 is the blocker

rgaudin · 2023-12-16T18:26:55Z

3.2.0 released

kelson42 · 2023-12-16T19:26:58Z

The CI still fails! It's not a problem with latest (python-)libzim. Probably nothing really bad, but we have to fix in priority to move forward.

Jaifroid · 2023-12-17T13:04:01Z

Just to double-check: has it really been decided that Headers should never be stored, given that they provide instructions to the browser about how to deal with Responses (see kiwix/overview#95 (comment))? In current warc2zim-generated archives, every Response is constructed from two lookups: a first-pass lookup to get the Response Header, and a second-pass lookup to get the Response Body. Only when both lookups have been done, does the backend have enough information to send a complete Response (mirroring that which was originally recorded) back to the browser. I fear (I hope the fear is unfounded, but I have some doubts) that if the functionality is removed, we're going to have trouble with replaying highly dynamic sites (as you know, it's pretty fragile as it is).

kelson42 · 2023-12-17T13:44:37Z

Just to double-check: has it really been decided that Headers should never be stored

We will implement proper headers storage and mgmt only if necessary to provide nice replay feature. So far, it seems this is not necessary. But, it might be implemented in the future.

codecov · 2023-12-18T09:28:43Z

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (1556717) 91.85% compared to head (d1a7d63) 86.23%.

❗ Current head d1a7d63 differs from pull request most recent head 0bfddc9. Consider uploading reports for the commit 0bfddc9 to get more accurate results

Files	Patch %	Lines
src/warc2zim/converter.py	81.48%	3 Missing and 2 partials ⚠️
src/warc2zim/url_rewriting.py	90.90%	1 Missing and 1 partial ⚠️
src/warc2zim/items.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##           warc2zim2     #130      +/-   ##
=============================================
- Coverage      91.85%   86.23%   -5.63%     
=============================================
  Files              5        5              
  Lines            442      414      -28     
  Branches          70       65       -5     
=============================================
- Hits             406      357      -49     
- Misses            27       46      +19     
- Partials           9       11       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Properly define how we store entries in zim file. We introduce `normalize` helper function class in place of `canonicalize`. We work on normalization on converter level. So we path the path to the items instead of letting them call `normalize`. Converted `test/data/video-vimeo.warc.gz` to zim was containing : ``` A/404.html A/f.vimeocdn.com/js_opt/modules/utils/vuid.min.js A/f.vimeocdn.com/p/3.45.3/css/player.css A/f.vimeocdn.com/p/3.45.3/js/player.js A/i.vimeocdn.com/player/354746.png?mw=200&mh=200 A/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 A/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 A/index.html A/load.js A/oembed.link/favicon.ico A/oembed.link/https://vimeo.com/347119375 A/player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 A/sw.js A/topFrame.html A/vod-progressive.akamaized.net/exp=1635528595~acl=%2Fvimeo-prod-skyfire-std-us%2F01%2F4423%2F13%2F347119375%2F1398505169.mp4~hmac=27c31f1990aab5e5429f7f7db5b2dcbcf8d2f5c92184d53102da36920d33d53e/vimeo-prod-skyfire-std-us/01/4423/13/347119375/1398505169.mp4 H/f.vimeocdn.com/js_opt/modules/utils/vuid.min.js H/f.vimeocdn.com/p/3.45.3/css/player.css H/f.vimeocdn.com/p/3.45.3/js/player.js H/i.vimeocdn.com/player/354746.png?mw=200&mh=200 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 H/oembed.link/favicon.ico H/oembed.link/https://vimeo.com/347119375 H/player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 H/vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4 H/vimeo.fuzzy.replayweb.page/video/347119375 H/vod-progressive.akamaized.net/exp=1635528595~acl=%2Fvimeo-prod-skyfire-std-us%2F01%2F4423%2F13%2F347119375%2F1398505169.mp4~hmac=27c31f1990aab5e5429f7f7db5b2dcbcf8d2f5c92184d53102da36920d33d53e/vimeo-prod-skyfire-std-us/01/4423/13/347119375/1398505169.mp4 ``` With this change it contains: ``` A/404.html A/index.html A/load.js A/sw.js A/topFrame.html H/f.vimeocdn.com/js_opt/modules/utils/vuid.min.js H/f.vimeocdn.com/p/3.45.3/css/player.css H/f.vimeocdn.com/p/3.45.3/js/player.js H/i.vimeocdn.com/player/354746.png?mw=200&mh=200 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 H/oembed.link/favicon.ico H/oembed.link/https://vimeo.com/347119375 H/player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 H/vod-progressive.akamaized.net/exp=1635528595~acl=%2Fvimeo-prod-skyfire-std-us%2F01%2F4423%2F13%2F347119375%2F1398505169.mp4~hmac=27c31f1990aab5e5429f7f7db5b2dcbcf8d2f5c92184d53102da36920d33d53e/vimeo-prod-skyfire-std-us/01/4423/13/347119375/1398505169.mp4 f.vimeocdn.com/js_opt/modules/utils/vuid.min.js f.vimeocdn.com/p/3.45.3/css/player.css f.vimeocdn.com/p/3.45.3/js/player.js i.vimeocdn.com/player/354746.png?mw=200&mh=200 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 oembed.link/favicon.ico oembed.link/https://vimeo.com/347119375 player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4 vimeo.fuzzy.replayweb.page/video/347119375 vod-progressive.akamaized.net/exp=1635528595~acl=%2Fvimeo-prod-skyfire-std-us%2F01%2F4423%2F13%2F347119375%2F1398505169.mp4~hmac=27c31f1990aab5e5429f7f7db5b2dcbcf8d2f5c92184d53102da36920d33d53e/vimeo-prod-skyfire-std-us/01/4423/13/347119375/1398505169.mp4 ```

Before, we were storing a entry using its full path and potentially create a redirect entry (using reduced path) pointing to the full path entry. Now, path reduction is part of normalization and so we directly store entries using their (potentially) reduced path. Converted `test/data/video-vimeo.warc.gz` goes from : ``` A/404.html A/index.html A/load.js A/sw.js A/topFrame.html H/f.vimeocdn.com/js_opt/modules/utils/vuid.min.js H/f.vimeocdn.com/p/3.45.3/css/player.css H/f.vimeocdn.com/p/3.45.3/js/player.js H/i.vimeocdn.com/player/354746.png?mw=200&mh=200 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 H/oembed.link/favicon.ico H/oembed.link/https://vimeo.com/347119375 H/player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 H/vod-progressive.akamaized.net/exp=1635528595~acl=%2Fvimeo-prod-skyfire-std-us%2F01%2F4423%2F13%2F347119375%2F1398505169.mp4~hmac=27c31f1990aab5e5429f7f7db5b2dcbcf8d2f5c92184d53102da36920d33d53e/vimeo-prod-skyfire-std-us/01/4423/13/347119375/1398505169.mp4 f.vimeocdn.com/js_opt/modules/utils/vuid.min.js f.vimeocdn.com/p/3.45.3/css/player.css f.vimeocdn.com/p/3.45.3/js/player.js i.vimeocdn.com/player/354746.png?mw=200&mh=200 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 oembed.link/favicon.ico oembed.link/https://vimeo.com/347119375 player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4 vimeo.fuzzy.replayweb.page/video/347119375 vod-progressive.akamaized.net/exp=1635528595~acl=%2Fvimeo-prod-skyfire-std-us%2F01%2F4423%2F13%2F347119375%2F1398505169.mp4~hmac=27c31f1990aab5e5429f7f7db5b2dcbcf8d2f5c92184d53102da36920d33d53e/vimeo-prod-skyfire-std-us/01/4423/13/347119375/1398505169.mp4 ``` to : ``` A/404.html A/index.html A/load.js A/sw.js A/topFrame.html H/f.vimeocdn.com/js_opt/modules/utils/vuid.min.js H/f.vimeocdn.com/p/3.45.3/css/player.css H/f.vimeocdn.com/p/3.45.3/js/player.js H/i.vimeocdn.com/player/354746.png?mw=200&mh=200 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 H/oembed.link/favicon.ico H/oembed.link/https://vimeo.com/347119375 H/player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 H/vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4 f.vimeocdn.com/js_opt/modules/utils/vuid.min.js f.vimeocdn.com/p/3.45.3/css/player.css f.vimeocdn.com/p/3.45.3/js/player.js i.vimeocdn.com/player/354746.png?mw=200&mh=200 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 oembed.link/favicon.ico oembed.link/https://vimeo.com/347119375 player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4 vimeo.fuzzy.replayweb.page/video/347119375 ``` Notice that `vod-progressive.akamaized.net` is not present. It is "replaced" by `vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4` which is now a plain entry instead of a redirect to `vod-progressive.akamaized.net[...]`.

We don't use it and we agree to not store them (at least for now). If we need them, we will see how to readd them. Converted `test/data/video-vimeo.warc.gz` goes from : ``` A/404.html A/index.html A/load.js A/sw.js A/topFrame.html H/f.vimeocdn.com/js_opt/modules/utils/vuid.min.js H/f.vimeocdn.com/p/3.45.3/css/player.css H/f.vimeocdn.com/p/3.45.3/js/player.js H/i.vimeocdn.com/player/354746.png?mw=200&mh=200 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 H/i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 H/oembed.link/favicon.ico H/oembed.link/https://vimeo.com/347119375 H/player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 H/vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4 f.vimeocdn.com/js_opt/modules/utils/vuid.min.js f.vimeocdn.com/p/3.45.3/css/player.css f.vimeocdn.com/p/3.45.3/js/player.js i.vimeocdn.com/player/354746.png?mw=200&mh=200 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 oembed.link/favicon.ico oembed.link/https://vimeo.com/347119375 player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4 vimeo.fuzzy.replayweb.page/video/347119375 ``` to: ``` A/404.html A/index.html A/load.js A/sw.js A/topFrame.html f.vimeocdn.com/js_opt/modules/utils/vuid.min.js f.vimeocdn.com/p/3.45.3/css/player.css f.vimeocdn.com/p/3.45.3/js/player.js i.vimeocdn.com/player/354746.png?mw=200&mh=200 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d.jpg?mw=80&q=85 i.vimeocdn.com/video/797382244-0106ae13e902e09d0f02d8f404fa80581f38d1b8b7846b3f8e87ef391ffb8c99-d?mw=1280&mh=720&q=70 oembed.link/favicon.ico oembed.link/https://vimeo.com/347119375 player.vimeo.com/video/347119375?h=1699409fe2&app_id=122963 vimeo-cdn.fuzzy.replayweb.page/01/4423/13/347119375/1398505169.mp4 vimeo.fuzzy.replayweb.page/video/347119375 ```

We don't need `WARCHeadersItem` anymore.

Remove other unnecessary files.

We don't have anything now in `A/` or `H/` subdirs. Remove the left over `A/` in test urls (was working thanks to libzim's compatibility layer)

We don't have files to add (and so, no directory).

Assignement expression[*] is new in python 3.8 And python 3.7 is already end of life. Also add testing on 3.12. [*]https://docs.python.org/3/whatsnew/3.8.html#assignment-expressions

mgautierfr · 2023-12-18T15:22:34Z

Rebased fixup. Ready to merge.

mgautierfr requested review from rgaudin and benoit74 December 8, 2023 11:07

benoit74 requested changes Dec 8, 2023

View reviewed changes

src/warc2zim/converter.py Show resolved Hide resolved

mgautierfr force-pushed the url_normalization branch 4 times, most recently from 366fc13 to 7f1de56 Compare December 8, 2023 15:17

mgautierfr changed the base branch from main to warc2zim2 December 11, 2023 10:20

mgautierfr force-pushed the url_normalization branch from 7f1de56 to cdba15c Compare December 11, 2023 10:22

rgaudin requested changes Dec 11, 2023

View reviewed changes

src/warc2zim/converter.py Outdated Show resolved Hide resolved

src/warc2zim/url_rewriting.py Show resolved Hide resolved

src/warc2zim/url_rewriting.py Outdated Show resolved Hide resolved

src/warc2zim/url_rewriting.py Outdated Show resolved Hide resolved

mgautierfr force-pushed the url_normalization branch from cdba15c to 4654a55 Compare December 11, 2023 14:34

rgaudin self-requested a review December 11, 2023 14:38

rgaudin approved these changes Dec 11, 2023

View reviewed changes

mgautierfr added 4 commits December 18, 2023 16:18

Store revisits as alias instead of WARCHeadersItem.

1cacf88

We don't need `WARCHeadersItem` anymore.

mgautierfr and others added 7 commits December 18, 2023 16:18

Do not add service worker stuff in zim file.

09f55eb

Remove other unnecessary files.

Properly set the main page.

1113c1a

Store static content in a _zim_static/ subdir instead of A/.

7355b80

We don't have anything now in `A/` or `H/` subdirs. Remove the left over `A/` in test urls (was working thanks to libzim's compatibility layer)

Deactivate searching file to add in templates directory.

e86f2b7

We don't have files to add (and so, no directory).

Python 3.7 is not supported.

49c49ec

Assignement expression[*] is new in python 3.8 And python 3.7 is already end of life. Also add testing on 3.12. [*]https://docs.python.org/3/whatsnew/3.8.html#assignment-expressions

Requires zimscraperlib 3.2.0 (with alias feature)

f706533

import future annotations as we support py3.8+

0bfddc9

mgautierfr force-pushed the url_normalization branch from d1a7d63 to 0bfddc9 Compare December 18, 2023 15:18

kelson42 merged commit 3f3eb34 into warc2zim2 Dec 19, 2023
10 checks passed

kelson42 deleted the url_normalization branch December 19, 2023 02:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce normalize and normalization schema. #130

Introduce normalize and normalization schema. #130

mgautierfr commented Dec 8, 2023

benoit74 left a comment •

edited

Loading

benoit74 commented Dec 8, 2023

mgautierfr commented Dec 8, 2023

benoit74 commented Dec 8, 2023

kelson42 commented Dec 10, 2023 •

edited

Loading

rgaudin left a comment

mgautierfr commented Dec 11, 2023

rgaudin left a comment

kelson42 commented Dec 16, 2023

rgaudin commented Dec 16, 2023

kelson42 commented Dec 16, 2023

rgaudin commented Dec 16, 2023

kelson42 commented Dec 16, 2023 •

edited

Loading

Jaifroid commented Dec 17, 2023

kelson42 commented Dec 17, 2023

codecov bot commented Dec 18, 2023 •

edited

Loading

mgautierfr commented Dec 18, 2023

Introduce normalize and normalization schema. #130

Introduce normalize and normalization schema. #130

Conversation

mgautierfr commented Dec 8, 2023

benoit74 left a comment • edited Loading

Choose a reason for hiding this comment

benoit74 commented Dec 8, 2023

mgautierfr commented Dec 8, 2023

benoit74 commented Dec 8, 2023

kelson42 commented Dec 10, 2023 • edited Loading

rgaudin left a comment

Choose a reason for hiding this comment

mgautierfr commented Dec 11, 2023

rgaudin left a comment

Choose a reason for hiding this comment

kelson42 commented Dec 16, 2023

rgaudin commented Dec 16, 2023

kelson42 commented Dec 16, 2023

rgaudin commented Dec 16, 2023

kelson42 commented Dec 16, 2023 • edited Loading

Jaifroid commented Dec 17, 2023

kelson42 commented Dec 17, 2023

codecov bot commented Dec 18, 2023 • edited Loading

Codecov Report

mgautierfr commented Dec 18, 2023

benoit74 left a comment •

edited

Loading

kelson42 commented Dec 10, 2023 •

edited

Loading

kelson42 commented Dec 16, 2023 •

edited

Loading

codecov bot commented Dec 18, 2023 •

edited

Loading