UI Revamp: restore search functionality

[benoit74]

In every ZIM, we need to have proper information in the two search indexes which are inside the ZIM so that suggestion search and full-text search are working properly in readers supporting these indexes.

Due to the move to Vue.JS in the UI Revamp, we do not have anymore one HTML file per video ; search indexes are hence not updated anymore, and search is just not working anymore.

We need to restore the search functionality by adding proper information in the two search indexes.

Nota: in former UI, suggestions and full-text search were only returning videos, not playlists, let's start with this for now.

Adding this search functionality in a Vue.JS-based scraper has been already done in freecodecamp: https://github.com/openzim/freecodecamp/blob/a1221073049895609d7712a8834474311535f0c1/scraper/src/fcc2zim/build.py#L68-L88

The idea is to create "fake" HTML pages which use the <meta http-equiv="refresh" content="0;URL='xxxx'" /> functionality to immediately redirect the user to the proper page. This allows to set proper title in the search index for suggestions. And this could be used to trick the search index by storing whatever we want in full-text search in the HTML body.

This is however clearly a bit tricky (create a fake HTML entry where we do not want to generate HTML anymore ... well done ...) + a waste of space (we store for instance the video description twice, once in the fake HTML and once in the JSON used by Vue.JS UI)

@rgaudin as far as I remember, we spoke about this same need for Kolibri, and as far as I remembered you mentioned a solution to have this full-text search working as well without having to store the content in a real HTML file but just by updating the search indexes appropriately; of course I do not find the notes I've made at that time, probably because I forgot to write them down in a Github issue... sorry about that ; would you mind to point us in the right direction again. I remember about using a ZIM redirect entry (with add_redirect), but I don't see now how it could work so I might be totally mistaken ^^ Or maybe these were more ideas for an issue to open at libzim level so that we can specify title and content, and redirect to a path including a fragment?

[rgaudin]

• ZIM redirects are from an entry to another entry. While libkiwix-server generates 302 responses, this is not standardized (again with the impl doc 🤓) and readers are free to (and apple does AFAIR) simply return the redirect-target's content
• The data for the search index associated with an entry is called IndexData in libzim
• It is possible to set IndexData manually in libzim, and in pylibzim (look at the tests for an example)
• By default, if there is no IndexData and the content-type is HTML, then libzim creates one (by parsing HTML and extracting text)
• It is this possible to have a stub HTML entry that's just an HTML redirect, without any content but a real-content indexdata associated and thus be returned in search.
• Suggestions are based on Title so the ZIM entry for the stub needs to be properly set as well and those need to be marked FRONT_ARTICLE.
• With the content in JSON files, the auto-index will not be triggered there so there won't be duplication

@benoit74 does that answers your question?

[benoit74]

@benoit74 does that answers your question?

Perfectly, plan is now crystal clear for me. Sorry again to made you repeat this a second time, my bad.

[kelson42]

ZIM redirects are from an entry to another entry. While libkiwix-server generates 302 responses, this is not standardized (again with the impl doc 🤓) and readers are free to (and apple does AFAIR) simply return the redirect-target's content

@rgaudin I confirm after discussion with dev. I would like to fix this now but miss a reprocase to open a quality issue. Do you have one?

[rgaudin]

I missed the discussion. How do you want to fix? 302 everywhere or 302 nowhere?