New Wallabag client

[karthink]

Hi @chenyanming,

Thank you for writing wallabag.el! This post is about a different Emacs package, but I don't think it would exist in its current form without this one.

---

I use Wallabag regularly (outside of Emacs) and tried to use wallabag.el, but discovered that it was unusable for my rather particular situation and was missing features I needed. Specifically:

1. I have been using read-it-later style apps since 2005 and been migrating my data every time I switch apps. As a result, I have ~15,000 articles in my Wallabag DB. This causes multiple problems:

   • I can't initialize the Wallabag DB because it tries to fetch all the entries in one HTTP request, which obviously fails.

   • Wallabag.el loads the entire database into memory twice, the second time as text properties in the wallabag-search buffer. So this consumes 500+ MB of RAM and the Wallabag buffer is unusable. Since Emacs does not like a 500+ MB font-locked buffer, Emacs itself becomes unusable.

2. There is no way to query the database except using a general full text search. This makes it difficult to filter by date, author etc, which I rely on to find things in a listing of 15K.

3. The DB update logic often misses entries or changes to entries if they are added/changed outside of wallabag.el. (This depends on the time-order in which entries are added/changed.) It does a full sweep to track certain changes but this is a bad idea when you have ~15,000 entries. The full sweep also fails because of the HTTP request thing.

4. Poor http request error handling and recovery.

Earlier this year I started writing some patches for wallabag.el to fix these issues, but this required pretty fundamental design changes, and I eventually had to give up on the idea of submitting a PR. I ended up writing a new Wallabag client from scratch.

Things that it has in common with wallabag.el:

• The visual design and UI is inspired by wallabag.el and Elfeed.
• The sqlite db schema is mostly reused from wallabag.el (for compatibility), but I added some new fields I needed.

The rest is all different:

• No shared code with wallabag.el, except for three face definitions.
• No emojis or sidebar menu.
• Incomplete tag support. (I don't use Wallabag tags)
• Fully featured search, including debouncing and some other tweaks to ensure Emacs is never unresponsive:

wallabag-search.mp4

• Minimal footprint: Does not load the db into memory, queries whatever is required instead. The wallabag-search buffer memory size with all 15000 entries showing is now about 25 MB, although with the new search filters there's never more than ~100 entries showing, which is < 800 KB.

• Automatically saves your place in articles so you can resume where you stopped reading.

• Handles updating any number of entries without http errors.

• Different update logic. Using some API endpoints available in newer versions of the Wallabag API, all actions are consistent across clients (no time-ordering dependence), and much less data is transferred compared to wallabag.el.

• Wallabag http requests: Better error handling, automatic oauth token refresh, retries on failure etc.

• Many more quality of life improvements (improved header status/messages, bookmark support, pixel-precision smooth scrolling, imenu support, Embark actions support, better EWW integration etc)

---

I have been using this client for a few months, and am considering making it public now. I hope you don't mind me publishing it -- I see that wallabag.el is GPLv3 licensed, but wanted to check with you in person since my package is clearly inspired by it.

Thanks again for your work on wallabag.el!

[chenyanming]

@karthink Sure! I would love to see the new wallabag client! I did not have much time to add new features for my package as well. By the way, I like your gptel!

[edmundmiller]

I might be late to the party, but it would be cool to add wallabag to https://github.com/fasheng/elfeed-protocol

[nanounanue]

Wow! This is awesome. See this kind of collaboration in real time makes me recover my faith in humanity. Thank you both @karthink and @chenyanming

@karthink my workflow and experience is very similar to you. Even the android client takes forever in my devices the first time (and in some devices it never loads :-( )

@karthink when are you planning to release the new client? It will be available in melpa? Will support integration with vertico, consult, etc?

[karthink]

@karthink Sure! I would love to see the new wallabag client! I did not have much time to add new features for my package as well. By the way, I like your gptel!

Thank you @chenyanming.

I uploaded it to github here.

Even the android client takes forever in my devices the first time (and in some devices it never loads :-( )

@karthink when are you planning to release the new client? It will be available in melpa? Will support integration with vertico, consult, etc?

@nanounanue The new package takes 4-5 minutes to fetch ~15K entries and populate an empty DB in my case.

The package is now live. I don't have any immediate plans to add it to MELPA. It doesn't have any vertico/consult integration, I'm not sure what that would mean.

[karthink]

I might be late to the party, but it would be cool to add wallabag to https://github.com/fasheng/elfeed-protocol

Wallabag provides RSS feeds that you can just subscribe to with Elfeed:

[edmundmiller]

I might be late to the party, but it would be cool to add wallabag to https://github.com/fasheng/elfeed-protocol

Wallabag provides RSS feeds that you can just subscribe to with Elfeed:

Thanks for sharing that! It's more of using the API for a first class integration with wallabag as the back end like wombag, rather than having tags stored in elfeed, it's just using elfeed as a front-end(my simple understanding at-least).

[chenyanming]

Thanks @karthink pointing out issues, haha, I recently developed https://github.com/chenyanming/paw, and integrated it with wallabag.el. I addressed the memory issues with paginations support. Now it is able to handle very large database, since wallabag-search-page-max-rows is the maximum entries it will load. Real time filting (unlike wombag, wallabag.el only use space to seperate keywords for searching multiple columns) is also handled properly. Cheers~