/api/opds for the OPDS service is very slow

[Putarku]

LRR Version and OS
docker 0.9.0

Bug Details
The loading speed of the link api/opds is very slow. My comic library has 40,000 comics. It takes several minutes for the link to load all the content when I open it.

So can we use a segmented loading mode to load the content of OPDS in batches?

Matching Logs
[2024-05-22 14:49:37.10877] [4830] [trace] [xDVW0yNu_gAN] GET "/api/opds"
[2024-05-22 14:49:37.11126] [4830] [trace] [xDVW0yNu_gAN] Routing to controller "LANraragi::Controller::Api::Other" and action "serve_opds_catalog"
[Search Cache] [debug] Search request: --title-0-0-0
[Search Engine] [debug] No cache available, doing a full DB parse.
[Categories] [debug] No category ID provided.
[Search Core] [debug] Found 40410 results after filtering.

It's hard to tell what the problem is just from the debug logs.

[krakerz]

+1 this
i was hoping lanraragi opds has default function for pagination or configurable to enable pagination, my 17k+ when tried in postman was like this (looks how much line is that) i guess 1 hit for entire database atm

[Difegue]

Yeah the /api/opds catalog will load the entire DB in one gigantic xml currently; Adding pagination to it using the serverside files-per-page setting would be nice.

As a workaround, it's possible to only load the contents of a category instead with the /api/opds?category=(CATEGORY_ID) endpoint.

[Difegue]

I've added barebones support to pagination for OPDS pages, both for categories and the main index.

Please give it a try 🙏

[krakerz]

have those commit pushed into nightly docker tag ?

[Difegue]

I've been having issues with the CI recently, but I just pushed a new image on the nightly tag for linux/386,linux/amd64,linux/arm64.

arm v6/v7 is being spotty rn

[krakerz]

I've been having issues with the CI recently, but I just pushed a new image on the nightly tag for linux/386,linux/amd64,linux/arm64.

arm v6/v7 is being spotty rn

ah RIP, i just noticed, my lanraragi is at raspberry pi, so i can't check it out for now, also thanks for the head ups

[Difegue]

I've reenabled ARMv6/v7 in the Docker pipeline so the latest nightlies should be good to go again. Sorry for the trouble 🙏

[krakerz]

I've reenabled ARMv6/v7 in the Docker pipeline so the latest nightlies should be good to go again. Sorry for the trouble 🙏

thanks, i can confirm the API for opds alone now didnt hit full database, although the param for pagination is kinda weird to me, but it works for now (ignore my params, too much used with other basic crud 🤣 i just used it by habit)

now what we just need is basic search and sort, from title for example, this is an example of kavita opds search

but again, thanks for implement the barebone of pagination, no more 1minute+ /hit API, so yay for now

[Difegue]

Yeah, since the start/next links are what's defined by the OPDS spec, not the query parameters themselves, I just reused the same parameters we already use in the search API.
The amount of items per page/query will depend on the same server-side setting that's currently used on the main page.

Adding search/sort would probably be easy as well all things considered, just needs some more code to tie that part of the spec to the search API. Certainly one for another issue 😤