Improvements to value icons:

[cbhaley]

1. Make average rating (avg_rating) and count variables available in templates
2. Pass the database to safe_format so it can be used in non-GUI contexts.

The second was needed to make item value templates able to use the DB api or the database formatter functions.

Note: I added the db as a parameter to metadata.py / fillout_tree, passed in from render_categories that already had the db. It seems that this db is a reference to Cache(), not Legacy(), even though the library broker seems to get the legacy version. I couldn't find where it became .new_api so I used the weak reference in cache to get the legacy version. There might be a better way to do this via the library broker.

[kovidgoyal]

LibraryBroker doesnt use Legacy() only Cache(), see the init_library()
function in library_broker.py

Indeed when running the server from the command line (non-GUI context)
or in the GUI when serving a library that is not currently open in the
GUI, library_database_instance will be none.

Does safe_format actually need the Legacy interface? user formatter
functions are loaded only for the default library IIRC. In a server
context where multiple libraries can be open, they cant really be used,
since they will be loaded only from the default library.

[cbhaley]

The base (built-in) server functions are loaded. Library-specific functions aren't, (or might not be) but that is OK in this context.

Dereferencing the weakref() from Cache() to the legacy database works. To test, I ran a GPM template that checks notes, something that uses the database. I can use this GPM template as an icon selector in the content server:

program:
	if has_note($category, $value) then 'flower.png' fi

The has_note() function uses new_api.

This python template that uses LibraryDatabase works in the content server as well:

python:
def evaluate(book, context):
	if context.db.all_ids():
		return 'flower.png'
	return ''

The all_ids() function exists only in legacy (LibraryDatabase) so the weakref deference must be working..

This tells me that the db being given to metadata.py / render_categories() was opened as a LibraryDatabase(). Chasing it backwards, it comes through metadata.py / categories_as_json() from srv.code.py / tag_browser(), which gets the database from srv.utils / get_library_data(), which gets it from ctx.library_info(). This is where I gave up tracing. What I do know is that ctx.library_info() gives back a Cache() object created within a LibraryDatabase() object.

Bottom line: it works. Having the db available in icon value templates dramatically increases the template's power.

[cbhaley]

Stored templates work as well. To test I:

1. Entered an icon template that uses a stored template.
2. Restarted calibre.
3. Opened the content server.
4. Selected a library without the stored template or any icon templates.
5. Changed library to the one with the stored and icon templates.
6. Looked at the icons. The template clearly worked.

[kovidgoyal]

Yes, as I said it will work only in the following scenario.

Running the server from within the GUI.

This is because that library was created with
LibraryDatabase() for use in the GUI, therefore the weakref remains set in the
Cache object from when it was created. Try running

calibre-server

from a command prompt you will see that it wont work.

The GUI uses GUILibraryBroker which overrides init_library to use
LibraryDatabase. The server use LibraryBroker which initialises it using
Cache alone.

[cbhaley]

You are right (of course), when running from the command line the LibraryDatabase object isn't available.

However, the stored templates for a library are available. I tried this by using the GUI to change the default library to something with no stored templates, opened the content server from the command line, and selected a library with templates and icon value rules that use them. They ran. Built-in template functions also ran.

I will change the formatter functions to use new_api, and document that the old api isn't available in icon templates.

However, we have a different problem. Composite columns that use the database don't work in the content server. Reason: the database isn't available to templates run by db.fields.CompositeField because that class doesn't have the database. This is not caused by the changes I'm making so I'm not going to worry about it for the moment. I can fix it if the database were passed to CompositeField()

[cbhaley]

Pushed changes:

1. Don't try to convert the database to a LibraryDatabase instance in metadata.py
2. Change formatter functions to use the new_api where possible. Also fix a bug where the wrong exception was raised attempting to get a database object from the weak reference.

These were tested using the command line content server.

[kovidgoyal]

I dont think we need to worry too much about the composite columns using
db functions. It's an edge case and I am fine saying it is simply not
supported in the content server. That said if you want to make it work,
I wont stop you :)

[cbhaley]

Would you be OK with Cache() passing an instance of itself to fields.create_field() when it creates the field? CompositeField would pass this instance when calling the formatter. This would permit using all formatter database functions that don't require information from view.py such as virtual libraries and marks.

Can I get the current virtual library in srv.metadata.py? Or even better, from Cache()? It seems like the content server does all VL work in the client.

[cbhaley]

Also, I'm now allowed to use embedded assignment (:=)? Your cleanup of formatter_functions does.

[kovidgoyal]

On Tue, Jan 21, 2025 at 05:45:17AM -0800, Charles Haley wrote: Also, I'm now allowed to use embedded assignment (:=)? Your cleanup of formatter_functions does.

Yes, that's python 3.8 which is the oldest version of python calibre supports.

[cbhaley]

Last question: why does this import work? from calibre.gui2 import get_gui
Shouldn't it be from calibre.gui2.ui import get_gui as it is everywhere else?

[kovidgoyal]

On Tue, Jan 21, 2025 at 05:43:38AM -0800, Charles Haley wrote: Would you be OK with Cache() passing an instance of itself to fields.create_field() when it creates the field? CompositeField would pass this instance when calling the formatter. This would permit using all formatter database functions that don't require information from view.py such as virtual libraries and marks.

No, that would be a reference cycle. In any case IIRC CompositeField doesnt create formatters, it just uses the pre-existing one on mi.

Can I get the current virtual library in srv.metadata.py? Or even better, from Cache()? It seems like the content server does all VL work in the client.

There is no current virtual library in the server. The server can serve multiple requests simultaneously. Various server endpoints take a vl parameter in the URL to restrict their results ot a virtual library. In this case the endpoint is tag_browser() in srv/code.py you can get the vl from there and pass it into the srv/metadata.py functions. In fact looking at the code the vl is passed into categories_as_json already.

[cbhaley]

On Tue, Jan 21, 2025 at 05:43:38AM -0800, Charles Haley wrote: Would you be OK with Cache() passing an instance of itself to fields.create_field() when it creates the field? CompositeField would pass this instance when calling the formatter. This would permit using all formatter database functions that don't require information from view.py such as virtual libraries and marks.

No, that would be a reference cycle. In any case IIRC CompositeField doesnt create formatters, it just uses the pre-existing one on mi.

The problem isn't creating a formatter. It is calling safe_format() with a database, which is done in CompositeField. What about passing a function to create_field() to get the Cache instance? Using the function the cache instance would exist only during the evaluation of the composite.

[kovidgoyal]

On Tue, Jan 21, 2025 at 05:52:34AM -0800, Charles Haley wrote: Last question: why does this import work? `from calibre.gui2 import get_gui` Shouldn't it be `from calibre.gui2.ui import get_gui` as it is everywhere else?

It doesnt, was a typo.

[cbhaley]

Alternatively I could pass a weakref to the field, thus avoiding the circular reference. This is better than an accessor function because the function could itself create the circular reference.

[kovidgoyal]

Yes a weakref is fine