Fix composites that use database methods not working in the content s…

…erver. Also fix template searches using the database API not working in the content server. Update the manual.

manual/gui.rst

@@ -732,28 +732,31 @@ To choose icons for values in categories, right-click on a value then choose `Ma
 
 * `Choose an icon for this value but not its children`. A dialog will open where you choose an icon for the value. Children of that value will not inherit that icon.
 * `Choose an icon for this value and its children`. A dialog will open where you choose an icon for the value. Any children that don't have their own specified icon will inherit this icon.
-* `Choose an existing icon for this value but not its children`. This option is offered if the value already has an icon that is inherited by the value's children. Selecting it will make the icon apply to the value but not its children.
-* `Choose an existing icon for this value and its children`. This option is offered if the value already has an icon that is not inherited by the value's children. Selecting it will make the icon apply to the value and its children.
+* `Use the existing icon for this value but not its children`. This option is offered if the value already has an icon that is inherited by the value's children. Selecting it will make the icon apply to the value but not its children.
+* `Use the existing icon for this value and its children`. This option is offered if the value already has an icon that is not inherited by the value's children. Selecting it will make the icon apply to the value and its children.
 * `Use the default icon for this value`. This option is offered if the item has an icon. It removes the icon from the value and any children inheriting the icon. The default icon is what is specified below.
+* `Reset all value icons to the default icon`. This option removes all item value icons for the category. It does not remove a template if one exists. There is no undo.
 * `Use/edit a template to choose the default value icon`. This option permits you to provide a calibre template that returns the name of an icon file to be used as a default icon. The template can use two variables:
 
   * ``category``: the lookup name of the category, for example ``authors``, ``series``, ``#mycolumn``.
   * ``value``: the value of the item within the category.
   * ``count``: the number of books with this value. If the value is part of a hierarchy then the count includes the children.
   * ``avg_rating``: the average rating for books with this value. If the value is part of a hierarchy then the average includes the children.
 
-  Book metadata such as title is not available. Template database functions such as book_count() and book_values() will work, but the performance might not be acceptable. Python templates have full access to the calibre database API.
+  Book metadata such as title is not available. Template database functions such as book_count() and book_values() will work, but the performance might not be acceptable. The following template functions will work in the GUI but won't work in the content server: ``connected_device_name()``, ``connected_device_uuid()``, ``current_virtual_library_name()``, ``is_marked()``, and ``virtual_libraries()``.
+
+  In the GUI, Python templates have full access to the calibre database. In the content server, Python templates have access to new API (see `API documentation for the database interface <https://manual.calibre-ebook.com/db_api.html>`_) but not the old API (LibraryDatabase).
 
   For example, this template specifies that any value in the clicked-on category beginning with `History` will have an icon named ``flower.png``::
 
     program:
       if substr($value, 0, 7) == 'History' then 'flower.png' fi
 
-  If the template returns the empty string (``''``) then the category icon will be used. If the template
+  If a template returns the empty string (``''``) then the category icon will be used. If the template
   returns a file name that doesn't exist then no icon is displayed.
 
 * `Use the category icon as the default`. This option specifies that the icon used for the category should be used for any value that doesn't otherwise have an icon. Selecting this option removes any template icon specification.
-* `Reset all value icons to the default icon`. This option removes all item value icons for the category. It does not remove a template if one exists. There is no undo.
+
 
 The icon is chosen using the following hierarchy:
 
@@ -762,7 +765,7 @@ The icon is chosen using the following hierarchy:
 #. The icon from a template, if a template exists and it returns a non-empty string.
 #. The default category icon, which always exists.
 
-Icons for item values are stored in the :file:`tb_icons` subfolder in the calibre configuration folder. Icons used by templates are in the :file:`template_icons` subfolder of :file:`tb_icons`.
+Icons are per-user, not per-library, stored in the calibre configuration folder. Icons for item values are stored in the :file:`tb_icons` subfolder. Icons used by templates are in the :file:`template_icons` subfolder of :file:`tb_icons`.
 
 
 .. raw:: html epub

src/calibre/db/cache.py

@@ -152,8 +152,10 @@ def __init__(self, backend, library_database_instance=None):
         self.shutting_down = False
         self.is_doing_rebuild_or_vacuum = False
         self.backend = backend
-        self.library_database_instance = (None if library_database_instance is None else
-                                          weakref.ref(library_database_instance))
+        # We want templates to have access to LibraryDatabase if we have it,
+        # otherwise this instance (Cache)
+        self.database_instance = (weakref.ref(self) if library_database_instance is None else
+                                  weakref.ref(library_database_instance))
         self.event_dispatcher = EventDispatcher()
         self.fields = {}
         self.composites = {}
@@ -433,13 +435,12 @@ def init(self):
 
             for field, table in iteritems(self.backend.tables):
                 self.fields[field] = create_field(field, table, bools_are_tristate,
-                                          self.backend.get_template_functions)
+                                                  self.backend.get_template_functions, self.database_instance)
                 if table.metadata['datatype'] == 'composite':
                     self.composites[field] = self.fields[field]
 
-            self.fields['ondevice'] = create_field('ondevice',
-                    VirtualTable('ondevice'), bools_are_tristate,
-                    self.backend.get_template_functions)
+            self.fields['ondevice'] = create_field('ondevice', VirtualTable('ondevice'), bools_are_tristate,
+                                                   self.backend.get_template_functions, self.database_instance)
 
             for name, field in iteritems(self.fields):
                 if name[0] == '#' and name.endswith('_index'):

src/calibre/db/fields.py

@@ -64,9 +64,10 @@ class Field:
     is_many_many = False
     is_composite = False
 
-    def __init__(self, name, table, bools_are_tristate, get_template_functions):
+    def __init__(self, name, table, bools_are_tristate, get_template_functions, cache_weakref):
         self.name, self.table = name, table
         dt = self.metadata['datatype']
+        self.cache_weakref = cache_weakref
         self.has_text_data = dt in {'text', 'comments', 'series', 'enumeration'}
         self.table_type = self.table.table_type
         self._sort_key = (sort_key if dt in ('text', 'series', 'enumeration') else IDENTITY)
@@ -237,11 +238,12 @@ class CompositeField(OneToOneField):
     is_composite = True
     SIZE_SUFFIX_MAP = {suffix:i for i, suffix in enumerate(('', 'K', 'M', 'G', 'T', 'P', 'E'))}
 
-    def __init__(self, name, table, bools_are_tristate, get_template_functions):
-        OneToOneField.__init__(self, name, table, bools_are_tristate, get_template_functions)
+    def __init__(self, name, table, bools_are_tristate, get_template_functions, cache_weakref):
+        OneToOneField.__init__(self, name, table, bools_are_tristate, get_template_functions, cache_weakref)
 
         self._render_cache = {}
         self._lock = Lock()
+        self.cache_weakref = cache_weakref
         m = self.metadata
         self._composite_name = '#' + m['label']
         try:
@@ -297,11 +299,12 @@ def bool_sort_key(self, val):
 
     def __render_composite(self, book_id, mi, formatter, template_cache):
         ' INTERNAL USE ONLY. DO NOT USE THIS OUTSIDE THIS CLASS! '
+        db = self.cache_weakref()
         ans = formatter.safe_format(
             self.metadata['display']['composite_template'], mi, _('TEMPLATE ERROR'),
             mi, column_name=self._composite_name, template_cache=template_cache,
             template_functions=self.get_template_functions(),
-            global_vars={rendering_composite_name:'1'}).strip()
+            global_vars={rendering_composite_name:'1'}, database=db).strip()
         with self._lock:
             self._render_cache[book_id] = ans
         return ans
@@ -404,7 +407,7 @@ def get_books_for_val(self, value, get_metadata, book_ids):
 
 class OnDeviceField(OneToOneField):
 
-    def __init__(self, name, table, bools_are_tristate, get_template_functions):
+    def __init__(self, name, table, bools_are_tristate, get_template_functions, cache_weakref):
         self.name = name
         self.book_on_device_func = None
         self.is_multiple = False
@@ -799,7 +802,7 @@ def get_news_category(self, tag_class, book_ids=None):
         return ans
 
 
-def create_field(name, table, bools_are_tristate, get_template_functions):
+def create_field(name, table, bools_are_tristate, get_template_functions, cache_weakref):
     cls = {
             ONE_ONE: OneToOneField,
             MANY_ONE: ManyToOneField,
@@ -819,4 +822,4 @@ def create_field(name, table, bools_are_tristate, get_template_functions):
         cls = CompositeField
     elif table.metadata['datatype'] == 'series':
         cls = SeriesField
-    return cls(name, table, bools_are_tristate, get_template_functions)
+    return cls(name, table, bools_are_tristate, get_template_functions, cache_weakref)

src/calibre/db/search.py

@@ -677,7 +677,7 @@ def fi(default_value=None):
                 val = mi.formatter.safe_format(template, {}, error_string, mi,
                                             column_name='search template',
                                             template_cache=template_cache,
-                                            global_vars=global_vars)
+                                            global_vars=global_vars, database=self.dbcache)
                 if val.startswith(error_string):
                     raise ParseException(val[len(error_string):])
                 if sep == 't':

src/calibre/utils/formatter.py

@@ -1759,9 +1759,10 @@ def _eval_python_template(self, template, column_name):
 
     def _run_python_template(self, compiled_template, arguments):
         try:
+            db = get_database(self.book, None)
+            db = db if db is not None else self.database
             self.python_context_object.set_values(
-                         db=(self.database if self.database is not None
-                             else get_database(self.book, get_database(self.book, None))),
+                         db=db,
                          globals=self.global_vars,
                          arguments=arguments,
                          formatter=self,

src/calibre/utils/formatter_functions.py

@@ -211,7 +211,7 @@ def get_database(mi, name):
         if name is not None:
             raise ValueError(_('In function {}: The database has been closed').format(name))
         return None
-    wr = getattr(cache, 'library_database_instance', None)
+    wr = getattr(cache, 'database_instance', None)
     if wr is None:
         if name is not None:
             only_in_gui_error(name)
@@ -249,9 +249,20 @@ def only_in_gui_error(self):
         only_in_gui_error(self.name)
 
     def get_database(self, mi, formatter=None):
-        if (db := getattr(formatter, 'database', None)) is not None:
-            return db
-        return get_database(mi, self.name)
+        # Prefer the db that comes from proxy_metadata because it is probably an
+        # instance of LibraryDatabase where the one in the formatter might be an
+        # instance of Cache
+        formatter_db = getattr(formatter, 'database', None)
+        if formatter_db is None:
+            # The formatter doesn't have a database. Try to get one from
+            # proxy_metadata. This will raise an exception because the name
+            # parameter is not None
+            return get_database(mi, self.name)
+        else:
+            # We have a formatter db. Try to get the db from proxy_metadata but
+            # don't raise an exception if one isn't available.
+            legacy_db = get_database(mi, None)
+            return legacy_db if legacy_db is not None else formatter_db
 
 
 class BuiltinFormatterFunction(FormatterFunction):
@@ -1678,7 +1689,8 @@ class BuiltinAnnotationCount(BuiltinFormatterFunction):
     __doc__ = doc = _(
 r'''
 ``annotation_count()`` -- return the total number of annotations of all types
-attached to the current book.[/] This function works only in the GUI.
+attached to the current book.[/] This function works only in the GUI and the
+content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals):
@@ -2441,7 +2453,7 @@ class BuiltinGetLink(BuiltinFormatterFunction):
 ans
 [/CODE]
 [/LIST]
-This function works only in the GUI.
+This function works only in the GUI and the content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, field_name, field_value):
@@ -2602,6 +2614,8 @@ class BuiltinCheckYesNo(BuiltinFormatterFunction):
 ``#bool`` is either True or undefined (neither True nor False).
 
 More than one of ``is_undefined``, ``is_false``, or ``is_true`` can be set to 1.
+
+This function works only in the GUI and the content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, field, is_undefined, is_false, is_true):
@@ -2862,7 +2876,7 @@ class BuiltinBookCount(BuiltinFormatterFunction):
 eliminates problems caused by the requirement to escape quotes in search
 expressions.
 [/LIST]
-This function can be used only in the GUI.
+This function can be used only in the GUI and the content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, query, use_vl):
@@ -2897,7 +2911,7 @@ class BuiltinBookValues(BuiltinFormatterFunction):
 searches that combine information from many books such as looking for series
 with only one book. It cannot be used in composite columns unless the tweak
 ``allow_template_database_functions_in_composites`` is set to True. This function
-can be used only in the GUI.
+can be used only in the GUI and the content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, column, query, sep, use_vl):
@@ -2936,7 +2950,7 @@ class BuiltinHasExtraFiles(BuiltinFormatterFunction):
 is supplied then the list is filtered to files that match ``pattern`` before the
 files are counted. The pattern match is case insensitive. See also the functions
 :ref:`extra_file_names`, :ref:`extra_file_size` and :ref:`extra_file_modtime`.
-This function can be used only in the GUI.
+This function can be used only in the GUI and the content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, *args):
@@ -2967,7 +2981,8 @@ class BuiltinExtraFileNames(BuiltinFormatterFunction):
 ``pattern``, a regular expression, is supplied then the list is filtered to
 files that match ``pattern``. The pattern match is case insensitive. See also
 the functions :ref:`has_extra_files`, :ref:`extra_file_modtime` and
-:ref:`extra_file_size`. This function can be used only in the GUI.
+:ref:`extra_file_size`. This function can be used only in the GUI and the
+content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, sep, *args):
@@ -2996,7 +3011,8 @@ class BuiltinExtraFileSize(BuiltinFormatterFunction):
 ``extra_file_size(file_name)`` -- returns the size in bytes of the extra file
 ``file_name`` in the book's ``data/`` folder if it exists, otherwise ``-1``.[/] See
 also the functions :ref:`has_extra_files`, :ref:`extra_file_names` and
-:ref:`extra_file_modtime`. This function can be used only in the GUI.
+:ref:`extra_file_modtime`. This function can be used only in the GUI and the
+content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, file_name):
@@ -3025,7 +3041,7 @@ class BuiltinExtraFileModtime(BuiltinFormatterFunction):
 the empty string, returns the modtime as the floating point number of seconds
 since the epoch.  See also the functions :ref:`has_extra_files`,
 :ref:`extra_file_names` and :ref:`extra_file_size`. The epoch is OS dependent.
-This function can be used only in the GUI.
+This function can be used only in the GUI and the content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, file_name, format_string):
@@ -3067,6 +3083,7 @@ class BuiltinGetNote(BuiltinFormatterFunction):
     get_note('authors', 'Isaac Asimov', 1)
 [/CODE]
 [/LIST]
+This function works only in the GUI and the content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, field_name, field_value, plain_text):
@@ -3135,6 +3152,7 @@ class BuiltinHasNote(BuiltinFormatterFunction):
 [CODE]
     list_count(has_note('authors', ''), '&') ==# list_count_field('authors')
 [/CODE]
+This function works only in the GUI and the content server.
 ''')
 
     def evaluate(self, formatter, kwargs, mi, locals, field_name, field_value):