Add fulltext search

[tobiasdiez]

Index all files linked in some entry and provide full text search of their content.

There have been already two attempts to implement this: #2264 #2838

[AEgit]

Just as a reminder: If someone tackles this (definitely interesting) suggestion, please also consider the performance impact on large databases.

[AEgit]

Actually, the more I think about, the more it is clear to me, that this will have a massive performance impact and therefore priority when implementing this feature (if it is implemented) should always be performance testing on large databases.

At the moment I have a database containing >20,000 items. The Windows 10 search takes several days (!) to index the database from scratch. I know that because in rare cases (usually because of some Windows update) the search index got corrupted and I had to rebuilt it. Once you have the search index, adding new items usually is not a big issue - the index updates itself. But the first time you do it has a big performance impact. Unfortunately, the search index on Windows 10 at the moment is just single-threaded, but that single thread uses 100% of one CPU core over multiple days when starting the indexing from scratch.

It might actually be better to rely on a separate search index tool (e.g., Recoll: https://www.lesbonscomptes.com/recoll/) and import that index instead of integrating such a performance-intensive feature in JabRef.

In any case, this feature probably should not be active as a default setting.

[ilippert]

Hmm, how about a "modular" solution that employs indexing solutions that are already available for an OS. for instance, several Linux distributions come with tracker - and could draw on that data.

so, jabref could proceed by checking whether a known indexing tool is available

• if yes, use it
• if no, propose another separate search index

[AEgit]

As far as I am ware of, this feature has not been added to JabRef at the moment:
JabRef 5.2--2020-12-09--d1fb9e2
Windows 10 10.0 amd64
Java 14.0.2

[ilippert]

Another consideration for performance: in case JabRef can detect whether it is in active use (e.g. some form of interaction within the last few seconds), then indexing could be stopped, so that indexing only runs when JabRef is not currently in use.

[koppor]

@AEgit We think, that is already fixed in our development version and consequently the change will be included in the next release.

We would like to ask you to use a development build from https://builds.jabref.org/main and report back if it works for you.

[AEgit]

JabRef 5.4--2021-12-10--eff8073
Windows 10 10.0 amd64
Java 16.0.2
JavaFX 17.0.1+1

As far as I can tell, the fulltext search works in the current dev. I have not tested the performance impact properly, since I only use it on a very reduced version of my actual work database. As long as #4237 is not implemented, I cannot switch from JabRef 3.8.2 to Jabref 5 - only when that is implemented, I will be able to test the performance impact of this change.

[Accacio]

Does this fulltext search work with text files linked to an entry? I use text files for my comments, and I cannot search the terms in the files.

[ThiloteE]

I just tried it with a .txt file on

JabRef 5.6--2022-01-30--8ca6b7f
Windows 10 10.0 amd64
Java 16.0.2
JavaFX 17.0.2-ea+3

and entering something into the JabRef search bar yielded zero results, so to answer your question, it does not seem to work with text files linked to an entry.

[Accacio]

I just tried it with a .txt file on

JabRef 5.6--2022-01-30--8ca6b7f Windows 10 10.0 amd64 Java 16.0.2 JavaFX 17.0.2-ea+3

and entering something into the JabRef search bar yielded zero results, so to answer your question, it does not seem to work with text files linked to an entry.

Thank you, maybe this issue should be reopened, @koppor ?

[Siedlerchr]

Please open a new issue, e.g. Support other file types for fulltext search

[Accacio]

Please open a new issue, e.g. Support other file types for fulltext search

Thank you, just did it