indexing - static initialisation and waste of ressources

[henning-gerhardt]

preface:
The following issue may only occur if you indexing a few ten thousand to million entries and / or adjust the elasticsearch configuration entries in kitodo_config.properties not correct or not fitting to your amount of data.

description:
One first initialisation of class IndexingService a static list of index worker and a dynamic list of counted database object per indexing type is created. This is happend even there is no indexing (full or partial / indexing not indexed) is required. This happening inside the class methods prepareIndexWorker() and countDatabaseObjects(). The amount of created worker per type is depending on the configuration value of elasticsearch.indexLimit. If you have a lot of entries inside the database a big collection list is created with inside waiting threads to be executed. As this worker list is only created on application start up there will no updates through new or deleted objects which is a source for errors which will happen if the application is running a long time and an indexing is started.

goal:
It would be better to create the worker instances on index all or indexing not indexed request.

[thomaslow]

This was mostly fixed in #5367. The number of threads no longer depends on the elasticsearch.indexLimit parameter. Instead, the number of threads can be configured via a separate parameter. And threads are only initialized when the indexing is triggered by the user.

As this worker list is only created on application start up there will no updates through new or deleted objects which is a source for errors which will happen if the application is running a long time and an indexing is started.

If any objects (processes, tasks, etc.) are added or edited while the indexing is still ongoing, they should be added or updated through the regular save-logic of the respective object. However, I never checked this scenario specifically.

[andre-hohmann]

@thomaslow : Thanks a lot for your hint. That helps us to clean up the issues.

@henning-gerhardt : Can this issue be closed?

[henning-gerhardt]

No, as the main issue still exist: lists are initialised while first access of the mentioned class and not modified in any way on runtime.