[MINDEXER-151] Speed up Index update from remote #255
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cstamas
force-pushed
the
MINDEXER-151-poc
branch
from
96ecb86 to
91bf3da
Compare
This change (ingesting GZIP files raw records into Lucene index on multiple threads) on my PC halves the execution time: while BasicUsageExample on master takes over 15 minutes to finish (when doing full update), this PR makes it under 7 minutes.
cstamas
force-pushed
the
MINDEXER-151-poc
branch
from
91bf3da to
4176f80
Compare
cstamas
changed the title
mbien
reviewed
Nov 22, 2022
Comment on lines
+254
to
255
| LOGGER.debug( "Merged silos..." ); | ||
| w.commit(); |
There was a problem hiding this comment.
I have been experimenting with this in the NetBeans maven support modules and MT extraction (#4999) and it works great!
Could maven-indexer cleanup the temp silo folders after the merge? It is currently leaving almost 6 GB behind after a full update.
There was a problem hiding this comment.
Yes, good idea, I forgot about it, sorry. Created https://issues.apache.org/jira/browse/MINDEXER-176
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce MT index update, allow users to specify the count of threads/silos to use while importing index. Without code change threads = 1 and there is no change in codepath (everything works as before).
This change (ingesting GZIP files raw records into Lucene index on multiple threads) on my PC halves the execution time: while BasicUsageExample on master takes over 15 minutes to finish (when doing full update), this PR makes it under 7 minutes when using 4 threads.
https://issues.apache.org/jira/browse/MINDEXER-151