upgrade to maven-indexer 7.0.1 and improve index downloads #4999

mbien · 2022-11-22T05:18:21Z

upgrade to maven-indexer 7.0.1. (NetBeans entered the modern era! :-) )

module requires now jdk 11+
apache lucene 9.5.0 with memory mapped index
uses panama on JDK 19 (PR1, PR2) (JDK 20 PR)
look out for: INFO [org.apache.lucene.store.MemorySegmentIndexInputProvider]: Using MemorySegmentIndexInput with Java 19; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
multi-threaded remote index extraction (at least twice as fast as ST, disabled by default)
removed hack which backported a bugfix
no EOL libs anymore in this module
remote index download permission UI
~~limited concurrent indexing~~ extracted and already integrated in NB 17

remote index (maven central) extraction takes ~6 minutes (post change) on my system (4 cores, hyperthreading off).
local .m2 indexing speed is about the same

benchmark post-PR (old 4 core intel i7 and NVMe, JDK 19):

4 threads:
INFO [org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl]: Indexing of central took 383.24 s.
index folder: 6.4GB

3 threads:
INFO [org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl]: Indexing of central took 453.92 s.
index folder: 6,2 GB

2 threads:
INFO [org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl]: Indexing of central took 557.24 s.
index folder: 5,9 GB

1 thread:
INFO [org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl]: Indexing of central took 916.25 s.
index folder: 5,6 GB

thread count is capped to 4 since it incurs a on-disk file size penalty, furthermore the parallel extraction is always followed by a single threaded merge, which means more threads would have diminishing returns and move the bottle neck to the ST merge phase.

NB 17 on JDK 19 as comparison:

INFO [org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl]: Indexing of central took 1118.49 s.
index folder: 5,8 GB

~~note: tmp files aren't cleaned up properly atm. This might fill up your tmp during testing. (edit: fixed via apache/maven-indexer#273)~~ fixed in 7.0.1

https://issues.apache.org/jira/browse/NETBEANS-1268

UI:
repo index download permission notifications:

Options (permission table + multi threading check box):

java/maven.indexer/manifest.mf

mbien · 2023-02-27T12:24:32Z

bump to maven-indexer 7.0.1

mbien · 2023-02-27T19:15:49Z

thanks to @cstamas for resolving the silo cleanup issue - works great!

mbien · 2023-03-17T03:30:30Z

draft: apache/maven-indexer#302 would give us more control over the temp storage location + give us the option to filter the index

some tests (time filter on y axis, field filter on x):

full: 5.6 -> 2.8 GB
2y: 2.6  -> 1.4 GB
1y: 1.4 -> 0,8 GB

- disable CV tests on JDK 8 - move some more jobs to JDK 11 this configuration would allow us to move new modules like rust/go/hcl to JDK 11 and to upgrade exiting modules, like the maven-indexer apache#4999.

vieiro · 2023-04-30T18:56:54Z

Hi @mbien,

I have no remote URLs to download from:

Is this as expected? How do I add one?

On other news, here's some stuff from my stdout (question: are we expected to System.out all artifacts in the local repo?)

...
     [exec] [Maven Repo Index Transfer/Scan] INFO org.eclipse.aether.internal.impl.DefaultArtifactResolver - Artifact org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-bom:pom:2.0.0 is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [central (https://repo.maven.apache.org/maven2, default, releases)]
     [exec] [Maven Repo Index Transfer/Scan] INFO org.eclipse.aether.internal.impl.DefaultArtifactResolver - Artifact org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-bom:pom:3.1.4 is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [central (https://repo.maven.apache.org/maven2/, default, releases+snapshots)]
     [exec] [Maven Repo Index Transfer/Scan] INFO org.eclipse.aether.internal.impl.DefaultArtifactResolver - Artifact org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-bom:pom:2.0.0 is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [central (https://repo.maven.apache.org/maven2/, default, releases+snapshots)]
     [exec] [Maven Repo Index Transfer/Scan] INFO org.eclipse.aether.internal.impl.DefaultArtifactResolver - Artifact org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.6 is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [central (https://repo.maven.apache.org/maven2/, default, releases+snapshots)]
     [exec] INFO [org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl]: Indexing of local took 18,09 s.

mbien · 2023-04-30T18:59:07Z

@vieiro this is a list of user given permissions. Try opening a pom or something which triggers indexing (e.g add dependency to a project via UI), nb will then show a notification, based on what you answer it will show up in the list. You don't have to change anything on the list unless you want to change the permission in post.

vieiro · 2023-04-30T19:24:03Z

@vieiro this is a list of user given permissions. Try opening a pom or something which triggers indexing (e.g add dependency to a project via UI), nb will then show a notification, based on what you answer it will show up in the list. You don't have to change anything on the list unless you want to change the permission in post.

Hi @mbien , thanks for the tip!

Works perfect now. I've tried to unpack the maven index from central, after a few minutes with fans at top speed and ~3.9Gb var/cache/mavenindex/ I decided to cancel it, and it's now disabled forever! Thanks very much for this!

matthiasblaesing

Thank you for your work. This is what I noticed:

I canceled a running remote index download, then I have no option to restart that, because the "Index Now" button only triggers a local index, not an index of the remote repositories. I did not look deeply into the mechanisms that hold the "indexing was started, but aborted" logic, but somewhere this seems to be stored, because it did not retrigger, when asking for completion. IMHO either the "Index Now" button also covers the remote repositories or the listview for the known repository should allow requesting an update on an individual basis.
The update frequency is still "Never", this contradicts my answer to allow downloading
Multithreaded indexing should IMHO be enabled by default (we should at least pretend, that hardware manufactures know how to build thermally stable hardware)

I suggest for the popup:

move text into bundle, so that it might be localiced

Text suggestion:

    pane.setText(MessageFormat.format(
            "Maven index download requested for: {0}<br/>"
            + "<ul>"
            + "<li><a href=\"allow-always\">Allow always</a></li>"
            + "<li><a href=\"allow-once\">Allow once</a></li>"
            + "<li><a href=\"deny-once\">Deny once</a></li>"
            + "<li><a href=\"deny-always\">Deny always</a></li>"
            + "</ul>"
            + "A repository index contains artifact metadata which is useful for some NetBeans features.<br />"
            + "<a href=\"disable-downloads\">Disable all remote downloads</a>",
            repo.getRepositoryUrl()
    ));

Leads to:

At least Jan noticed, that maven download insisted on downloading while on the bus, that might be a situation where I'd like it to "Deny once".

Looking at the indexing notification: Will this be a problem for LSP?

mbien · 2023-04-30T20:32:55Z

I canceled a running remote index download, then I have no option to restart that, because the "Index Now" button only triggers a local index, not an index of the remote repositories.

right, canceling will add a delay before it will try to run again. This is intended. I am not sure how important the usecase is to start the indexing right after a manual cancel again. I agree that the "index now" button needs to be looked at since it is only for the local index and this is not clearly communicated in the UI. Restart after cancel is more of a dev testing issue for me and less of a common usecase.

Multithreaded indexing should IMHO be enabled by default

I don't want to do this before filtering is available since this has an index size overhead. This would be another PR. We also got feedback of notebook usage etc which I don't want to ignore either.

I suggest for the popup:

I will take a look at this. But I won't implement "run once" for now since this complicates things unnecessarily. Indexing is not a run-once feature, either you turn it on or off. Cancelling pauses it as mentioned before.

Looking at the indexing notification: Will this be a problem for LSP?

likely. IndexingNotificationProvider.java has one method more now, if this isn't implemented and nobody sets the permission, indexing won't run now. Let me add a default implementation which simply allows all downloads to have the same behavior as before (assuming there is a IndexingNotificationProvider in the lookup!).

mbien

changes:

"update now" button will trigger updates for all repos (permission required, download pause is reset)
- if a remote repo is already up2date, it won't update the repo
- changed label text to "Check for Updates"
added a dummy impl for IndexingNotificationProvider to retain old behavior even when no proper implementation is found
extracted text to bundle, updated the notification a bit

IMHO (...) the listview for the known repository should allow requesting an update on an individual basis.

i think this might be a potential upgrade for some time later. I thought about it too but I opted for not allowing too much micromanagement for now.

mbien · 2023-05-01T00:58:30Z

java/maven.indexer/src/org/netbeans/modules/maven/indexer/NexusRepositoryIndexerImpl.java

+            if (!RepositoryPreferences.isIndexDownloadAllowedFor(repo)) {
+                IndexingNotificationProvider np = Lookup.getDefault().lookup(IndexingNotificationProvider.class);
+                if(np != null) {
+                    np.requestPermissionsFor(repo);
+                }
+                return true;


i thought about returning false when no IndexingNotificationProvider is found. But it is probably better to add a dummy implementation which simply allows everything. Once all users of this API have their own implementation, the dummy can be removed.

I think it is a better behavior to not a allow downloads by default if there is no provider.

matthiasblaesing · 2023-05-01T15:45:35Z

I think this is mostly good to go. The only thing missing is IMHO restoring periodic updates. Given, that you now clear all downloads, there is IMHO no reason to still keep it on never and it IMHO contradicts the "allow" setting.

vieiro · 2023-05-01T17:28:04Z

Great job, @mbien!

There's a "gobally" -> "globally" typo you may want to take a look at before merging.

mbien · 2023-05-01T20:19:30Z

There's a "gobally" -> "globally" typo you may want to take a look at before merging.

thanks! I wouldn't have noticed this.

The only thing missing is IMHO restoring periodic updates. Given, that you now clear all downloads, there is IMHO no reason to still keep it on never and it IMHO contradicts the "allow" setting.

good point. Probably better to remove the never option now. The usecase of never is also covered by the index downloads check box. Going to take a look at it.

mbien · 2023-05-02T06:37:02Z

last commit removes the never update frequency. Typo is fixed too but already squashed.

planning to squash it into two commits before merge

- module requires now jdk 11+ - apache lucene 9.5.0 with memory mapped index - will use panama on JDK 19 - multi-threaded remote index extraction (at least twice as fast as ST) - revert "hack: backported LUCENE-10431 hashcode hotfix to EOL lucene 8.11.1." (commit 35b41f6.)

- implement permission system via netbeans notifications - multithreaded extraction setting for maven indexer options - better temp storage handling (extract directly to cache folder) - cancellation sets a timeout, pausing starts of other repo updates - 'update now' button should update everything and reset timeout - remove 'never' update frequency since it was replaced by a combination of other settings - replace some deprecated usages

mbien · 2023-05-05T00:05:05Z

last test run with all tests enabled. thanks for the reviews -> going to merge today

mbien · 2023-05-05T04:09:36Z

merging earlier than planned since I got some time for testing and the next steps (search service/filters)

Lucene 9.6 is about to be released which has a better panama implementation which supports using panama on JDK 19 and 20 in contrast to only 19. Going to have to bump to that too once available.

mbien added Upgrade Library Library (Dependency) Upgrade Maven [ci] enable "build tools" tests labels Nov 22, 2022

mbien mentioned this pull request Nov 22, 2022

[MINDEXER-151] Speed up Index update from remote apache/maven-indexer#255

Merged

mbien force-pushed the indexer7 branch 3 times, most recently from b9c9ff2 to 6c8c9c9 Compare November 22, 2022 07:57

mbien mentioned this pull request Jan 5, 2023

Jakarta EE 10 - Platform/API/RI support #4692

Merged

mbien commented Jan 10, 2023

View reviewed changes

java/maven.indexer/manifest.mf Outdated Show resolved Hide resolved

mbien added the ci:all-tests [ci] enable all tests label Jan 13, 2023

mbien force-pushed the indexer7 branch 2 times, most recently from 4837a0b to c7b719f Compare January 13, 2023 09:27

mbien mentioned this pull request Jan 13, 2023

allow concurrent local indexing while downloading remote indexes. #5290

Merged

mbien added the ci:dev-build [ci] produce a dev-build zip artifact (7 days expiration, see link on workflow summary page) label Feb 27, 2023

mbien force-pushed the indexer7 branch from c7b719f to d7204d3 Compare February 27, 2023 12:21

mbien changed the title ~~upgrade to maven-indexer 7.0.0~~ upgrade to maven-indexer 7.0.1 Feb 27, 2023

mbien force-pushed the indexer7 branch from b136904 to 6d683fb Compare February 27, 2023 15:10

mbien added the performance label Feb 27, 2023

mbien force-pushed the indexer7 branch from 6d683fb to c04b890 Compare February 27, 2023 23:14

mbien mentioned this pull request Mar 12, 2023

Improved maven indexer failure modes in low space situations #5655

Merged

mbien removed the ci:all-tests [ci] enable all tests label Mar 16, 2023

mbien force-pushed the indexer7 branch from c04b890 to 975477a Compare March 16, 2023 06:00

mbien mentioned this pull request Apr 13, 2023

Unpacking index is extremely slow #5809

Closed

mbien mentioned this pull request Apr 19, 2023

prepare NB 19 release cycle for JDK 11. #5852

Merged

mbien force-pushed the indexer7 branch from 975477a to 5251824 Compare April 22, 2023 02:45

mbien force-pushed the indexer7 branch from 5e74b28 to c402bba Compare April 28, 2023 06:49

mbien requested a review from lkishalmi April 28, 2023 07:08

mbien added the Upgrade JDK Upgrade to the JDK requirements of a module. label Apr 28, 2023

mbien changed the title ~~upgrade to maven-indexer 7.0.1~~ upgrade to maven-indexer 7.0.1 and improve index downloads Apr 29, 2023

matthiasblaesing reviewed Apr 30, 2023

View reviewed changes

mbien force-pushed the indexer7 branch from c402bba to 2c8c642 Compare May 1, 2023 00:41

mbien commented May 1, 2023

View reviewed changes

mbien added the ci:all-tests [ci] enable all tests label May 1, 2023

apache locked and limited conversation to collaborators May 1, 2023

apache unlocked this conversation May 1, 2023

mbien removed ci:all-tests [ci] enable all tests do not merge Don't merge this PR, it is not ready or just demonstration purposes. labels May 1, 2023

mbien force-pushed the indexer7 branch from 3d5c12a to 41afb0e Compare May 2, 2023 06:28

mbien added the ci:all-tests [ci] enable all tests label May 2, 2023

matthiasblaesing approved these changes May 4, 2023

View reviewed changes

mbien added 2 commits May 5, 2023 01:55

mbien force-pushed the indexer7 branch from 41afb0e to c5b3b30 Compare May 5, 2023 00:03

mbien removed the Need Squashing label May 5, 2023

mbien merged commit 5d5a120 into apache:master May 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade to maven-indexer 7.0.1 and improve index downloads #4999

upgrade to maven-indexer 7.0.1 and improve index downloads #4999

mbien commented Nov 22, 2022 •

edited

Loading

mbien commented Feb 27, 2023

mbien commented Feb 27, 2023

mbien commented Mar 17, 2023 •

edited

Loading

vieiro commented Apr 30, 2023

mbien commented Apr 30, 2023 •

edited

Loading

vieiro commented Apr 30, 2023

matthiasblaesing left a comment

mbien commented Apr 30, 2023 •

edited

Loading

mbien left a comment •

edited

Loading

mbien May 1, 2023

matthiasblaesing commented May 1, 2023

vieiro commented May 1, 2023

mbien commented May 1, 2023

mbien commented May 2, 2023

mbien commented May 5, 2023

mbien commented May 5, 2023 •

edited

Loading

upgrade to maven-indexer 7.0.1 and improve index downloads #4999

upgrade to maven-indexer 7.0.1 and improve index downloads #4999

Conversation

mbien commented Nov 22, 2022 • edited Loading

mbien commented Feb 27, 2023

mbien commented Feb 27, 2023

mbien commented Mar 17, 2023 • edited Loading

vieiro commented Apr 30, 2023

mbien commented Apr 30, 2023 • edited Loading

vieiro commented Apr 30, 2023

matthiasblaesing left a comment

Choose a reason for hiding this comment

mbien commented Apr 30, 2023 • edited Loading

mbien left a comment • edited Loading

Choose a reason for hiding this comment

mbien May 1, 2023

Choose a reason for hiding this comment

matthiasblaesing commented May 1, 2023

vieiro commented May 1, 2023

mbien commented May 1, 2023

mbien commented May 2, 2023

mbien commented May 5, 2023

mbien commented May 5, 2023 • edited Loading

mbien commented Nov 22, 2022 •

edited

Loading

mbien commented Mar 17, 2023 •

edited

Loading

mbien commented Apr 30, 2023 •

edited

Loading

mbien commented Apr 30, 2023 •

edited

Loading

mbien left a comment •

edited

Loading

mbien commented May 5, 2023 •

edited

Loading