Crawl (remote) Maven repos

[vorburger]

@cstamas is there something somewhere (in Mima, Maven; or elsewhere that you know of) to "crawl" e.g. https://repo1.maven.org/maven2/ ? I mean something which doesn't start with a specific Artifact, but which just, given a (remote) Maven repo URL, gives you "all the Artifacts on that repo".

If that does not exist, I guess I could also do it by hacking my very own HTTP Directory Index HTML page parser (probably using jsoup) ... but maybe there are already libraries which can do this in the Maven ecosystem which I am not aware of?

[cstamas]

Ha! This is just a renewed debate in Maven PMC, as state of affairs with Maven repo part is a bit strange: it is held by an US company (history). We (Maven PMC) do want access to Maven Central.

As on surface:

• crawling is forbidden (your IP will be banned)
• index published is scarce, and may become even more scarce

So do not crawl.

There ARE simple workarounds:

• https://github.com/apache/maven-indexer backend-remoterepository
• needs (and has broken heuristics), like Toolbox LIST command

[cstamas@blondie mima (main)]$ jbang toolbox@maveniverse
Toolbox 0.6.1 (MIMA Runtime 'standalone-static' version 2.4.21)

          Maven version 3.9.9
                Managed true
                Basedir /home/cstamas/Worx/maveniverse/mima
                Offline false

             MAVEN_HOME /home/cstamas/.sdkman/candidates/maven/current
           settings.xml /home/cstamas/.sdkman/candidates/maven/current/conf/settings.xml
         toolchains.xml /home/cstamas/.sdkman/candidates/maven/current/conf/toolchains.xml

              USER_HOME /home/cstamas/.m2
           settings.xml /home/cstamas/.m2/settings.xml
  settings-security.xml /home/cstamas/.m2/settings-security.xml
       local repository /home/cstamas/.m2/repository-oss

               PROFILES
                 Active [oss-development]
               Inactive []

    REMOTE REPOSITORIES
                        central (https://repo.maven.apache.org/maven2/, default, releases)
prompt> list org.apache.maven
org.apache.maven:apache-maven
org.apache.maven:archetype
...

Etc. But there are all (broken) circumventions, again, Maven PMC is working on getting access to whole central for exactly this purpose.
https://bsky.app/profile/brunoborges.bsky.social/post/3lexnq3cimc2e

[vorburger]

FTR: #1072 (comment) ==> https://storage-download.googleapis.com/maven-central/index.html (and https://storage-download.googleapis.com/maven-central/ which has at least some indexes) is on a public GCP GCS bucket.

[vorburger]

https://mvnrepository.com/repos has LOTs.

I'll maintain a list of well known pulic Maven repos here:

• https://repo.maven.apache.org/maven2/
• https://storage-download.googleapis.com/maven-central/ (a mirror, example artifact)
• https://repository.jboss.org/nexus/service/local/ (https://repository.jboss.org/nexus/content/repositories/releases/ ?)
• https://maven.google.com (Android artifacts)
• http://repo.spring.io/release (source)
• https://clojars.org/repo/
• https://jitpack.io
• https://maven.pkg.github.com/OWNER/REPOSITORY GitHub Repos' Packages Apache Maven Registry (examples?)
• https://repo.jenkins-ci.org/releases/ (found on https://docs.deps.dev/api/v3alpha/#data, see Deps.dev #1084)

In the future I'll publish and maintain this in a repos.ttl, instead of in this comment.