FileOp for file transporter #578
Conversation
Problem is that file transport is also remote transport, so things like checksumming are expected to happen (just like for HTTP or any other remote transport).
|
This work will save lot of disk space for large projects and it is definitely worth to expllre this option. I wonder how we can make this plugins that handle files and they don't expect links. But it only a matter of finding them and adding support. We should also think about dealing with the compability of these new modes with existing projects and caches maybe without asking the user to clean everything ? |
|
I'd like to know more about requirements, as transport-file is still transport (considered remote) with all the fluff remote transport is expected to do, like mentioned checksum calculation/comparison (and this Pr will work for smaller files, but as I read Javadoc, mapping size limit is 2GB?). |
Just like replaced `TestFileUtils.createTempFile` just create new path, as WinFS is wonky
|
@cstamas @eolivelli Wow! Thanks a lot for putting this together so quickly! @cstamas I'm not sure if you're talking about "internal requirements" or if you'd like to understand more about this setup where having the
EDIT: Seems like I had some reading to do first :) Found I'm happy to help in any way I can, though! :) |
|
I did some extensive tests with My experiments were based on the fact that having N repos mapped as "tail" local repos seemed analogous to declaring N repositories via Unfortunately I think symlinks would be much easier for Maven plugin developers to adapt to. Symlinks might also even work out of the box in some cases. In fact, some of the tests I did with It seems this will be a trade-off between how unsupported |
|
Hej, sadly yes. The legacy bits in Maven effectively prevent ChainedMavenRepository "mainstream" use, see issue for explanation: https://issues.apache.org/jira/browse/MNG-7706 In short, all "modern" plugins are ok, but in a moment this legacy API is touched (is deprecated btw), the chained falls apart. Quarkus plugin is different beast from that story above, as they already have their own chained implementation, moreover, they construct their own Resolver embedded in Maven... And as for symlinking (as hardlinking fails on Windows for some reason), it may work and save disk space (with all the shortcomings already mentioned, like all the plugins should be able to handle symlinks in local repo) but speed wise it may not be huge boost, since as I said, "file" transport is remote transport, and hence, checksumming (and in future, things like signature validation) is must and expected. |
|
On related note: due some strange habit, people wanting "latest and greatest" (features and compatibility) are NOT compiling against latest, and our effort to "signal" people to move off APIs (by deprecation) goes completely unnoticed, like here: repaint-io/maven-tiles#157 Hence we started adding "plugin warnings" that again, caused user uproar :) All that fuss was partially related to "move off deprecated ArtifactRepository classes" among other things, as we knew we cannot improve underlying LRM (in Resolver) without Maven cruft following it. |
|
Have to add: this is big time ourselves committing a mistake as well, as we (as in "maven project") let "two parallel worlds" (Resolver and Maven2 inherited APIs) coexists for too long (two repository APIs, even to artifact and artifact version APIs), without giving any guidance to users "where to go". This is now changing, and Maven3 to Maven4 migration will have it's rough edges, and will require "vigilance" from our users too, but Maven4 goal is to keep lives of those migrating simple (so Maven3 plugins not touching into deepest internals will still work as before). Extensions are completely different story... |
Got it. Makes total sense. I guess the only real performance gain then would be in not having to actually move bits around and write them as new files. Hard to say if it's worth it to have it or not, though. As for all the complexity in managing deprecations and having people actually act on using the new stuff, I can only imagine how careful the considerations must be to change/evolve something so widely used like Maven, especially in a foundational part like Resolver. If I may, I guess the simple-enough repository structure and mental model that Maven has for dealing with dependencies kind of encourages people to endeavor a little more on its internals (i.e. "How hard can it be?" 😅), extending it in ways that seem safe but have all sorts of evolution and compatibility problems.. But that's just my limited perspective on it as a user 😝 I'm glad to hear that Maven4 will try and address some of those issues! I'm looking forward for the first stable release. In the mean time, I couldn't let it go with the Quarkus Maven Plugin and... I got it working. I'm pretty sure this is not the whole thing, but at least my immediate problem was solved by this little patch: tiagobento/quarkus@86a25e0 I did it against the 3.8.4 tag (because that's what I'm using in my project, currently), but I'll try and send a PR targeting the latest stream. Hopefully I can get it to be part of a future patch release on the next LTS coming up alter this month (3.15). @cstamas Thank you so much for pointing out that the CLRM mechanism exists. It may be the final missing piece on this whole thing I'm trying to do. As for the symlinks, I guess my need would be a lot less latent now that I understand the gains may not be as great as I initially imagined. Of course, if you want to continue pursuing the feasibility of both symbolic and hard links, I'll be available for helping you test it if you need it! |
|
@aloubyansky ping, especially the interesting patch tiagobento/quarkus@86a25e0 |
Problem is that file transport is also remote transport, so things like checksumming are expected to happen (just like for HTTP or any other remote transport).