Add documentation for coursier usage #5382

wisechengyi · 2018-01-23T21:55:17Z

No description provided.

baroquebobcat

Looks pretty good to me, a part from one bit of duplication.

baroquebobcat · 2018-01-23T22:05:22Z

examples/src/java/org/pantsbuild/example/README.md

+        '-A', 'jar,bundle,test-jar,maven-plugin,src,doc,aar'
+      ]
+
+    [cache.resolve.coursier]


you've got cach.resolve.coursier in here twice.

ah thanks. corrected

stuhood

I'm concerned that we're exposing all of coursier's options right out of the box, as that seems likely to get us in the same situation we were in with ivy, where if someone had a custom config, it was very difficult to validate that it didn't break certain rules like the ivy_cache_dir location.

If it wouldn't be too restrictive, would consider starting with something like a --maven-resolvers list option, and then expanding the options we expose as we go. Maybe with an escape hatch for "--extra-coursier-options", or something.

stuhood · 2018-01-23T22:54:40Z

examples/src/java/org/pantsbuild/example/README.md

+    [resolve.coursier]
+    # jvm option in case of large resolves
+    jvm_options: ['-Xmx4g', '-XX:MaxMetaspaceSize=256m']
+    # The opposite of resolve.ivy.soft_excludes


Given that we have #4437 outstanding, is it too late to make soft_excludes the default for coursier?

Ah I made the default behavior consistent with ivy in oss 35e711f#diff-0b873245a3a9ee84a1f817502a221652R52

If you think we are not crossing release boundary yet (not sure if 1.4.0rc0 counts), I can just put a patch to flip the default.

I'd be happy to cherry pick that in.

stuhood · 2018-01-23T22:56:30Z

examples/src/java/org/pantsbuild/example/README.md

+        '-r', 'https://repo1.maven.org/maven2/',
+        '-r', 'https://dl.google.com/dl/android/maven2/',
+
+        # Quiet mode


Isn't the output consumed by the workunit? Is quiet required?

it will be consumed by the workunit, but it's good to turn on sometimes to visualize the resolve on pants server.

it is consumed by workunit. but sometimes printing into workunit and viewing from pants server helps debugging.

stuhood · 2018-01-23T22:57:12Z

examples/src/java/org/pantsbuild/example/README.md

+        # Do not use default public maven repo.
+        '--no-default',
+
+        # Concurrent workers


This seems like something pants should set by inspecting available cores (or just letting coursier do that... presumably this value has a default?)

filed #5384

stuhood · 2018-01-23T22:57:26Z

examples/src/java/org/pantsbuild/example/README.md

+        # Concurrent workers
+        '-n', '10',
+
+        # Specify the type of artifacts to fetch


Ditto... this feels like something that the task should set.

benjyw · 2018-01-26T19:49:15Z

examples/src/java/org/pantsbuild/example/README.md

@@ -211,6 +211,57 @@ Toolchain
 Pants uses [Ivy](http://ant.apache.org/ivy/) to resolve `jar` dependencies. To change how Pants
 resolves these, configure `resolve.ivy`.

+Starting release `1.4.0.dev26`, Pants added an option to pick [coursier](https://github.com/coursier/coursier)


Nit: s/Starting release/At release/

benjyw · 2018-01-26T19:52:05Z

examples/src/java/org/pantsbuild/example/README.md

+
+    fetch_options: [
+        # Specify maven repos
+        '-r', 'https://repo1.maven.org/maven2/',


Are there examples for custom repo URL formats using placeholders? The equivalent of this in an ivysettings:

<url name="foo.bar"> <artifact pattern="https://foo.com/bar/[organisation]/[artifact]/[revision]/[artifact]-[revision].[ext]" /> </url>

For example, I just tried to emulate that with this:

'-r', 'ivy:https://foo.com/bar/[organisation]/[artifact]/[revision]/[artifact]-[revision].[ext]'

But unfortunately it replaces the [artifact] placeholder with the string "ivy", instead of with the name of my artifact. So maybe ivy emulation is broken?

Turns out that [artifact] should be replaced with [module]. So this is an example of the kind of thing that should go into a "how to migrate from ivy" tutorial!!

Update: it's actually a bit more complicated than that - module and artifact are two distinct ivy concepts, both of which appear to map to maven "artifact", at least for the purpose of placeholder substitution. Not sure why Coursier would decide that artifact should be "ivy", maybe because it's looking for "ivy.xml"...

But unfortunately it replaces the [artifact] placeholder with the string "ivy", instead of with the name of my artifact. So maybe ivy emulation is broken?

if you suspect resolving against ivy repo is broken, please file a ticket at https://github.com/coursier/coursier with a minimal repro.

Ugh but there are still other issues. For example, it appears to insist on the presence of an ivy.xml file, which Ivy itself does not.

OK, the behavior is weird. It assumes that your pattern ends with /[artifact].[ext] so that it can substitute artifact=ivy, ext=xml and fetch an ivy.xml. But if your pattern doesn't end with those, it'll still do the substitution and try to fetch the file . So, e.g., if your pattern ends with /[module].jar it'll fetch the jar and then try to parse it as XML...

Ivy deals with this by having two separate patterns in the settings - one for the artifacts and one for the ivy.xml.

See coursier/coursier#754 and coursier/coursier#755

benjyw · 2018-01-26T19:54:46Z

examples/src/java/org/pantsbuild/example/README.md

+    allow_global_excludes: False
+
+    [cache.resolve.coursier]
+    # In order to enable remote caching, need to relativize the classpath entries serialized.


Can you show how to turn off remote caching but leave local caching? Is that even possible on a cache-by-cache basis? I don't remember any more.

@benjyw : Yea, it's possible.

but...

@wisechengyi : Is this code even writing to the cache? If not, these settings shouldn't be relevant. If it is, you should probably just disable cache writing for now?

I honestly think that invalidation (without caching) might be sufficient for coursier.

My understanding is that --cache-ignore would even skip fingerprinting which coursier resolve needs, so if there is a way to still do fingerprinting but disabling cache checking, that'll be great.

Discussed offline: if the code doesn't write to the cache, then these settings don't matter.

And I think that we should almost certainly do our first caching of coursier via #4397, rather than via the v1 Task caching.

benjyw · 2018-01-26T19:56:34Z

examples/src/java/org/pantsbuild/example/README.md

+    fetch_options: [
+        # Specify maven repos
+        '-r', 'https://repo1.maven.org/maven2/',
+        '-r', 'https://dl.google.com/dl/android/maven2/',


Does this with with local filesystem repos? If so, an example would be great.

benjyw

I agree with Stu that maybe we don't need to expose all coursier options? At least not in the documentation. People can always go read the coursier docs for those.

But what's still missing for me is a "here's how to migrate from ivy" tutorial. Specifically, how to convert your ivysettings.xml to coursier options.

benjyw · 2018-01-26T20:48:17Z

Is there an equivalent of an ivy report for coursier? Its error messages are not very helpful, in general. E.g.,

 java.lang.AssertionError: assertion failed
                       	at scala.Predef$.assert(Predef.scala:156)
                       	at coursier.TermDisplay$UpdateDisplayRunnable.newEntry(TermDisplay.scala:165)
                       	at coursier.TermDisplay.downloadingArtifact(TermDisplay.scala:444)
                       	at coursier.Cache$$anonfun$remote$1$1$$anonfun$apply$26.apply(Cache.scala:669)
                       	at coursier.Cache$$anonfun$remote$1$1$$anonfun$apply$26.apply(Cache.scala:669)
                       	at scala.Option.foreach(Option.scala:257)
                       	at coursier.Cache$$anonfun$remote$1$1.apply(Cache.scala:669)
                       	at coursier.Cache$$anonfun$remote$1$1.apply(Cache.scala:556)
                       	at scalaz.concurrent.Task$.Try(Task.scala:456)
                       	at scalaz.concurrent.Task$$anonfun$apply$25.apply(Task.scala:353)
                       	at scalaz.concurrent.Task$$anonfun$apply$25.apply(Task.scala:353)
                       	at scalaz.concurrent.Future$$anonfun$apply$15$$anon$3.call(Future.scala:432)
                       	at scalaz.concurrent.Future$$anonfun$apply$15$$anon$3.call(Future.scala:432)
                       	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                       	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                       	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                       	at java.lang.Thread.run(Thread.java:748)

benjyw · 2018-01-26T20:55:12Z

I'd love to switch to Coursier, because Ivy is so painful and over-engineered and complex. But so far Coursier seems to be not great at dealing with anything other than "download from a standard Maven-style repo"...

benjyw · 2018-01-26T21:22:23Z

BTW while banging on Coursier I noticed a minor issue around workunits in the resolve task. Filed an issue.

wisechengyi · 2018-02-01T06:29:08Z

Thanks @baroquebobcat @stuhood @benjyw. Let me summarize and see if there's anything missing.

Limit the options exposed to user to, e.g. number of workers, list of maven repos. Move others to advanced options if user wants to further dig up coursier options.
Suggest the migration path from ivy. Clearly call out what coursier cannot do.
Suggest the generic method for debugging coursier resolves. @benjyw the particular stacktrace was fixed already on master as discussed in slack.
Leave caching options default, because [resolve.coursier] writes to results_dir but does not enable caching.

stuhood · 2018-02-01T23:28:49Z

Yea, sounds about right. Thanks.

Suggest the migration path from ivy. Clearly call out what coursier cannot do.

I think that this boils down to either describing or scripting conversion of the most commonly configured bits of ivy: basically, resolvers.

Suggest the generic method for debugging coursier resolves.

Yea. Some equivalent of ./pants resolve.ivy --open (even if it were text based) will be important.

wisechengyi · 2018-04-08T07:50:11Z

Closed in favor of #5675

Address points from #5382 * Extract the following coursier options and assign sensible defaults * artifact types * cache dir * fetch options * repos * Add `--report` flag to show coursier output * Add documentation

Add documentation for coursier usage

1c05a23

wisechengyi requested review from baroquebobcat, benjyw and mateor January 23, 2018 21:56

baroquebobcat approved these changes Jan 23, 2018

View reviewed changes

addr comment

0cc269f

stuhood reviewed Jan 23, 2018

View reviewed changes

benjyw reviewed Jan 26, 2018

View reviewed changes

wisechengyi mentioned this pull request Apr 8, 2018

Make coursier resolve more friendly #5675

Merged

wisechengyi closed this Apr 8, 2018

wisechengyi deleted the coursier_doc branch April 9, 2018 23:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation for coursier usage #5382

Add documentation for coursier usage #5382

wisechengyi commented Jan 23, 2018

baroquebobcat left a comment

baroquebobcat Jan 23, 2018

wisechengyi Jan 23, 2018

stuhood left a comment •

edited

Loading

stuhood Jan 23, 2018

wisechengyi Jan 23, 2018 •

edited

Loading

stuhood Jan 26, 2018

stuhood Jan 23, 2018

wisechengyi Jan 23, 2018

wisechengyi Jan 23, 2018

stuhood Jan 23, 2018 •

edited

Loading

wisechengyi Jan 23, 2018

stuhood Jan 23, 2018

benjyw Jan 26, 2018

benjyw Jan 26, 2018 •

edited

Loading

benjyw Jan 26, 2018

benjyw Jan 26, 2018 •

edited

Loading

wisechengyi Jan 26, 2018

benjyw Jan 26, 2018

benjyw Jan 26, 2018 •

edited

Loading

benjyw Jan 26, 2018

benjyw Jan 26, 2018

stuhood Jan 26, 2018 •

edited

Loading

wisechengyi Jan 26, 2018

stuhood Jan 26, 2018 •

edited

Loading

benjyw Jan 26, 2018

benjyw left a comment

benjyw commented Jan 26, 2018

benjyw commented Jan 26, 2018

benjyw commented Jan 26, 2018

wisechengyi commented Feb 1, 2018

stuhood commented Feb 1, 2018

wisechengyi commented Apr 8, 2018

Add documentation for coursier usage #5382

Add documentation for coursier usage #5382

Conversation

wisechengyi commented Jan 23, 2018

baroquebobcat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wisechengyi Jan 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Jan 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjyw Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjyw Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjyw Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjyw left a comment

Choose a reason for hiding this comment

benjyw commented Jan 26, 2018

benjyw commented Jan 26, 2018

benjyw commented Jan 26, 2018

wisechengyi commented Feb 1, 2018

stuhood commented Feb 1, 2018

wisechengyi commented Apr 8, 2018

stuhood left a comment •

edited

Loading

wisechengyi Jan 23, 2018 •

edited

Loading

stuhood Jan 23, 2018 •

edited

Loading

benjyw Jan 26, 2018 •

edited

Loading

benjyw Jan 26, 2018 •

edited

Loading

benjyw Jan 26, 2018 •

edited

Loading

stuhood Jan 26, 2018 •

edited

Loading

stuhood Jan 26, 2018 •

edited

Loading