Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add documentation for coursier usage #5382

Closed
wants to merge 2 commits into from
Closed

Add documentation for coursier usage #5382

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1c05a23
Add documentation for coursier usage
wisechengyi Jan 23, 2018
0cc269f
addr comment
wisechengyi Jan 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions examples/src/java/org/pantsbuild/example/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,57 @@ Toolchain
Pants uses [Ivy](http://ant.apache.org/ivy/) to resolve `jar` dependencies. To change how Pants
resolves these, configure `resolve.ivy`.

Starting release `1.4.0.dev26`, Pants added an option to pick [coursier](https://github.com/coursier/coursier)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: s/Starting release/At release/

as the JVM 3rdparty resolver, with performance improvement being the main motivation. The goal is to retire ivy
eventually. Example config to use coursier:

:::ini
# This will turn on coursier and turn off ivy.
[resolver]
resolver: coursier

[resolve.coursier]
# jvm option in case of large resolves
jvm_options: ['-Xmx4g', '-XX:MaxMetaspaceSize=256m']
# The opposite of resolve.ivy.soft_excludes
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we have #4437 outstanding, is it too late to make soft_excludes the default for coursier?

Copy link
Contributor Author

@wisechengyi wisechengyi Jan 23, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I made the default behavior consistent with ivy in oss 35e711f#diff-0b873245a3a9ee84a1f817502a221652R52

If you think we are not crossing release boundary yet (not sure if 1.4.0rc0 counts), I can just put a patch to flip the default.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be happy to cherry pick that in.

allow_global_excludes: False

[cache.resolve.coursier]
# In order to enable remote caching, need to relativize the classpath entries serialized.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you show how to turn off remote caching but leave local caching? Is that even possible on a cache-by-cache basis? I don't remember any more.

Copy link
Member

@stuhood stuhood Jan 26, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjyw : Yea, it's possible.

but...

@wisechengyi : Is this code even writing to the cache? If not, these settings shouldn't be relevant. If it is, you should probably just disable cache writing for now?

I honestly think that invalidation (without caching) might be sufficient for coursier.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that --cache-ignore would even skip fingerprinting which coursier resolve needs, so if there is a way to still do fingerprinting but disabling cache checking, that'll be great.

Copy link
Member

@stuhood stuhood Jan 26, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline: if the code doesn't write to the cache, then these settings don't matter.

And I think that we should almost certainly do our first caching of coursier via #4397, rather than via the v1 Task caching.

# https://github.com/pantsbuild/pants/issues/5187
# We still want to compute fingerprints so we can validate or invalidate,
# but do not want to check local or remote cache.
read: False
write: False

[export]
# Same if needed for large resolves
jvm_options: ['-Xmx4g', '-XX:MaxMetaspaceSize=256m']

[coursier]
# Change the following if you choose to [build coursier jar from scratch](https://github.com/coursier/coursier/blob/master/DEVELOPMENT.md#build-with-pants)
# or to fetch from different location.
bootstrap_jar_url: <url>
version: <version>

fetch_options: [
# Specify maven repos
'-r', 'https://repo1.maven.org/maven2/',
Copy link
Contributor

@benjyw benjyw Jan 26, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there examples for custom repo URL formats using placeholders? The equivalent of this in an ivysettings:

<url name="foo.bar">
  <artifact pattern="https://foo.com/bar/[organisation]/[artifact]/[revision]/[artifact]-[revision].[ext]" />
</url>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, I just tried to emulate that with this:

'-r', 'ivy:https://foo.com/bar/[organisation]/[artifact]/[revision]/[artifact]-[revision].[ext]'

But unfortunately it replaces the [artifact] placeholder with the string "ivy", instead of with the name of my artifact. So maybe ivy emulation is broken?

Copy link
Contributor

@benjyw benjyw Jan 26, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out that [artifact] should be replaced with [module]. So this is an example of the kind of thing that should go into a "how to migrate from ivy" tutorial!!

Update: it's actually a bit more complicated than that - module and artifact are two distinct ivy concepts, both of which appear to map to maven "artifact", at least for the purpose of placeholder substitution. Not sure why Coursier would decide that artifact should be "ivy", maybe because it's looking for "ivy.xml"...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But unfortunately it replaces the [artifact] placeholder with the string "ivy", instead of with the name of my artifact. So maybe ivy emulation is broken?

if you suspect resolving against ivy repo is broken, please file a ticket at https://github.com/coursier/coursier with a minimal repro.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh but there are still other issues. For example, it appears to insist on the presence of an ivy.xml file, which Ivy itself does not.

Copy link
Contributor

@benjyw benjyw Jan 26, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, the behavior is weird. It assumes that your pattern ends with /[artifact].[ext] so that it can substitute artifact=ivy, ext=xml and fetch an ivy.xml. But if your pattern doesn't end with those, it'll still do the substitution and try to fetch the file . So, e.g., if your pattern ends with /[module].jar it'll fetch the jar and then try to parse it as XML...

Ivy deals with this by having two separate patterns in the settings - one for the artifacts and one for the ivy.xml.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See coursier/coursier#754 and coursier/coursier#755

'-r', 'https://dl.google.com/dl/android/maven2/',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this with with local filesystem repos? If so, an example would be great.


# Quiet mode
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't the output consumed by the workunit? Is quiet required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will be consumed by the workunit, but it's good to turn on sometimes to visualize the resolve on pants server.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is consumed by workunit. but sometimes printing into workunit and viewing from pants server helps debugging.

'-q',

# Do not use default public maven repo.
'--no-default',

# Concurrent workers
Copy link
Member

@stuhood stuhood Jan 23, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like something pants should set by inspecting available cores (or just letting coursier do that... presumably this value has a default?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filed #5384

'-n', '10',

# Specify the type of artifacts to fetch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto... this feels like something that the task should set.

'-A', 'jar,bundle,test-jar,maven-plugin,src,doc,aar'
]

Pants uses [Nailgun](https://github.com/martylamb/nailgun) to speed up compiles. Nailgun is a
JVM daemon that runs in the background. This means you don't need to start up a JVM and load
classes for each JVM-based operation. Things go faster.
Expand Down