python_distribution plugin conflicts with pants publish --changed-since

[idan-at]

Describe the bug
Assuming X is being published, and X depends on Y, pants publish will also run package, for both X & Y.
Pants does not provide any context to plugins, so its impossible for a plugin running for Y, in charge of calculating its version, to know which version to provide.

Pants version
2.16.0

OS
Both

Additional info
I'm managing a Python monorepo with Pants. As part of my CI flow, I'm running:

pants --changed-dependents=transitive --changed-since=$last_publish_commit_sha publish

I also have a custom plugin that calculates the next version based on existing git tags (it also pushes the tag).
Assuming X changed, and X uses Y, in order to generate setup.py for X, pants packages Y for its version.
Because the plugin assumes it is invoked for changed packages only, it returns the next version of Y, which never gets published.

Slack thread
Somewhat Related issues: #18179, #16360

[idan-at]

@benjyw any thoughts on this one? 🙏

[benjyw]

Hey @idan-at, this is a tricky one. Sounds like we need a proper API for generating versions, that knows which files in the closure of a given target have change. I.e., to separate "generating setup.py" from "publishing".

[idan-at]

Indeed sounds complex. Is there anything I can do to push this effort?

[benjyw]

A helpful thing would be to come up with a suggestion for what you think the API should look like - since you already have a plugin that needs to generate versions, you are in a good position to suggest a useful design.

[idan-at]

WDYT about adding a new option for --changed-dependents, all, which will traverse the dependencies trees both ways? (transitively).

For example, if A depends on B which depends on C, and B changes, using the new flag will run the command on A,B,C.

[benjyw]

Hmm, I think it's more like we need to calculate the version in a separate pass than the full publish, no? You'd need to do that regardless of what you publish.

[idan-at]

Yeah that will help, but from my understanding, that's a deep change to how publish currently works, making it less likely to happen 😅

[benjyw]

Can you elaborate on how --changed-dependents would solve your issue? The behavior today is to invoke for all dependents in order to get their versions, but if you turn that off how would you get the version for Y? Is your custom plugin set up so that the version is stored somewhere?

[idan-at]

Yes. Some background about the plugin:
The plugin calculates the version according to the existing git tag. So if there's a tag X-1.0.0, then the version will be determined as 1.0.1.

Now, if A uses B and only A changes, but a version request is also sent to the plugin for B, then B's version will also be bumped (the plugin can't tell if B changed or not, so it can't tell if it needs to return the version from the latest git tag, or to rely on it to return the next one).

If we'll provide the all option, then both A and B will be published, and the versions will work.
The solution is not optimal and is just a patch (instead of only publishing A, we'll have to publish A's entire branch). It's suboptimal, but it still reduces the number of packages being published in a large monorepo.

[benjyw]

Oh I see, so you'd actually publish more than you do today, or really have to (you'd publish A and B, when really you only need to publish A) just so that the versions line up?

[idan-at]

Exactly (but it’s still better than publishing all the repository packages)

[benjyw]

I'd be leery of baking that into Pants, I feel like this requires a better solution. But as a workaround I think you can get this behavior with no modifications to Pants, using --changed-since=main --changed-dependents=transitive --filter-target-type=python_distribution to select all the distributions that are affected?

[benjyw]

Oh wait, no, that gives the dependents, you want the dependencies. So you'd need a more complicated query.

[benjyw]

You might have to dump your build graph using pants peek and write some custom logic outside of Pant, but it's do-able.