Pants triggering a rebuild even if dev-dependency updated

[asgoel]

Describe the bug
When I upgrade a package that is defined in pyproject.toml (via poetry_requirements) dev dependencies that my binary does not explicitly depend on (such as something like pytest), pants will still list the binary as something that has "been changed" and thus try and rebuild the binary. I think this is because the pyproject.toml file itself ends up being listed as a transitive dependency of the original so whenever the file changes, pants thinks the binary needs to be re-built. Is this intentional/is there a way to get around it?

Pants version
2.11

OS
Mac and Linux

Additional info
Add any other information about the problem here, such as attachments or links to gists, if relevant.

[jsirois]

To put some concrete spin on this, in the Pants repo, this change:

diff --git a/3rdparty/python/requirements.txt b/3rdparty/python/requirements.txt
index 551931f1e..0e7e39f0b 100644
--- a/3rdparty/python/requirements.txt
+++ b/3rdparty/python/requirements.txt
@@ -4,6 +4,8 @@
 # Additionally, it increases the surface area of Pants's supply chain for security.
 # Consider pinging us on Slack if you're thinking a new dependency might be needed.

+cowsay
+
 ansicolors==1.1.8
 chevron==0.14.0  # Should only be used by build-support.
 fasteners==0.16.3

Leads to this claim:

$ ./pants --changed-since=HEAD --changed-dependees=transitive filter --target-type=pex_binary
build-support/bin#_generate_all_lockfiles_helper.py
build-support/bin#_release_helper.py
build-support/bin#changelog.py
build-support/bin#generate_docs.py
build-support/bin#generate_github_workflows.py
build-support/bin#generate_user_list.py
build-support/bin#reversion.py
src/python/pants/bin:pants
testprojects/src/python/hello/main:main
testprojects/src/python/native:main

So nothing in the repo depends on the new depency cowsay, yet I get back from --changed-since that every pex_binary target has changed, which is certainly untrue. So --changed-since is currently "may have changed since". As @asgoel points out, it would be nice if we could say "definitely has changed since".

[cburroughs]

Unsure if this is orthogonal or a "happy accident" ontop of the underlying issue, but in #20368 I'm relying on "3rdparty/py acting as a dependency of the entire lockfile.

[cburroughs]

After thinking about this more, I'm not sure this is feasible without Pants having a much better sense of the transitive dependency graph a la #12733. If a transitive 3rdparty dependency changed, I'd still want to count that as a "change" and run the test.

[huonw]

I think pex-tool/pex#2411 might help allow reducing the severity of some aspects of this, by allowing more process-caching to kick in.

If we could subset a lockfile, we could optimise building a pex_binary (or whatever) by:

1. first cheaply compute the relevant subset of a lockfile, then
2. use the subset as input to the more expensive "build pex_binary" (or whatever) processes

If the lockfile hasn't changed at all, both will be able to be served from cache. If the lockfile has changed but the subset has not, the first will rerun, but the second process will be able to be served from cache. If the subset has changed, both processes run.

This doesn't help with eliminating the static dependencies, meaning --changed-dependents=transitive will still flag things that may not "need" to run.