Allow subclassing Fields for custom Field behavior with the Target API
#9286
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Eric-Arellano
requested review from
stuhood,
benjyw,
illicitonion,
TansyArron and
gshuflin
TansyArron
reviewed
Mar 13, 2020
| f"The target `{self}` does not have a field `{field}`. Before calling " | ||
| f"`my_tgt.get({field.__name__})`, call `my_tgt.has_fields([{field.__name__}])` to " | ||
| "filter out any irrelevant Targets." | ||
| ) |
There was a problem hiding this comment.
👍 💯 for good error messages!
TansyArron
approved these changes
Mar 13, 2020
Eric-Arellano
added a commit
that referenced
this pull request
Mar 16, 2020
This fleshes out how we use `AsyncField`s, which are much more complex than `PrimitiveField`s. ## Result ```python sources = await Get[SourcesResult](SourcesRequest, tgt.get(Sources).request) print(sources.snapshot.files) ``` This also works: ```python if tgt.has_fields([PythonLibrarySources]): sources1 = await Get[Sources](SourcesRequest, tgt.get(PythonLibrarySources).request) sources2 = await Get[Sources](SourcesRequest, tgt.get(Sources).request) assert sources1 == sources2 ``` `PythonSources` and its subclasses will validate that all resulting files end in `*.py` (new behavior). `PythonLibrarySources` and `PythonTestsSources` will use the previous default globs. `PythonBinarySources` will enforce that `sources` is 0 or 1 files (previous behavior). ## Solution ### Ensuring support for subclassed `AsyncField`s With the Target API, we allow new targets to subclass `Field`s for custom behavior. For example, `PythonLibrarySources` might use the default globs of `*.py` whereas `PythonTestSources` might use the default globs of `test_*.py`. To allow these custom subclasses of `Field`s, we added support in #9286 for substituting in the subclass with the original parent class. For example, `my_python_library.get(Sources) == my_python_library.get(PythonSources) == my_python_library.get(PythonLibrarySources)`. This works great with `PrimitiveField` but is tricky to implement with `AsyncField` due to the engine not supporting subclasses. Originally, I tried achieving this extensibility through a union, which would allow the engine to have multiple ways to get a common result type like `SourcesResult`. But, this created a problem that there became multiple paths in the rule graph to compute the same product, e.g. `Sources->SourcesResult`, `PythonSources->SourcesResult`, etc. Instead, each `AsyncField` should define a simple `Request` dataclass that simply wraps the underlying `AsyncField`. This allows us to have only one path from `SourcesRequest -> SourcesResult`, but still give custom behavior in the underlying `SourcesRequest`. Within the hydration rule, the rule will call standardized extension points provided by the underlying field. **This means that the onus is on the `AsyncField` author to expose certain entry points for customizing the field's behavior.** For example, `Sources` defines the entry points of `default_globs` and `validate_snapshot()`. `Dependencies` might provide entry points like `inject_dependencies()` and `validate_dependencies()` (not necessarily, only possibilities). While this approach has lots of boilerplate and less extensibility than `PrimitiveField`s, it solves the graph ambiguity and still allows for subclassing an `AsyncField`. ### Fixing `__eq__` for `Field`s The previous naive implementation resulted in `Field`s only comparing their classvar `alias`, rather than their actual underlying values. This meant that the engine would cache values when it should not have. This tweaks how we use dataclasses to ensure that the engine works correctly with `AsyncField`s.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
One of the requirements for the Target API's extensibility goal is that plugin authors can create new target types that behave almost identically to the pre-existing target type, but that somehow override a field. For example,
python3_librarymay override theCompatibilityfield to always returnCPython>=3.5.(We don't allow plugins to change the behavior of a pre-existing target. You must create a new target type.)
Plugin authors may subclass a particular
Fieldcurrently, but then core Pants code will stop working correctly. For example, Core Pants code may filter out targets by theCompatibilityfield. So, aPython3Compatibilityfield that subclassesCompatibilitywould no longer be recognized when callingmy_tgt.has_fields([Compatibility])ormy_tgt.get(Compatibility), asPython3Compatibility != Compatibility, even though it is a subclass.Solution
Modify
Target.get()andTarget.has_fields()to first check if the exact type is registered on the Target, then fall back to checking if the requested Field is a superclass of any of the registered fields.Result
A plugin author can now define
Python3Library, which is identical toPythonLibrary, except for havingPython3Compatibilityin lieu ofCompatibility. Code that depends onmy_tgt.has_fields([Compatibility])will continue to work with this custom target type.If a plugin needs the more precise
Field, that will work too.my_tgt.has_fields([Python3Compatibility])would returnTrueforPython3Library, but notPythonLibrary. This is a useful property—we can, for example, filter to get all targets withPythonSources, even if other targets have the fieldSources.