diff --git a/src/python/pants/engine/BUILD b/src/python/pants/engine/BUILD index 3d2bae81869..39b21d9933b 100644 --- a/src/python/pants/engine/BUILD +++ b/src/python/pants/engine/BUILD @@ -182,6 +182,7 @@ python_library( sources=['native.py'], dependencies=[ ':native_engine_shared_library', + ':selectors', '3rdparty/python:cffi', '3rdparty/python:setuptools', 'src/python/pants/binaries:binary_util', diff --git a/src/python/pants/engine/README.md b/src/python/pants/engine/README.md index 34d40a368ae..83e0158c563 100644 --- a/src/python/pants/engine/README.md +++ b/src/python/pants/engine/README.md @@ -1,145 +1,144 @@ -# The New Engine +# The (New) Engine -## Scheduling +## API -In the current RoundEngine, work is scheduled and then later performed via the `Task` interface. In -the new engine execution occurs via simple functions, with inputs selected via an input -selection clause made up of `Selector` objects (described later). +The end user API for the engine is based on the registration of `@rule`s, which are functions +or coroutines with statically declared inputs and outputs. A Pants (plugin) developer can write +and install additional `@rule`s to extend the functionality of Pants. -## History - -The need for an engine that could schedule all work as a result of linking required products to -their producers in multiple rounds was identified sometime in the middle of 2013 as a result of -new requirements on the `IdeaGen` task forced by the use of pants in the Twitter birdcage repo. The -design document for this "RoundEngine" is -[here](https://docs.google.com/document/d/1MwOFcr4W6KbzPdbaj_ntJ36a0NRoiKyWLed0ziobsr4/edit#heading=h.rsohbvtm7zng). -Some work was completed along these lines and an initial version of the `RoundEngine` was -integrated into the pants mainline and is used today. +The set of installed `@rule`s is statically checked as a closed world: this compilation step occurs +on startup, and identifies all unreachable or unsatisfiable rules before execution begins. This +allows most composition errors to be detected immediately, and also provides for easy introspection +of the build. To inspect the set of rules that are installed and which product types can be +computed, you can pass the `--native-engine-visualize-to=$dir` flag, which will write out a graph +of reachable `@rule`s. -Work stalled on the later phases of the `RoundEngine` and talks re-booted about the future of the -`RoundEngine`. Foursquare folks had been thinking about general problems with the `RoundEngine` as -it stood and proposed the idea of a "tuple-engine". With some license taken in representation, this -idea took the `RoundEngine` to the extreme of generating a round for each target-task pair. The -pair formed the tuple of schedulable work and this concept combined with others to form the design -[here][tuple-design]. +Once the engine is instantiated with a valid set of `@rule`s, a caller can synchronously request +computation of any of the product types provided by those `@rule`s by calling: -Meanwhile, need for fine-grained parallelism was acute to help speed up jvm compilation, especially -in the context of scala and mixed scala & java builds. Twitter spiked on a project to implement -a target-level scheduling system scoped to just the jvm compilation tasks. This bore fruit and -served as further impetus to get a "tuple-engine" designed and constructed to bring the benefits -seen in the jvm compilers to the wider pants world of tasks. - -### API - -#### End User API +```python +# Request a ThingINeed (a `Product`) for the thing_i_have (a `Subject`). +thing_i_need, = scheduler.product_request(ThingINeed, [thing_i_have]) +``` -The end user API for the engine is based on the registration of `Rules`, which are made up of: +The engine then takes care of concurrently executing all dependencies of the matched `@rule`s to +produce the requested value. -1. a `Product` or return type of a function, -2. a list of dependency `Selectors` which match inputs to the function, -3. the function itself. +### Products and Subjects -A `Rule` fully declares the inputs and outputs for its function: there is no imperative API for -requesting additional inputs during execution of a function. While a tight constraint, -this has the advantage of forcing decomposition of work into functions which are loosely -coupled by only the types of their inputs and outputs, and which are naturally isolated, cacheable, -and parallelizable. +The engine executes your `@rule`s in order to (recursively) compute a `Product` of the requested +type for a given `Subject`. This recursive type search leads to a very loosely coupled (and yet +still statically checked) form of dependency injection. -A function is guaranteed to execute only when all of its inputs are ready for use. The Scheduler -considers executing a Rule when it determines that it needs to produce the declared -output `Product` type of that function for a particular `Subject`. But the Scheduler will only -actually run a Rule if it is able to (recursively) find sources for each of the -function's inputs. +When an `@rule` runs, it runs for a particular `Subject` value, which is part of the unique +identity for that instance of the `@rule`. An `@rule` can request dependencies for different +`Subject` values as it runs (see the section on `Get` requests below). Because the subject for +an `@rule` is chosen by callers, a `Subject` can be of any (hashable) type that a user might want +to compute a product for. -See below for more information on `Products`, `Subjects`, and `Selectors`. +The return value of an `@rule` for a particular `Subject` is known as a `Product`. At some level, +you can think of (`subject_value`, `product_type`) as a "key" that uniquely identifies a particular +Product value and `@rule` execution. -#### Internal API +#### Example -Internally, the `Scheduler` uses end user `Rules` to create private `Node` objects and -build a `Graph` of futures that links them to their dependency Nodes. A Node represents a unique -computation and the data for a Node implicitly acts as its own key/identity. +As a very simple example, you might register the following `@rule` that can compute a `String` +Product given a single `Int` input. -To compute a value for a Node, the Scheduler uses the `Node.run` method starting from requested -roots. If a Node needs more inputs, it requests them via `Context.get`, which will declare a -dependency, and memoize the computation represented by the `Node`. - -The initial Nodes are [launched by the scheduler](https://github.com/pantsbuild/pants/blob/16d43a06ba3751e22fdc7f69f009faeb59a33930/src/rust/engine/src/scheduler.rs#L116-L126), -but the rest of the scheduling is driven by Nodes recursively calling `Context.get` to request -dependencies. - -### Products and Subjects - -A `Product` is a strongly typed value specific to a particular `Subject`. End user Rules execute -in order to (recursively) compute a Product for a Subject: as a very simple example, one might -register the following Rule that can compute a `String` Product given a single `Int` input -by calling the `str` function: +```python +@rule(StringType, [Select(IntType)]) +def int_to_str(an_int): + return '{}'.format(an_int) +``` - @rule(StringType, [Select(IntType)]) - def int_to_str(an_int): - return str(an_int) +The first argument to the `@rule` decorator is the Product (ie, return) type for the `@rule`. The +second argument is a list of `Selectors` that declare the types of the input arguments to the +`@rule`. In this case, because the Product type is `StringType` and there is one `Selector` +(`Select(IntType)`), this `@rule` represents a conversion from `IntType` to `StrType`, with no +other inputs. -When the Scheduler wants to decide whether it can use this Rule to create a string for a +When the engine statically checks whether it can use this `@rule` to create a string for a Subject, it will first see whether there are any ways to get an IntType for that Subject. If -the subject is already of `type(subject) == IntType`, then the Rule will be able to -execute immediately. On the other hand, if the type _doesn't_ match, the Scheduler doesn't give up: -it will next look for any other registered Rules that can compute an IntType Product for the +the subject is already of `type(subject) == IntType`, then the `@rule` will be satisfiable without +any other dependencies. On the other hand, if the type _doesn't_ match, the engine doesn't give up: +it will next look for any other registered `@rule`s that can compute an IntType Product for the Subject (and so on, recursively.) -This recursive type search leads to some very interesting (and, admittedly, somewhat "magical") -properties. If there is any path through the Rule graph that allows for conversion -from one type to another, it will be found and executed. +In practical use, using basic types like `StringType` or `IntType` does not provide enough +information to disambiguate between various types of data: So declaring small `datatype` +definitions to provide a unique and descriptive type is strongly recommended: -### Selectors +```python +class FormattedInt(datatype('FormattedInt', ['content'])): pass -As demonstrated above, the `Selector` classes select function inputs in the context of a particular -`Subject` (and its `Variants`: discussed below). For example, it might select a `Product` for the given -Subject (`Select`), or for other Subject(s) selected from fields of a Product (`SelectDependencies`, -`SelectProjection`). +@rule(FormattedInt, [Select(IntType)]) +def int_to_str(an_int): + return FormattedInt('{}'.format(an_int)) +``` -One very important thing to keep in mind is that Selectors like `SelectDependencies` and `SelectProjection` -"change" the Subject within a particular subgraph. For example, `SelectDependencies` -results in new subgraphs for each Subject in a list of values that was computed for some original Subject. -Concretely, a Rule could use SelectDependencies to select FileContent for each entry in a Files list, -and then concatentate that content into a string: +### Selectors and Gets - @rule(StringType, [SelectDependencies(FileContent, Files)]) - def concat(file_content_list): - return ''.join(fc.content for fc in file_content_list) +As demonstrated above, the `Selector` classes select `@rule` inputs in the context of a particular +`Subject` (and its `Variants`: discussed below). But it is frequently necessary to "change" the +subject and request products for subjects other than the one that the `@rule` is running for. -This Rule declares that: "for any Subject for which we can compute a 'Files' object, we can also -compute a StringType". Each subgraph will contain an attempt to get FileContent for a different -File Subject from the Files list. +In cases where this is necessary, `@rule`s may be written as coroutines (ie, using the python +`yield` statement) that yield "`Get` requests" that request products for other subjects. Just like +`@rule` parameter Selectors, `Get` requests instantiated in the body of an `@rule` are statically +checked to be satisfiable in the set of installed `@rule`s. -In practical use, using `StringType` or `IntType` directly would probably not provide enough information -to disambiguate between various types of data: So declaring small `datatype` definitions to provide -a unique and descriptive type is strongly recommended: +#### Example - class ConcattedFiles(datatype('ConcattedFiles', ['content'])): - pass +For example, you could declare an `@rule` that requests FileContent for each entry in a Files list, +and then concatentates that content into a (typed) string: + +```python +@rule(ConcattedFiles, [Select(Files)]) +def concat(files): + file_content_list = yield [Get(FileContent, File(f)) for f in files] + yield ConcattedFiles(''.join(fc.content for fc in file_content_list)) +``` + +This `@rule` declares that: "for any Subject for which we can compute `Files`, we can also compute +`ConcattedFiles`". Each yielded `Get` request results in FileContent for a different File Subject +from the Files list. ### Variants -Certain Rules will also need parameters provided by their dependents in order to tailor their output -Products to their consumers. For example, a javac planner might need to know -the version of the java platform for a given dependent binary target (say Java 6), or an ivy Rule -might need to identify a globally consistent ivy resolve for a test target. To allow for this the -engine introduces the concept of `variants`, which are passed recursively from dependents to -dependencies. +Certain `@rule`s will also need parameters provided by their dependents in order to tailor their output +Products to their consumers. For example, a javac `@rule` might need to know the version of the java +platform for a given dependent binary target (say Java 9), or an ivy `@rule` might need to identify a +globally consistent ivy resolve for a test target. To allow for this the engine introduces the +concept of `Variants`, which are passed recursively from dependents to dependencies. If a Rule uses a `SelectVariants` Selector to indicate that a variant is required, consumers can use a `@[type]=[name]` address syntax extension to pass a variant that matches a particular configuration -for a Rule. A dependency declared as `src/java/com/example/lib:lib` specifies no particular variant, but +for a `@rule`. A dependency declared as `src/java/com/example/lib:lib` specifies no particular variant, but `src/java/com/example/lib:lib@java=java8` asks for the configured variant of the lib named "java8". -Additionally, it is possible to specify the "default" variants for an Address by installing a Rule -function that can provide `Variants(default=..)`. Again, since the purpose of variants is to collect +Additionally, it is possible to specify the "default" variants for an Address by installing an `@rule` +function that can provide `Variants(default=..)`. Since the purpose of variants is to collect information from dependents, only default variant values which have not been set by a dependent will be used. +## Internal API + +Internally, the engine uses end user `@rule`s to create private `Node` objects and +build a `Graph` of futures that links them to their dependency Nodes. A Node represents a unique +computation and the data for a Node implicitly acts as its own key/identity. + +To compute a value for a Node, the engine uses the `Node.run` method starting from requested +roots. If a Node needs more inputs, it requests them via `Context.get`, which will declare a +dependency, and memoize the computation represented by the requested `Node`. + +The initial Nodes are [launched by the engine](https://github.com/pantsbuild/pants/blob/16d43a06ba3751e22fdc7f69f009faeb59a33930/src/rust/engine/src/scheduler.rs#L116-L126), +but the rest of execution is driven by Nodes recursively calling `Context.get` to request their +dependencies. + ## Execution -The Scheduler executes work concurrently wherever possible; to help visualize executions, a visualization -tool is provided that, after executing a `ProductGraph`, generates a `dot` file that can be rendered using +The engine executes work concurrently wherever possible; to help visualize executions, a visualization +tool is provided that, after executing a `Graph`, generates a `dot` file that can be rendered using Graphviz: ```console @@ -159,3 +158,26 @@ class. This hash is maintained by `build-support/bin/native/bootstrap.sh` and output to the `native_engine_version` file in this directory. Any modification to this resource file's location will need adjustments in `build-support/bin/native/bootstrap.sh` to ensure the linking continues to work. + +## History + +The need for an engine that could schedule all work as a result of linking required products to +their producers in multiple rounds was identified sometime in the middle of 2013 as a result of +new requirements on the `IdeaGen` task forced by the use of pants in the Twitter birdcage repo. The +design document for this "RoundEngine" is +[here](https://docs.google.com/document/d/1MwOFcr4W6KbzPdbaj_ntJ36a0NRoiKyWLed0ziobsr4/edit#heading=h.rsohbvtm7zng). +Some work was completed along these lines and an initial version of the `RoundEngine` was +integrated into the pants mainline and is used today. + +Work stalled on the later phases of the `RoundEngine` and talks re-booted about the future of the +`RoundEngine`. Foursquare folks had been thinking about general problems with the `RoundEngine` as +it stood and proposed the idea of a "tuple-engine". With some license taken in representation, this +idea took the `RoundEngine` to the extreme of generating a round for each target-task pair. The +pair formed the tuple of schedulable work and this concept combined with others to form the design +[here][tuple-design]. + +Meanwhile, need for fine-grained parallelism was acute to help speed up jvm compilation, especially +in the context of scala and mixed scala & java builds. Twitter spiked on a project to implement +a target-level scheduling system scoped to just the jvm compilation tasks. This bore fruit and +served as further impetus to get a "tuple-engine" designed and constructed to bring the benefits +seen in the jvm compilers to the wider pants world of tasks. diff --git a/src/python/pants/engine/build_files.py b/src/python/pants/engine/build_files.py index a187c0541fd..6a4669bfe0c 100644 --- a/src/python/pants/engine/build_files.py +++ b/src/python/pants/engine/build_files.py @@ -20,10 +20,10 @@ from pants.engine.mapper import AddressFamily, AddressMap, AddressMapper, ResolveError from pants.engine.objects import Locatable, SerializableFactory, Validatable from pants.engine.rules import RootRule, SingletonRule, TaskRule, rule -from pants.engine.selectors import Select, SelectDependencies, SelectProjection +from pants.engine.selectors import Get, Select, SelectDependencies from pants.engine.struct import Struct from pants.util.dirutil import fast_relpath_optional -from pants.util.objects import Collection, datatype +from pants.util.objects import datatype class ResolvedTypeMismatchError(ResolveError): @@ -35,47 +35,26 @@ def _key_func(entry): return key -class BuildDirs(datatype('BuildDirs', ['dependencies'])): - """A list of Stat objects for directories containing build files.""" - - -class BuildFiles(datatype('BuildFiles', ['files_content'])): - """The FileContents of BUILD files in some directory""" - - -class BuildFileGlobs(datatype('BuildFilesGlobs', ['path_globs'])): - """A wrapper around PathGlobs that are known to match a build file pattern.""" - - -@rule(BuildFiles, - [SelectProjection(FilesContent, PathGlobs, 'path_globs', BuildFileGlobs)]) -def build_files(files_content): - return BuildFiles(files_content) - - -@rule(BuildFileGlobs, [Select(AddressMapper), Select(Dir)]) -def buildfile_path_globs_for_dir(address_mapper, directory): - patterns = tuple(join(directory.path, p) for p in address_mapper.build_patterns) - return BuildFileGlobs(PathGlobs.create('', - include=patterns, - exclude=address_mapper.build_ignore_patterns)) - - -@rule(AddressFamily, [Select(AddressMapper), Select(Dir), Select(BuildFiles)]) -def parse_address_family(address_mapper, path, build_files): - """Given the contents of the build files in one directory, return an AddressFamily. +@rule(AddressFamily, [Select(AddressMapper), Select(Dir)]) +def parse_address_family(address_mapper, directory): + """Given an AddressMapper and a directory, return an AddressFamily. The AddressFamily may be empty, but it will not be None. """ - files_content = build_files.files_content.dependencies + patterns = tuple(join(directory.path, p) for p in address_mapper.build_patterns) + path_globs = PathGlobs.create('', + include=patterns, + exclude=address_mapper.build_ignore_patterns) + files_content = yield Get(FilesContent, PathGlobs, path_globs) + if not files_content: - raise ResolveError('Directory "{}" does not contain build files.'.format(path)) + raise ResolveError('Directory "{}" does not contain build files.'.format(directory.path)) address_maps = [] - for filecontent_product in files_content: + for filecontent_product in files_content.dependencies: address_maps.append(AddressMap.parse(filecontent_product.path, filecontent_product.content, address_mapper.parser)) - return AddressFamily.create(path.path, address_maps) + yield AddressFamily.create(directory.path, address_maps) class UnhydratedStruct(datatype('UnhydratedStruct', ['address', 'struct', 'dependencies'])): @@ -107,17 +86,16 @@ def _raise_did_you_mean(address_family, name): .format(name, address_family.namespace, possibilities)) -@rule(UnhydratedStruct, - [Select(AddressMapper), - SelectProjection(AddressFamily, Dir, 'spec_path', Address), - Select(Address)]) -def resolve_unhydrated_struct(address_mapper, address_family, address): - """Given an Address and its AddressFamily, resolve an UnhydratedStruct. +@rule(UnhydratedStruct, [Select(AddressMapper), Select(Address)]) +def resolve_unhydrated_struct(address_mapper, address): + """Given an AddressMapper and an Address, resolve an UnhydratedStruct. Recursively collects any embedded addressables within the Struct, but will not walk into a - dependencies field, since those are requested explicitly by tasks using SelectDependencies. + dependencies field, since those should be requested explicitly by rules. """ + address_family = yield Get(AddressFamily, Dir(address.spec_path)) + struct = address_family.addressables.get(address) addresses = address_family.addressables if not struct or address not in addresses: @@ -148,7 +126,7 @@ def collect_dependencies(item): collect_dependencies(struct) - return UnhydratedStruct( + yield UnhydratedStruct( filter(lambda build_address: build_address == address, addresses)[0], struct, dependencies) @@ -222,17 +200,18 @@ def _hydrate(item_type, spec_path, **kwargs): return item -@rule(BuildFileAddresses, - [Select(AddressMapper), - SelectDependencies(AddressFamily, BuildDirs, field_types=(Dir,)), - Select(Specs)]) -def addresses_from_address_families(address_mapper, address_families, specs): - """Given a list of AddressFamilies matching a list of Specs, return matching Addresses. +@rule(BuildFileAddresses, [Select(AddressMapper), Select(Specs)]) +def addresses_from_address_families(address_mapper, specs): + """Given an AddressMapper and list of Specs, return matching BuildFileAddresses. Raises a AddressLookupError if: - there were no matching AddressFamilies, or - the Spec matches no addresses for SingleAddresses. """ + # Capture a Snapshot covering all paths for these Specs, then group by directory. + snapshot = yield Get(Snapshot, PathGlobs, _spec_to_globs(address_mapper, specs)) + dirnames = set(dirname(f.stat.path) for f in snapshot.files) + address_families = yield [Get(AddressFamily, Dir(d)) for d in dirnames] # NB: `@memoized` does not work on local functions. def by_directory(): @@ -294,20 +273,11 @@ def include(address_families, predicate=None): else: raise ValueError('Unrecognized Spec type: {}'.format(spec)) - return BuildFileAddresses(addresses) + yield BuildFileAddresses(addresses) -@rule(BuildDirs, [Select(AddressMapper), Select(Snapshot)]) -def filter_build_dirs(address_mapper, snapshot): - """Given a Snapshot matching a build pattern, return parent directories as BuildDirs.""" - dirnames = set(dirname(f.stat.path) for f in snapshot.files) - return BuildDirs(tuple(Dir(d) for d in dirnames)) - - -@rule(PathGlobs, [Select(AddressMapper), Select(Specs)]) -def spec_to_globs(address_mapper, specs): - """Given a Spec object, return a PathGlobs object for the build files that it matches. - """ +def _spec_to_globs(address_mapper, specs): + """Given a Specs object, return a PathGlobs object for the build files that it matches.""" patterns = set() for spec in specs.dependencies: if type(spec) is DescendantAddresses: @@ -340,9 +310,6 @@ def _recursive_dirname(f): yield '' -BuildFilesCollection = Collection.of(BuildFiles) - - def create_graph_rules(address_mapper, symbol_table): """Creates tasks used to parse Structs from BUILD files. @@ -351,9 +318,6 @@ def create_graph_rules(address_mapper, symbol_table): """ symbol_table_constraint = symbol_table.constraint() return [ - TaskRule(BuildFilesCollection, - [SelectDependencies(BuildFiles, BuildDirs, field_types=(Dir,))], - BuildFilesCollection), # A singleton to provide the AddressMapper. SingletonRule(AddressMapper, address_mapper), # Support for resolving Structs from Addresses. @@ -367,13 +331,9 @@ def create_graph_rules(address_mapper, symbol_table): resolve_unhydrated_struct, # BUILD file parsing. parse_address_family, - build_files, - buildfile_path_globs_for_dir, # Spec handling: locate directories that contain build files, and request # AddressFamilies for each of them. addresses_from_address_families, - filter_build_dirs, - spec_to_globs, # Root rules representing parameters that might be provided via root subjects. RootRule(Address), RootRule(BuildFileAddress), diff --git a/src/python/pants/engine/legacy/address_mapper.py b/src/python/pants/engine/legacy/address_mapper.py index b5016f4a632..360509d09f3 100644 --- a/src/python/pants/engine/legacy/address_mapper.py +++ b/src/python/pants/engine/legacy/address_mapper.py @@ -13,7 +13,6 @@ from pants.build_graph.address_lookup_error import AddressLookupError from pants.build_graph.address_mapper import AddressMapper from pants.engine.addressable import BuildFileAddresses -from pants.engine.build_files import BuildFilesCollection from pants.engine.mapper import ResolveError from pants.engine.nodes import Throw from pants.util.dirutil import fast_relpath @@ -33,14 +32,10 @@ def __init__(self, scheduler, build_root): self._build_root = build_root def scan_build_files(self, base_path): - specs = (DescendantAddresses(base_path),) - build_files_collection, = self._scheduler.product_request(BuildFilesCollection, [Specs(specs)]) + build_file_addresses = self._internal_scan_specs([DescendantAddresses(base_path)], + missing_is_fatal=False) - build_files_set = set() - for build_files in build_files_collection.dependencies: - build_files_set.update(f.path for f in build_files.files_content.dependencies) - - return build_files_set + return {bfa.rel_path for bfa in build_file_addresses} @staticmethod def any_is_declaring_file(address, file_paths): diff --git a/src/python/pants/engine/legacy/graph.py b/src/python/pants/engine/legacy/graph.py index e25d409fff9..e8eccbd84fc 100644 --- a/src/python/pants/engine/legacy/graph.py +++ b/src/python/pants/engine/legacy/graph.py @@ -23,7 +23,7 @@ from pants.engine.fs import PathGlobs, Snapshot from pants.engine.legacy.structs import BundleAdaptor, BundlesField, SourcesField, TargetAdaptor from pants.engine.rules import TaskRule, rule -from pants.engine.selectors import Select, SelectDependencies, SelectProjection +from pants.engine.selectors import Get, Select, SelectDependencies from pants.source.wrapped_globs import EagerFilesetWithSpec, FilesetRelPathWrapper from pants.util.dirutil import fast_relpath from pants.util.objects import Collection, datatype @@ -299,18 +299,19 @@ class HydratedTargets(Collection.of(HydratedTarget)): """An intransitive set of HydratedTarget objects.""" -@rule(TransitiveHydratedTargets, [SelectDependencies(TransitiveHydratedTarget, - BuildFileAddresses, - field_types=(Address,), - field='addresses')]) -def transitive_hydrated_targets(transitive_hydrated_targets): - """Kicks off recursion on expansion of TransitiveHydratedTarget objects. +@rule(TransitiveHydratedTargets, [Select(BuildFileAddresses)]) +def transitive_hydrated_targets(build_file_addresses): + """Given BuildFileAddresses, kicks off recursion on expansion of TransitiveHydratedTargets. The TransitiveHydratedTarget struct represents a structure-shared graph, which we walk and flatten here. The engine memoizes the computation of TransitiveHydratedTarget, so when multiple TransitiveHydratedTargets objects are being constructed for multiple roots, their structure will be shared. """ + + transitive_hydrated_targets = yield [Get(TransitiveHydratedTarget, Address, a) + for a in build_file_addresses.addresses] + closure = set() to_visit = deque(transitive_hydrated_targets) @@ -321,25 +322,20 @@ def transitive_hydrated_targets(transitive_hydrated_targets): closure.add(tht.root) to_visit.extend(tht.dependencies) - return TransitiveHydratedTargets(tuple(tht.root for tht in transitive_hydrated_targets), closure) + yield TransitiveHydratedTargets(tuple(tht.root for tht in transitive_hydrated_targets), closure) -@rule(TransitiveHydratedTarget, [Select(HydratedTarget), - SelectDependencies(TransitiveHydratedTarget, - HydratedTarget, - field_types=(Address,), - field='addresses')]) -def transitive_hydrated_target(root, dependencies): - return TransitiveHydratedTarget(root, dependencies) +@rule(TransitiveHydratedTarget, [Select(HydratedTarget)]) +def transitive_hydrated_target(root): + dependencies = yield [Get(TransitiveHydratedTarget, Address, d) for d in root.dependencies] + yield TransitiveHydratedTarget(root, dependencies) -@rule(HydratedTargets, [SelectDependencies(HydratedTarget, - BuildFileAddresses, - field_types=(Address,), - field='addresses')]) -def hydrated_targets(targets): - """Requests HydratedTarget instances.""" - return HydratedTargets(targets) +@rule(HydratedTargets, [Select(BuildFileAddresses)]) +def hydrated_targets(build_file_addresses): + """Requests HydratedTarget instances for BuildFileAddresses.""" + targets = yield [Get(HydratedTarget, Address, a) for a in build_file_addresses.addresses] + yield HydratedTargets(targets) class HydratedField(datatype('HydratedField', ['name', 'value'])): @@ -372,22 +368,22 @@ def _eager_fileset_with_spec(spec_path, filespec, snapshot, include_dirs=False): files_hash=snapshot.fingerprint) -@rule(HydratedField, - [Select(SourcesField), - SelectProjection(Snapshot, PathGlobs, 'path_globs', SourcesField)]) -def hydrate_sources(sources_field, snapshot): - """Given a SourcesField and a Snapshot for its path_globs, create an EagerFilesetWithSpec.""" +@rule(HydratedField, [Select(SourcesField)]) +def hydrate_sources(sources_field): + """Given a SourcesField, request a Snapshot for its path_globs and create an EagerFilesetWithSpec.""" + + snapshot = yield Get(Snapshot, PathGlobs, sources_field.path_globs) fileset_with_spec = _eager_fileset_with_spec(sources_field.address.spec_path, sources_field.filespecs, snapshot) - return HydratedField(sources_field.arg, fileset_with_spec) + yield HydratedField(sources_field.arg, fileset_with_spec) + +@rule(HydratedField, [Select(BundlesField)]) +def hydrate_bundles(bundles_field): + """Given a BundlesField, request Snapshots for each of its filesets and create BundleAdaptors.""" + snapshot_list = yield [Get(Snapshot, PathGlobs, pg) for pg in bundles_field.path_globs_list] -@rule(HydratedField, - [Select(BundlesField), - SelectDependencies(Snapshot, BundlesField, 'path_globs_list', field_types=(PathGlobs,))]) -def hydrate_bundles(bundles_field, snapshot_list): - """Given a BundlesField and a Snapshot for each of its filesets create a list of BundleAdaptors.""" bundles = [] zipped = zip(bundles_field.bundles, bundles_field.filespecs_list, @@ -403,7 +399,7 @@ def hydrate_bundles(bundles_field, snapshot_list): snapshot, include_dirs=True) bundles.append(BundleAdaptor(**kwargs)) - return HydratedField('bundles', bundles) + yield HydratedField('bundles', bundles) def create_legacy_graph_tasks(symbol_table): diff --git a/src/python/pants/engine/native.py b/src/python/pants/engine/native.py index b0167c27a0e..781f09fdc98 100644 --- a/src/python/pants/engine/native.py +++ b/src/python/pants/engine/native.py @@ -17,6 +17,7 @@ import pkg_resources import six +from pants.engine.selectors import Get, constraint_for from pants.util.contextutil import temporary_dir from pants.util.dirutil import safe_mkdir, safe_mkdtemp from pants.util.memo import memoized_property @@ -83,6 +84,12 @@ Value value; } PyResult; +typedef struct { + uint8_t tag; + ValueBuffer values; + ValueBuffer constraints; +} PyGeneratorResponse; + typedef struct { int64_t hash_; Value value; @@ -92,25 +99,26 @@ typedef void ExternContext; // On the rust side the integration is defined in externs.rs -typedef void (*extern_ptr_log)(ExternContext*, uint8_t, uint8_t*, uint64_t); -typedef uint8_t extern_log_level; -typedef Ident (*extern_ptr_identify)(ExternContext*, Value*); -typedef _Bool (*extern_ptr_equals)(ExternContext*, Value*, Value*); -typedef Value (*extern_ptr_clone_val)(ExternContext*, Value*); -typedef void (*extern_ptr_drop_handles)(ExternContext*, Handle*, uint64_t); -typedef Buffer (*extern_ptr_type_to_str)(ExternContext*, TypeId); -typedef Buffer (*extern_ptr_val_to_str)(ExternContext*, Value*); -typedef _Bool (*extern_ptr_satisfied_by)(ExternContext*, Value*, Value*); -typedef _Bool (*extern_ptr_satisfied_by_type)(ExternContext*, Value*, TypeId*); -typedef Value (*extern_ptr_store_list)(ExternContext*, Value**, uint64_t, _Bool); -typedef Value (*extern_ptr_store_bytes)(ExternContext*, uint8_t*, uint64_t); -typedef Value (*extern_ptr_store_i32)(ExternContext*, int32_t); -typedef Value (*extern_ptr_project)(ExternContext*, Value*, uint8_t*, uint64_t, TypeId*); -typedef ValueBuffer (*extern_ptr_project_multi)(ExternContext*, Value*, uint8_t*, uint64_t); -typedef Value (*extern_ptr_project_ignoring_type)(ExternContext*, Value*, uint8_t*, uint64_t); -typedef Value (*extern_ptr_create_exception)(ExternContext*, uint8_t*, uint64_t); -typedef PyResult (*extern_ptr_call)(ExternContext*, Value*, Value*, uint64_t); -typedef PyResult (*extern_ptr_eval)(ExternContext*, uint8_t*, uint64_t); +typedef void (*extern_ptr_log)(ExternContext*, uint8_t, uint8_t*, uint64_t); +typedef uint8_t extern_log_level; +typedef Ident (*extern_ptr_identify)(ExternContext*, Value*); +typedef _Bool (*extern_ptr_equals)(ExternContext*, Value*, Value*); +typedef Value (*extern_ptr_clone_val)(ExternContext*, Value*); +typedef void (*extern_ptr_drop_handles)(ExternContext*, Handle*, uint64_t); +typedef Buffer (*extern_ptr_type_to_str)(ExternContext*, TypeId); +typedef Buffer (*extern_ptr_val_to_str)(ExternContext*, Value*); +typedef _Bool (*extern_ptr_satisfied_by)(ExternContext*, Value*, Value*); +typedef _Bool (*extern_ptr_satisfied_by_type)(ExternContext*, Value*, TypeId*); +typedef Value (*extern_ptr_store_list)(ExternContext*, Value**, uint64_t, _Bool); +typedef Value (*extern_ptr_store_bytes)(ExternContext*, uint8_t*, uint64_t); +typedef Value (*extern_ptr_store_i32)(ExternContext*, int32_t); +typedef Value (*extern_ptr_project)(ExternContext*, Value*, uint8_t*, uint64_t, TypeId*); +typedef ValueBuffer (*extern_ptr_project_multi)(ExternContext*, Value*, uint8_t*, uint64_t); +typedef Value (*extern_ptr_project_ignoring_type)(ExternContext*, Value*, uint8_t*, uint64_t); +typedef Value (*extern_ptr_create_exception)(ExternContext*, uint8_t*, uint64_t); +typedef PyResult (*extern_ptr_call)(ExternContext*, Value*, Value*, uint64_t); +typedef PyGeneratorResponse (*extern_ptr_generator_send)(ExternContext*, Value*, Value*); +typedef PyResult (*extern_ptr_eval)(ExternContext*, uint8_t*, uint64_t); typedef void Tasks; typedef void Scheduler; @@ -136,6 +144,7 @@ extern_ptr_log, extern_log_level, extern_ptr_call, + extern_ptr_generator_send, extern_ptr_eval, extern_ptr_identify, extern_ptr_equals, @@ -159,6 +168,7 @@ Tasks* tasks_create(void); void tasks_task_begin(Tasks*, Function, TypeConstraint); +void tasks_add_get(Tasks*, TypeConstraint, TypeId); void tasks_add_select(Tasks*, TypeConstraint); void tasks_add_select_variant(Tasks*, TypeConstraint, Buffer); void tasks_add_select_dependencies(Tasks*, TypeConstraint, TypeConstraint, Buffer, TypeIdBuffer); @@ -189,6 +199,7 @@ TypeConstraint, TypeConstraint, TypeConstraint, + TypeConstraint, TypeId, TypeId, Buffer, @@ -226,24 +237,25 @@ CFFI_EXTERNS = ''' extern "Python" { - void extern_log(ExternContext*, uint8_t, uint8_t*, uint64_t); - PyResult extern_call(ExternContext*, Value*, Value*, uint64_t); - PyResult extern_eval(ExternContext*, uint8_t*, uint64_t); - Ident extern_identify(ExternContext*, Value*); - _Bool extern_equals(ExternContext*, Value*, Value*); - Value extern_clone_val(ExternContext*, Value*); - void extern_drop_handles(ExternContext*, Handle*, uint64_t); - Buffer extern_type_to_str(ExternContext*, TypeId); - Buffer extern_val_to_str(ExternContext*, Value*); - _Bool extern_satisfied_by(ExternContext*, Value*, Value*); - _Bool extern_satisfied_by_type(ExternContext*, Value*, TypeId*); - Value extern_store_list(ExternContext*, Value**, uint64_t, _Bool); - Value extern_store_bytes(ExternContext*, uint8_t*, uint64_t); - Value extern_store_i32(ExternContext*, int32_t); - Value extern_project(ExternContext*, Value*, uint8_t*, uint64_t, TypeId*); - Value extern_project_ignoring_type(ExternContext*, Value*, uint8_t*, uint64_t); - ValueBuffer extern_project_multi(ExternContext*, Value*, uint8_t*, uint64_t); - Value extern_create_exception(ExternContext*, uint8_t*, uint64_t); + void extern_log(ExternContext*, uint8_t, uint8_t*, uint64_t); + PyResult extern_call(ExternContext*, Value*, Value*, uint64_t); + PyGeneratorResponse extern_generator_send(ExternContext*, Value*, Value*); + PyResult extern_eval(ExternContext*, uint8_t*, uint64_t); + Ident extern_identify(ExternContext*, Value*); + _Bool extern_equals(ExternContext*, Value*, Value*); + Value extern_clone_val(ExternContext*, Value*); + void extern_drop_handles(ExternContext*, Handle*, uint64_t); + Buffer extern_type_to_str(ExternContext*, TypeId); + Buffer extern_val_to_str(ExternContext*, Value*); + _Bool extern_satisfied_by(ExternContext*, Value*, Value*); + _Bool extern_satisfied_by_type(ExternContext*, Value*, TypeId*); + Value extern_store_list(ExternContext*, Value**, uint64_t, _Bool); + Value extern_store_bytes(ExternContext*, uint8_t*, uint64_t); + Value extern_store_i32(ExternContext*, int32_t); + Value extern_project(ExternContext*, Value*, uint8_t*, uint64_t, TypeId*); + Value extern_project_ignoring_type(ExternContext*, Value*, uint8_t*, uint64_t); + ValueBuffer extern_project_multi(ExternContext*, Value*, uint8_t*, uint64_t); + Value extern_create_exception(ExternContext*, uint8_t*, uint64_t); } ''' @@ -459,6 +471,41 @@ def extern_create_exception(context_handle, msg_ptr, msg_len): msg = to_py_str(msg_ptr, msg_len) return c.to_value(Exception(msg)) + @ffi.def_extern() + def extern_generator_send(context_handle, func, arg): + """Given a generator, send it the given value and return a response.""" + c = ffi.from_handle(context_handle) + try: + res = c.from_value(func).send(c.from_value(arg)) + if isinstance(res, Get): + # Get. + values = [res.subject] + constraints = [constraint_for(res.product)] + tag = 2 + elif type(res) in (tuple, list): + # GetMulti. + values = [g.subject for g in res] + constraints = [constraint_for(g.product) for g in res] + tag = 3 + else: + # Break. + values = [res] + constraints = [] + tag = 0 + except Exception as e: + # Throw. + val = e + val._formatted_exc = traceback.format_exc() + values = [val] + constraints = [] + tag = 1 + + return ( + tag, + c.vals_buf([c.to_value(v) for v in values]), + c.vals_buf([c.to_value(v) for v in constraints]) + ) + @ffi.def_extern() def extern_call(context_handle, func, args_ptr, args_len): """Given a callable, call it.""" @@ -637,6 +684,7 @@ def init_externs(): self.ffi_lib.extern_log, logger.getEffectiveLevel(), self.ffi_lib.extern_call, + self.ffi_lib.extern_generator_send, self.ffi_lib.extern_eval, self.ffi_lib.extern_identify, self.ffi_lib.extern_equals, @@ -710,7 +758,8 @@ def new_scheduler(self, constraint_file, constraint_link, constraint_process_request, - constraint_process_result): + constraint_process_result, + constraint_generator): """Create and return an ExternContext and native Scheduler.""" def func(constraint): @@ -743,6 +792,7 @@ def tc(constraint): tc(constraint_link), tc(constraint_process_request), tc(constraint_process_result), + tc(constraint_generator), # Types. TypeId(self.context.to_id(six.text_type)), TypeId(self.context.to_id(six.binary_type)), diff --git a/src/python/pants/engine/rules.py b/src/python/pants/engine/rules.py index aa0cc4d3a12..fecd05f3cbe 100644 --- a/src/python/pants/engine/rules.py +++ b/src/python/pants/engine/rules.py @@ -5,14 +5,17 @@ from __future__ import (absolute_import, division, generators, nested_scopes, print_function, unicode_literals, with_statement) +import ast +import inspect import logging from abc import abstractproperty from collections import OrderedDict +from types import TypeType from twitter.common.collections import OrderedSet from pants.engine.addressable import Exactly -from pants.engine.selectors import type_or_constraint_repr +from pants.engine.selectors import Get, type_or_constraint_repr from pants.util.meta import AbstractClass from pants.util.objects import datatype @@ -20,6 +23,17 @@ logger = logging.getLogger(__name__) +class _RuleVisitor(ast.NodeVisitor): + def __init__(self): + super(_RuleVisitor, self).__init__() + self.gets = [] + + def visit_Call(self, node): + if not isinstance(node.func, ast.Name) or node.func.id != Get.__name__: + return + self.gets.append(Get.extract_constraints(node)) + + def rule(output_type, input_selectors): """A @decorator that declares that a particular static function may be used as a TaskRule. @@ -28,8 +42,29 @@ def rule(output_type, input_selectors): :param list input_selectors: A list of Selector instances that matches the number of arguments to the @decorated function. """ + def wrapper(func): - func._rule = TaskRule(output_type, input_selectors, func) + if not inspect.isfunction(func): + raise ValueError('The @rule decorator must be applied innermost of all decorators.') + + caller_frame = inspect.stack()[1][0] + module_ast = ast.parse(inspect.getsource(func)) + + def resolve_type(name): + resolved = caller_frame.f_globals.get(name) or caller_frame.f_builtins.get(name) + if not isinstance(resolved, (TypeType, Exactly)): + raise ValueError('Expected either a `type` constructor or TypeConstraint instance; ' + 'got: {}'.format(name)) + return resolved + + gets = [] + for node in ast.iter_child_nodes(module_ast): + if isinstance(node, ast.FunctionDef) and node.name == func.__name__: + rule_visitor = _RuleVisitor() + rule_visitor.visit(node) + gets.extend(Get(resolve_type(p), resolve_type(s)) for p, s in rule_visitor.gets) + + func._rule = TaskRule(output_type, input_selectors, func, input_gets=gets) return func return wrapper @@ -50,10 +85,13 @@ def input_selectors(self): """Collection of input selectors.""" -class TaskRule(datatype('TaskRule', ['output_constraint', 'input_selectors', 'func']), Rule): - """A Rule that runs a task function when all of its input selectors are satisfied.""" +class TaskRule(datatype('TaskRule', ['output_constraint', 'input_selectors', 'input_gets', 'func']), Rule): + """A Rule that runs a task function when all of its input selectors are satisfied. + + TODO: Make input_gets non-optional when more/all rules are using them. + """ - def __new__(cls, output_type, input_selectors, func): + def __new__(cls, output_type, input_selectors, func, input_gets=None): # Validate result type. if isinstance(output_type, Exactly): constraint = output_type @@ -68,8 +106,14 @@ def __new__(cls, output_type, input_selectors, func): raise TypeError("Expected a list of Selectors for rule `{}`, got: {}".format( func.__name__, type(input_selectors))) + # Validate gets. + input_gets = [] if input_gets is None else input_gets + if not isinstance(input_gets, list): + raise TypeError("Expected a list of Gets for rule `{}`, got: {}".format( + func.__name__, type(input_gets))) + # Create. - return super(TaskRule, cls).__new__(cls, constraint, tuple(input_selectors), func) + return super(TaskRule, cls).__new__(cls, constraint, tuple(input_selectors), tuple(input_gets), func) def __str__(self): return '({}, {!r}, {})'.format(type_or_constraint_repr(self.output_constraint), diff --git a/src/python/pants/engine/scheduler.py b/src/python/pants/engine/scheduler.py index 3e98ea5fb1f..dcf9a0f4903 100644 --- a/src/python/pants/engine/scheduler.py +++ b/src/python/pants/engine/scheduler.py @@ -9,6 +9,7 @@ import os import time from collections import defaultdict +from types import GeneratorType from pants.base.exceptions import TaskError from pants.base.project_tree import Dir, File, Link @@ -111,6 +112,7 @@ def __init__(self, native, build_root, work_dir, ignore_patterns, rule_index): constraint_for(Link), constraint_for(ExecuteProcessRequest), constraint_for(ExecuteProcessResult), + constraint_for(GeneratorType), ) def _root_type_ids(self): @@ -189,10 +191,9 @@ def _register_singleton(self, output_constraint, rule): def _register_task(self, output_constraint, rule): """Register the given TaskRule with the native scheduler.""" - input_selects = rule.input_selectors func = rule.func self._native.lib.tasks_task_begin(self._tasks, Function(self._to_key(func)), output_constraint) - for selector in input_selects: + for selector in rule.input_selectors: selector_type = type(selector) product_constraint = self._to_constraint(selector.product) if selector_type is Select: @@ -216,6 +217,10 @@ def _register_task(self, output_constraint, rule): self._to_constraint(selector.input_product)) else: raise ValueError('Unrecognized Selector type: {}'.format(selector)) + for get in rule.input_gets: + self._native.lib.tasks_add_get(self._tasks, + self._to_constraint(get.product), + TypeId(self._to_id(get.subject))) self._native.lib.tasks_task_end(self._tasks) def visualize_graph_to_file(self, execution_request, filename): diff --git a/src/python/pants/engine/selectors.py b/src/python/pants/engine/selectors.py index 6f6cda4203c..eb26582de14 100644 --- a/src/python/pants/engine/selectors.py +++ b/src/python/pants/engine/selectors.py @@ -5,6 +5,7 @@ from __future__ import (absolute_import, division, generators, nested_scopes, print_function, unicode_literals, with_statement) +import ast from abc import abstractproperty import six @@ -32,11 +33,59 @@ def constraint_for(type_or_constraint): raise TypeError("Expected a type or constraint: got: {}".format(type_or_constraint)) +class Get(datatype('Get', ['product', 'subject'])): + """Experimental synchronous generator API. + + May be called equivalently as either: + # verbose form: Get(product_type, subject_type, subject) + # shorthand form: Get(product_type, subject_type(subject)) + """ + + @staticmethod + def extract_constraints(call_node): + """Parses a `Get(..)` call in one of its two legal forms to return its type constraints. + + :param call_node: An `ast.Call` node representing a call to `Get(..)`. + :return: A tuple of product type id and subject type id. + """ + def render_args(): + return ', '.join(a.id for a in call_node.args) + + if len(call_node.args) == 2: + product_type, subject_constructor = call_node.args + if not isinstance(product_type, ast.Name) or not isinstance(subject_constructor, ast.Call): + raise ValueError('Two arg form of {} expected (product_type, subject_type(subject)), but ' + 'got: ({})'.format(Get.__name__, render_args())) + return (product_type.id, subject_constructor.func.id) + elif len(call_node.args) == 3: + product_type, subject_type, _ = call_node.args + if not isinstance(product_type, ast.Name) or not isinstance(subject_type, ast.Name): + raise ValueError('Three arg form of {} expected (product_type, subject_type, subject), but ' + 'got: ({})'.format(Get.__name__, render_args())) + return (product_type.id, subject_type.id) + else: + raise ValueError('Invalid {}; expected either two or three args, but ' + 'got: ({})'.format(Get.__name__, render_args())) + + def __new__(cls, *args): + if len(args) == 2: + product, subject = args + elif len(args) == 3: + product, subject_type, subject = args + if type(subject) is not subject_type: + raise TypeError('Declared type did not match actual type for {}({}).'.format( + Get.__name__, ', '.join(str(a) for a in args))) + else: + raise Exception('Expected either two or three arguments to {}; got {}.'.format( + Get.__name__, args)) + return super(Get, cls).__new__(cls, product, subject) + + class Selector(AbstractClass): - # The type constraint for the product type for this selector. @property def type_constraint(self): + """The type constraint for the product type for this selector.""" return constraint_for(self.product) @abstractproperty diff --git a/src/rust/engine/fs/src/lib.rs b/src/rust/engine/fs/src/lib.rs index 8e6564859b4..3560ebbe618 100644 --- a/src/rust/engine/fs/src/lib.rs +++ b/src/rust/engine/fs/src/lib.rs @@ -697,19 +697,14 @@ pub trait VFS: Clone + Send + Sync + 'static { // If there were any new PathGlobs, continue the expansion. if expansion.todo.is_empty() { - future::Loop::Break(expansion) + future::Loop::Break(expansion.outputs) } else { future::Loop::Continue(expansion) } }) - }).map(|expansion| { - assert!( - expansion.todo.is_empty(), - "Loop shouldn't have exited with work to do: {:?}", - expansion.todo, - ); + }).map(|expansion_outputs| { // Finally, capture the resulting PathStats from the expansion. - expansion.outputs.into_iter().map(|(k, _)| k).collect() + expansion_outputs.into_iter().map(|(k, _)| k).collect() }) .to_boxed() } diff --git a/src/rust/engine/src/externs.rs b/src/rust/engine/src/externs.rs index 9e06ef4682b..f0297027243 100644 --- a/src/rust/engine/src/externs.rs +++ b/src/rust/engine/src/externs.rs @@ -146,6 +146,33 @@ pub fn call(func: &Value, args: &[Value]) -> Result { with_externs(|e| (e.call)(e.context, func, args.as_ptr(), args.len() as u64)).into() } +pub fn generator_send(generator: &Value, arg: &Value) -> Result { + let response = with_externs(|e| (e.generator_send)(e.context, generator, arg)); + match response.res_type { + PyGeneratorResponseType::Break => Ok(GeneratorResponse::Break(response.values.unwrap_one())), + PyGeneratorResponseType::Throw => Err(PyResult::failure_from(response.values.unwrap_one())), + PyGeneratorResponseType::Get => { + let mut interns = INTERNS.write().unwrap(); + let constraint = TypeConstraint(interns.insert(response.constraints.unwrap_one())); + Ok(GeneratorResponse::Get(Get( + constraint, + interns.insert(response.values.unwrap_one()), + ))) + } + PyGeneratorResponseType::GetMulti => { + let mut interns = INTERNS.write().unwrap(); + let continues = response + .constraints + .to_vec() + .into_iter() + .zip(response.values.to_vec().into_iter()) + .map(|(c, v)| Get(TypeConstraint(interns.insert(c)), interns.insert(v))) + .collect(); + Ok(GeneratorResponse::GetMulti(continues)) + } + } +} + /// /// NB: Panics on failure. Only recommended for use with built-in functions, such as /// those configured in types::Types. @@ -206,84 +233,35 @@ where pub type ExternContext = raw::c_void; pub struct Externs { - context: *const ExternContext, - log: LogExtern, - log_level: u8, - call: CallExtern, - eval: EvalExtern, - identify: IdentifyExtern, - equals: EqualsExtern, - clone_val: CloneValExtern, - drop_handles: DropHandlesExtern, - satisfied_by: SatisfiedByExtern, - satisfied_by_type: SatisfiedByTypeExtern, - store_list: StoreListExtern, - store_bytes: StoreBytesExtern, - store_i32: StoreI32Extern, - project: ProjectExtern, - project_ignoring_type: ProjectIgnoringTypeExtern, - project_multi: ProjectMultiExtern, - type_to_str: TypeToStrExtern, - val_to_str: ValToStrExtern, - create_exception: CreateExceptionExtern, + pub context: *const ExternContext, + pub log_level: u8, + pub log: LogExtern, + pub call: CallExtern, + pub generator_send: GeneratorSendExtern, + pub eval: EvalExtern, + pub identify: IdentifyExtern, + pub equals: EqualsExtern, + pub clone_val: CloneValExtern, + pub drop_handles: DropHandlesExtern, + pub satisfied_by: SatisfiedByExtern, + pub satisfied_by_type: SatisfiedByTypeExtern, + pub store_list: StoreListExtern, + pub store_bytes: StoreBytesExtern, + pub store_i32: StoreI32Extern, + pub project: ProjectExtern, + pub project_ignoring_type: ProjectIgnoringTypeExtern, + pub project_multi: ProjectMultiExtern, + pub type_to_str: TypeToStrExtern, + pub val_to_str: ValToStrExtern, + pub create_exception: CreateExceptionExtern, // TODO: This type is also declared on `types::Types`. - py_str_type: TypeId, + pub py_str_type: TypeId, } // The pointer to the context is safe for sharing between threads. unsafe impl Sync for Externs {} unsafe impl Send for Externs {} -impl Externs { - pub fn new( - ext_context: *const ExternContext, - log: LogExtern, - log_level: u8, - call: CallExtern, - eval: EvalExtern, - identify: IdentifyExtern, - equals: EqualsExtern, - clone_val: CloneValExtern, - drop_handles: DropHandlesExtern, - type_to_str: TypeToStrExtern, - val_to_str: ValToStrExtern, - satisfied_by: SatisfiedByExtern, - satisfied_by_type: SatisfiedByTypeExtern, - store_list: StoreListExtern, - store_bytes: StoreBytesExtern, - store_i32: StoreI32Extern, - project: ProjectExtern, - project_ignoring_type: ProjectIgnoringTypeExtern, - project_multi: ProjectMultiExtern, - create_exception: CreateExceptionExtern, - py_str_type: TypeId, - ) -> Externs { - Externs { - context: ext_context, - log: log, - log_level: log_level, - call: call, - eval: eval, - identify: identify, - equals: equals, - clone_val: clone_val, - drop_handles: drop_handles, - satisfied_by: satisfied_by, - satisfied_by_type: satisfied_by_type, - store_list: store_list, - store_bytes: store_bytes, - store_i32: store_i32, - project: project, - project_ignoring_type: project_ignoring_type, - project_multi: project_multi, - type_to_str: type_to_str, - val_to_str: val_to_str, - create_exception: create_exception, - py_str_type: py_str_type, - } - } -} - pub type LogExtern = extern "C" fn(*const ExternContext, u8, str_ptr: *const u8, str_len: u64); // TODO: Type alias used to avoid rustfmt breaking itself by rendering a 101 character line. @@ -324,11 +302,17 @@ pub struct PyResult { value: Value, } +impl PyResult { + fn failure_from(v: Value) -> Failure { + let traceback = project_str(&v, "_formatted_exc"); + Failure::Throw(v, traceback) + } +} + impl From for Result { fn from(result: PyResult) -> Self { if result.is_throw { - let traceback = project_str(&result.value, "_formatted_exc"); - Err(Failure::Throw(result.value, traceback)) + Err(PyResult::failure_from(result.value)) } else { Ok(result.value) } @@ -350,6 +334,32 @@ impl From> for PyResult { } } +// Only constructed from the python side. +#[allow(dead_code)] +#[repr(u8)] +pub enum PyGeneratorResponseType { + Break = 0, + Throw = 1, + Get = 2, + GetMulti = 3, +} + +#[repr(C)] +pub struct PyGeneratorResponse { + res_type: PyGeneratorResponseType, + values: ValueBuffer, + constraints: ValueBuffer, +} + +#[derive(Debug)] +pub struct Get(pub TypeConstraint, pub Key); + +pub enum GeneratorResponse { + Break(Value), + Get(Get), + GetMulti(Vec), +} + // The result of an `identify` call, including the __hash__ of a Value and its TypeId. #[repr(C)] pub struct Ident { @@ -358,7 +368,13 @@ pub struct Ident { pub type_id: TypeId, } -// Points to an array containing a series of values allocated by Python. +/// +/// Points to an array containing a series of values allocated by Python. +/// +/// TODO: An interesting optimization might be possible where we avoid actually +/// allocating the values array for values_len == 1, and instead store the Value in +/// the `handle_` field. +/// #[repr(C)] pub struct ValueBuffer { values_ptr: *mut Value, @@ -375,6 +391,20 @@ impl ValueBuffer { |value_vec| unsafe { value_vec.iter().map(|v| v.clone_without_handle()).collect() }, ) } + + /// Asserts that the ValueBuffer contains one value, and returns it. + pub fn unwrap_one(&self) -> Value { + assert!( + self.values_len == 1, + "ValueBuffer contained more than one value: {}", + self.values_len + ); + with_vec( + self.values_ptr, + self.values_len as usize, + |value_vec| unsafe { value_vec.iter().next().unwrap().clone_without_handle() }, + ) + } } // Points to an array of TypeIds. @@ -467,6 +497,9 @@ pub type CreateExceptionExtern = pub type CallExtern = extern "C" fn(*const ExternContext, *const Value, *const Value, u64) -> PyResult; +pub type GeneratorSendExtern = + extern "C" fn(*const ExternContext, *const Value, *const Value) -> PyGeneratorResponse; + pub type EvalExtern = extern "C" fn(*const ExternContext, python_ptr: *const u8, python_len: u64) -> PyResult; diff --git a/src/rust/engine/src/lib.rs b/src/rust/engine/src/lib.rs index 30907d87839..f8d71dd3971 100644 --- a/src/rust/engine/src/lib.rs +++ b/src/rust/engine/src/lib.rs @@ -42,10 +42,11 @@ use std::path::{Path, PathBuf}; use context::Core; use core::{Failure, Function, Key, TypeConstraint, TypeId, Value}; use externs::{Buffer, BufferBuffer, CallExtern, CloneValExtern, CreateExceptionExtern, - DropHandlesExtern, EqualsExtern, EvalExtern, ExternContext, Externs, IdentifyExtern, - LogExtern, ProjectExtern, ProjectIgnoringTypeExtern, ProjectMultiExtern, PyResult, - SatisfiedByExtern, SatisfiedByTypeExtern, StoreBytesExtern, StoreI32Extern, - StoreListExtern, TypeIdBuffer, TypeToStrExtern, ValToStrExtern}; + DropHandlesExtern, EqualsExtern, EvalExtern, ExternContext, Externs, + GeneratorSendExtern, IdentifyExtern, LogExtern, ProjectExtern, + ProjectIgnoringTypeExtern, ProjectMultiExtern, PyResult, SatisfiedByExtern, + SatisfiedByTypeExtern, StoreBytesExtern, StoreI32Extern, StoreListExtern, + TypeIdBuffer, TypeToStrExtern, ValToStrExtern}; use rule_graph::{GraphMaker, RuleGraph}; use scheduler::{ExecutionRequest, RootResult, Scheduler}; use tasks::Tasks; @@ -119,10 +120,11 @@ impl RawNodes { #[no_mangle] pub extern "C" fn externs_set( - ext_context: *const ExternContext, + context: *const ExternContext, log: LogExtern, log_level: u8, call: CallExtern, + generator_send: GeneratorSendExtern, eval: EvalExtern, identify: IdentifyExtern, equals: EqualsExtern, @@ -141,11 +143,12 @@ pub extern "C" fn externs_set( create_exception: CreateExceptionExtern, py_str_type: TypeId, ) { - externs::set_externs(Externs::new( - ext_context, + externs::set_externs(Externs { + context, log, log_level, call, + generator_send, eval, identify, equals, @@ -163,7 +166,7 @@ pub extern "C" fn externs_set( project_multi, create_exception, py_str_type, - )); + }); } #[no_mangle] @@ -206,6 +209,7 @@ pub extern "C" fn scheduler_create( type_link: TypeConstraint, type_process_request: TypeConstraint, type_process_result: TypeConstraint, + type_generator: TypeConstraint, type_string: TypeId, type_bytes: TypeId, build_root_buf: Buffer, @@ -244,6 +248,7 @@ pub extern "C" fn scheduler_create( link: type_link, process_request: type_process_request, process_result: type_process_result, + generator: type_generator, string: type_string, bytes: type_bytes, }, @@ -323,6 +328,13 @@ pub extern "C" fn tasks_task_begin( }) } +#[no_mangle] +pub extern "C" fn tasks_add_get(tasks_ptr: *mut Tasks, product: TypeConstraint, subject: TypeId) { + with_tasks(tasks_ptr, |tasks| { + tasks.add_get(product, subject); + }) +} + #[no_mangle] pub extern "C" fn tasks_add_select(tasks_ptr: *mut Tasks, product: TypeConstraint) { with_tasks(tasks_ptr, |tasks| { diff --git a/src/rust/engine/src/nodes.rs b/src/rust/engine/src/nodes.rs index 07cdb278e60..60d4cea056d 100644 --- a/src/rust/engine/src/nodes.rs +++ b/src/rust/engine/src/nodes.rs @@ -9,6 +9,7 @@ use std::collections::BTreeMap; use std::fmt; use std::os::unix::ffi::OsStrExt; use std::path::{Path, PathBuf}; +use std::sync::Arc; use futures::future::{self, Future}; use tempdir::TempDir; @@ -124,6 +125,21 @@ impl Select { } } + pub fn new_with_entries( + product: TypeConstraint, + subject: Key, + variants: Variants, + entries: rule_graph::Entries, + ) -> Select { + let selector = selectors::Select::without_variant(product); + Select { + selector: selector, + subject: subject, + variants: variants, + entries: entries, + } + } + pub fn new_with_selector( selector: selectors::Select, subject: Key, @@ -356,7 +372,7 @@ impl Select { product: self.product().clone(), variants: self.variants.clone(), task: task, - entry: entry.clone(), + entry: Arc::new(entry.clone()), }) }) .collect::>>() @@ -883,7 +899,7 @@ pub struct Task { product: TypeConstraint, variants: Variants, task: tasks::Task, - entry: rule_graph::Entry, + entry: Arc, } impl Task { @@ -908,6 +924,60 @@ impl Task { } } } + + fn gen_get( + context: &Context, + entry: Arc, + gets: Vec, + ) -> NodeFuture> { + let get_futures = gets + .into_iter() + .map(|get| { + let externs::Get(product, subject) = get; + let entries = context + .core + .rule_graph + .edges_for_inner(&entry) + .expect("edges for task exist.") + .entries_for(&rule_graph::SelectKey::JustGet(selectors::Get { + product: product, + subject: subject.type_id().clone(), + })); + Select::new_with_entries(product, subject, Default::default(), entries) + .run(context.clone()) + .map_err(|e| was_required(e)) + }) + .collect::>(); + future::join_all(get_futures).to_boxed() + } + + /// + /// Given a python generator Value, loop to request the generator's dependencies until + /// it completes with a result Value. + /// + fn generate( + context: Context, + entry: Arc, + generator: Value, + ) -> NodeFuture { + future::loop_fn(externs::eval("None").unwrap(), move |input| { + let context = context.clone(); + let entry = entry.clone(); + future::result(externs::generator_send(&generator, &input)).and_then(move |response| { + match response { + externs::GeneratorResponse::Get(get) => Self::gen_get(&context, entry, vec![get]) + .map(|vs| future::Loop::Continue(vs.into_iter().next().unwrap())) + .to_boxed() as BoxFuture<_, _>, + externs::GeneratorResponse::GetMulti(gets) => Self::gen_get(&context, entry, gets) + .map(|vs| future::Loop::Continue(externs::store_list(vs.iter().collect(), false))) + .to_boxed() as BoxFuture<_, _>, + externs::GeneratorResponse::Break(val) => { + future::ok(future::Loop::Break(val)).to_boxed() as BoxFuture<_, _> + } + } + }) + }).to_boxed() + } } impl Node for Task { @@ -923,11 +993,22 @@ impl Node for Task { .collect::>(), ); - let task = self.task.clone(); + let func = self.task.func.clone(); + let entry = self.entry.clone(); deps .then(move |deps_result| match deps_result { - Ok(deps) => externs::call(&externs::val_for(&task.func.0), &deps), - Err(err) => Err(err), + Ok(deps) => externs::call(&externs::val_for(&func.0), &deps), + Err(failure) => Err(failure), + }) + .then(move |task_result| match task_result { + Ok(val) => { + if externs::satisfied_by(&context.core.types.generator, &val) { + Self::generate(context, entry, val) + } else { + ok(val) + } + } + Err(failure) => err(failure), }) .to_boxed() } diff --git a/src/rust/engine/src/rule_graph.rs b/src/rust/engine/src/rule_graph.rs index bebc2fb3612..818a21b0502 100644 --- a/src/rust/engine/src/rule_graph.rs +++ b/src/rust/engine/src/rule_graph.rs @@ -8,7 +8,7 @@ use std::io; use core::{Function, Key, TypeConstraint, TypeId, Value, ANY_TYPE}; use externs; -use selectors::{Select, SelectDependencies, Selector}; +use selectors::{Get, Select, SelectDependencies, Selector}; use tasks::{Task, Tasks}; #[derive(Eq, Hash, PartialEq, Clone, Debug)] @@ -49,7 +49,11 @@ impl Entry { #[derive(Eq, Hash, PartialEq, Clone, Debug)] pub struct RootEntry { subject_type: TypeId, + // TODO: A RootEntry can only have one declared `Selector`, and no declared `Get`s, but these + // are shaped as Vecs to temporarily minimize the re-shuffling in `_construct_graph`. Remove in + // a future commit. clause: Vec, + gets: Vec, } impl From for Entry { @@ -146,6 +150,8 @@ impl Entry { /// #[derive(Eq, Hash, PartialEq, Clone, Debug)] pub enum SelectKey { + // A Get for a particular product/subject pair. + JustGet(Get), // A bare select with no projection. JustSelect(Select), // The initial select of a multi-select operator, eg SelectDependencies. @@ -323,10 +329,18 @@ impl<'t> GraphMaker<'t> { let mut was_unfulfillable = false; match entry { Entry::InnerEntry(InnerEntry { - rule: Task { ref clause, .. }, + rule: Task { + ref clause, + ref gets, + .. + }, .. }) - | Entry::Root(RootEntry { ref clause, .. }) => { + | Entry::Root(RootEntry { + ref clause, + ref gets, + .. + }) => { for selector in clause { match selector { &Selector::Select(ref select) => { @@ -503,6 +517,39 @@ impl<'t> GraphMaker<'t> { } } } + for get in gets { + match get { + &Get { + ref subject, + ref product, + } => { + let rules_or_literals_for_selector = rhs(&self.tasks, subject.clone(), product); + if rules_or_literals_for_selector.is_empty() { + mark_unfulfillable( + &mut unfulfillable_rules, + &entry, + subject.clone(), + format!( + "no rule was available to compute {} for {}", + type_constraint_str(product.clone()), + type_str(subject.clone()) + ), + ); + was_unfulfillable = true; + continue; + } + add_rules_to_graph( + &mut rules_to_traverse, + &mut rule_dependency_edges, + &mut unfulfillable_rules, + &mut root_rule_dependency_edges, + &entry, + SelectKey::JustGet(get.clone()), + rules_or_literals_for_selector, + ); + } + } + } } _ => panic!( "Entry type that cannot dependencies was not filtered out {:?}", @@ -611,6 +658,7 @@ impl<'t> GraphMaker<'t> { variant_key: None, }), ], + gets: vec![], }) } } @@ -638,6 +686,7 @@ pub struct RuleGraph { unfulfillable_rules: UnfulfillableRuleMap, } +// TODO: Take by reference. fn type_constraint_str(type_constraint: TypeConstraint) -> String { let str_val = externs::call_method(&to_val(type_constraint), "graph_str", &[]) .expect("string from calling repr"); @@ -761,6 +810,7 @@ impl RuleGraph { let root = RootEntry { subject_type: subject_type, clause: vec![selector], + gets: vec![], }; self.root_dependencies.get(&root).map(|e| e.clone()) } @@ -852,7 +902,6 @@ impl RuleGraph { }) } - // TODO instead of this, make own fmt thing that accepts externs pub fn visualize(&self, f: &mut io::Write) -> io::Result<()> { if self.root_dependencies.is_empty() && self.rule_dependency_edges.is_empty() { write!(f, "digraph {{\n")?; @@ -1029,13 +1078,17 @@ fn update_edges_based_on_unfulfillable_entry( } fn rhs_for_select(tasks: &Tasks, subject_type: TypeId, select: &Select) -> Entries { - if externs::satisfied_by_type(&select.product, &subject_type) { + rhs(tasks, subject_type, &select.product) +} + +fn rhs(tasks: &Tasks, subject_type: TypeId, product_type: &TypeConstraint) -> Entries { + if externs::satisfied_by_type(product_type, &subject_type) { // NB a matching subject is always picked first vec![Entry::new_subject_is_product(subject_type)] - } else if let Some(&(ref key, _)) = tasks.gen_singleton(&select.product) { - vec![Entry::new_singleton(key.clone(), select.product.clone())] + } else if let Some(&(ref key, _)) = tasks.gen_singleton(product_type) { + vec![Entry::new_singleton(key.clone(), product_type.clone())] } else { - match tasks.gen_tasks(&select.product) { + match tasks.gen_tasks(product_type) { Some(ref matching_tasks) => matching_tasks .iter() .map(|t| Entry::new_inner(subject_type, t)) diff --git a/src/rust/engine/src/selectors.rs b/src/rust/engine/src/selectors.rs index ff7159cadf2..f47cfdf658d 100644 --- a/src/rust/engine/src/selectors.rs +++ b/src/rust/engine/src/selectors.rs @@ -3,6 +3,12 @@ use core::{Field, TypeConstraint, TypeId}; +#[derive(Clone, Debug, Eq, Hash, PartialEq)] +pub struct Get { + pub product: TypeConstraint, + pub subject: TypeId, +} + #[derive(Clone, Debug, Eq, Hash, PartialEq)] pub struct Select { pub product: TypeConstraint, diff --git a/src/rust/engine/src/tasks.rs b/src/rust/engine/src/tasks.rs index 625b2929c1c..c65043e36ea 100644 --- a/src/rust/engine/src/tasks.rs +++ b/src/rust/engine/src/tasks.rs @@ -5,12 +5,13 @@ use std::collections::{HashMap, HashSet}; use core::{Field, Function, Key, TypeConstraint, TypeId, Value, FNV}; use externs; -use selectors::{Select, SelectDependencies, SelectProjection, Selector}; +use selectors::{Get, Select, SelectDependencies, SelectProjection, Selector}; #[derive(Clone, Debug, Eq, Hash, PartialEq)] pub struct Task { pub product: TypeConstraint, pub clause: Vec, + pub gets: Vec, pub func: Function, pub cacheable: bool, } @@ -95,10 +96,23 @@ impl Tasks { cacheable: true, product: product, clause: Vec::new(), + gets: Vec::new(), func: func, }); } + pub fn add_get(&mut self, product: TypeConstraint, subject: TypeId) { + self + .preparing + .as_mut() + .expect("Must `begin()` a task creation before adding gets!") + .gets + .push(Get { + product: product, + subject: subject, + }); + } + pub fn add_select(&mut self, product: TypeConstraint, variant_key: Option) { self.clause(Selector::Select(Select { product: product, @@ -163,6 +177,7 @@ impl Tasks { tasks, ); task.clause.shrink_to_fit(); + task.gets.shrink_to_fit(); tasks.push(task); } } diff --git a/src/rust/engine/src/types.rs b/src/rust/engine/src/types.rs index 3ad622b07e8..956a6c3f8c1 100644 --- a/src/rust/engine/src/types.rs +++ b/src/rust/engine/src/types.rs @@ -22,6 +22,7 @@ pub struct Types { pub link: TypeConstraint, pub process_request: TypeConstraint, pub process_result: TypeConstraint, + pub generator: TypeConstraint, pub string: TypeId, pub bytes: TypeId, } diff --git a/tests/python/pants_test/engine/examples/planners.py b/tests/python/pants_test/engine/examples/planners.py index 2621341521f..8cd6c89d769 100644 --- a/tests/python/pants_test/engine/examples/planners.py +++ b/tests/python/pants_test/engine/examples/planners.py @@ -22,7 +22,7 @@ from pants.engine.parser import SymbolTable from pants.engine.rules import SingletonRule, TaskRule, rule from pants.engine.scheduler import LocalScheduler -from pants.engine.selectors import Select, SelectDependencies, SelectProjection, SelectVariant +from pants.engine.selectors import Get, Select, SelectDependencies, SelectVariant from pants.engine.struct import HasProducts, Struct, StructWithDeps, Variants from pants.util.meta import AbstractClass from pants.util.objects import datatype @@ -92,11 +92,6 @@ class ScalaInferredDepsSources(Sources): extensions = ('.scala',) -class ImportedJVMPackages(datatype('ImportedJVMPackages', ['dependencies'])): - """Holds a list of 'JVMPackageName' dependencies.""" - pass - - class JVMPackageName(datatype('JVMPackageName', ['name'])): """A typedef to represent a fully qualified JVM package name.""" pass @@ -106,10 +101,10 @@ class SourceRoots(datatype('SourceRoots', ['srcroots'])): """Placeholder for the SourceRoot subsystem.""" +@printing_func @rule(Address, [Select(JVMPackageName), SelectDependencies(AddressFamily, Snapshot, field='dir_stats', field_types=(Dir,))]) -@printing_func def select_package_address(jvm_package_name, address_families): """Return the Address from the given AddressFamilies which provides the given package.""" addresses = [address for address_family in address_families @@ -123,8 +118,8 @@ def select_package_address(jvm_package_name, address_families): return addresses[0].to_address() -@rule(PathGlobs, [Select(JVMPackageName), Select(SourceRoots)]) @printing_func +@rule(PathGlobs, [Select(JVMPackageName), Select(SourceRoots)]) def calculate_package_search_path(jvm_package_name, source_roots): """Return PathGlobs to match directories where the given JVMPackageName might exist.""" rel_package_dir = jvm_package_name.name.replace('.', os_sep) @@ -132,11 +127,11 @@ def calculate_package_search_path(jvm_package_name, source_roots): return PathGlobs.create('', include=specs) -@rule(ImportedJVMPackages, - [SelectProjection(FilesContent, PathGlobs, 'path_globs', ScalaInferredDepsSources)]) @printing_func -def extract_scala_imports(source_files_content): - """A toy example of dependency inference. Would usually be a compiler plugin.""" +@rule(ScalaSources, [Select(ScalaInferredDepsSources)]) +def reify_scala_sources(sources): + """Given a ScalaInferredDepsSources object, create ScalaSources.""" + source_files_content = yield Get(FilesContent, PathGlobs, sources.path_globs) packages = set() import_re = re.compile(r'^import ([^;]*);?$') for filecontent in source_files_content.dependencies: @@ -144,18 +139,12 @@ def extract_scala_imports(source_files_content): match = import_re.search(line) if match: packages.add(match.group(1).rsplit('.', 1)[0]) - return ImportedJVMPackages([JVMPackageName(p) for p in packages]) + dependency_addresses = yield [Get(Address, JVMPackageName(p)) for p in packages] -@rule(ScalaSources, - [Select(ScalaInferredDepsSources), - SelectDependencies(Address, ImportedJVMPackages, field_types=(JVMPackageName,))]) -@printing_func -def reify_scala_sources(sources, dependency_addresses): - """Given a ScalaInferredDepsSources object and its inferred dependencies, create ScalaSources.""" kwargs = sources._asdict() kwargs['dependencies'] = list(set(dependency_addresses)) - return ScalaSources(**kwargs) + yield ScalaSources(**kwargs) class Requirement(Struct): @@ -214,8 +203,8 @@ def __init__(self, org, name, **kwargs): super(ManagedJar, self).__init__(org=org, name=name, **kwargs) -@rule(Jar, [Select(ManagedJar), SelectVariant(ManagedResolve, 'resolve')]) @printing_func +@rule(Jar, [Select(ManagedJar), SelectVariant(ManagedResolve, 'resolve')]) def select_rev(managed_jar, managed_resolve): (org, name) = (managed_jar.org, managed_jar.name) rev = managed_resolve.revs.get('{}#{}'.format(org, name), None) @@ -224,14 +213,14 @@ def select_rev(managed_jar, managed_resolve): return Jar(org=managed_jar.org, name=managed_jar.name, rev=rev) -@rule(Classpath, [Select(Jar)]) @printing_func +@rule(Classpath, [Select(Jar)]) def ivy_resolve(jars): return Classpath(creator='ivy_resolve') -@rule(Classpath, [Select(ResourceSources)]) @printing_func +@rule(Classpath, [Select(ResourceSources)]) def isolate_resources(resources): """Copies resources into a private directory, and provides them as a Classpath entry.""" return Classpath(creator='isolate_resources') @@ -287,8 +276,8 @@ class BuildPropertiesConfiguration(Struct): pass -@rule(Classpath, [Select(BuildPropertiesConfiguration)]) @printing_func +@rule(Classpath, [Select(BuildPropertiesConfiguration)]) def write_name_file(name): """Write a file containing the name of this target in the CWD.""" return Classpath(creator='write_name_file') @@ -317,18 +306,18 @@ class ScroogeJavaConfiguration(ScroogeConfiguration): @rule(ScalaSources, [Select(ThriftSources), - SelectVariant(ScroogeScalaConfiguration, 'thrift'), - SelectProjection(Classpath, Address, 'tool_address', Scrooge)]) -def gen_scrooge_scala_thrift(sources, config, scrooge_classpath): - return gen_scrooge_thrift(sources, config, scrooge_classpath) + SelectVariant(ScroogeScalaConfiguration, 'thrift')]) +def gen_scrooge_scala_thrift(sources, config): + scrooge_classpath = yield Get(Classpath, Address, Scrooge.tool_address) + yield gen_scrooge_thrift(sources, config, scrooge_classpath) @rule(JavaSources, [Select(ThriftSources), - SelectVariant(ScroogeJavaConfiguration, 'thrift'), - SelectProjection(Classpath, Address, 'tool_address', Scrooge)]) -def gen_scrooge_java_thrift(sources, config, scrooge_classpath): - return gen_scrooge_thrift(sources, config, scrooge_classpath) + SelectVariant(ScroogeJavaConfiguration, 'thrift')]) +def gen_scrooge_java_thrift(sources, config): + scrooge_classpath = yield Get(Classpath, Address, Scrooge.tool_address) + yield gen_scrooge_thrift(sources, config, scrooge_classpath) @printing_func @@ -339,18 +328,18 @@ def gen_scrooge_thrift(sources, config, scrooge_classpath): return ScalaSources(files=['Fake.scala'], dependencies=config.dependencies) +@printing_func @rule(Classpath, [Select(JavaSources), SelectDependencies(Classpath, JavaSources, field_types=(Address, Jar))]) -@printing_func def javac(sources, classpath): return Classpath(creator='javac') +@printing_func @rule(Classpath, [Select(ScalaSources), SelectDependencies(Classpath, ScalaSources, field_types=(Address, Jar))]) -@printing_func def scalac(sources, classpath): return Classpath(creator='scalac') @@ -469,7 +458,6 @@ def setup_json_scheduler(build_root, native): ] + [ # scala dependency inference reify_scala_sources, - extract_scala_imports, select_package_address, calculate_package_search_path, SingletonRule(SourceRoots, SourceRoots(('src/java','src/scala'))), diff --git a/tests/python/pants_test/engine/test_build_files.py b/tests/python/pants_test/engine/test_build_files.py index 50549102e4a..48046e2e1cd 100644 --- a/tests/python/pants_test/engine/test_build_files.py +++ b/tests/python/pants_test/engine/test_build_files.py @@ -9,32 +9,28 @@ import unittest from pants.build_graph.address import Address -from pants.engine.addressable import (Exactly, SubclassesOf, addressable, addressable_dict, - addressable_list) -from pants.engine.build_files import ResolvedTypeMismatchError, create_graph_rules -from pants.engine.fs import create_fs_rules +from pants.engine.addressable import Exactly, addressable, addressable_dict +from pants.engine.build_files import (ResolvedTypeMismatchError, create_graph_rules, + parse_address_family) +from pants.engine.fs import Dir, FileContent, FilesContent, PathGlobs, create_fs_rules from pants.engine.mapper import AddressMapper, ResolveError from pants.engine.nodes import Return, Throw from pants.engine.parser import SymbolTable -from pants.engine.struct import HasProducts, Struct, StructWithDeps +from pants.engine.struct import Struct, StructWithDeps from pants_test.engine.examples.parsers import (JsonParser, PythonAssignmentsParser, PythonCallbacksParser) from pants_test.engine.scheduler_test_base import SchedulerTestBase +from pants_test.engine.util import Target, run_rule -class Target(Struct, HasProducts): - - def __init__(self, name=None, configurations=None, **kwargs): - super(Target, self).__init__(name=name, **kwargs) - self.configurations = configurations - - @property - def products(self): - return self.configurations - - @addressable_list(SubclassesOf(Struct)) - def configurations(self): - pass +class ParseAddressFamilyTest(unittest.TestCase): + def test_empty(self): + """Test that parsing an empty BUILD file results in an empty AddressFamily.""" + address_mapper = AddressMapper(JsonParser(TestTable())) + af = run_rule(parse_address_family, address_mapper, Dir('/dev/null'), { + (FilesContent, PathGlobs): lambda _: FilesContent([FileContent('/dev/null/BUILD', '')]) + }) + self.assertEquals(len(af.objects_by_name), 0) class ApacheThriftConfiguration(StructWithDeps): diff --git a/tests/python/pants_test/engine/test_rules.py b/tests/python/pants_test/engine/test_rules.py index ab01aaa7c36..565fbb04b4e 100644 --- a/tests/python/pants_test/engine/test_rules.py +++ b/tests/python/pants_test/engine/test_rules.py @@ -303,8 +303,8 @@ def test_full_graph_for_planner_example(self): else: pass - self.assertTrue(10 < len(all_rules)) - self.assertTrue(30 < len(root_rule_lines)) # 2 lines per entry + self.assertTrue(6 < len(all_rules)) + self.assertTrue(12 < len(root_rule_lines)) # 2 lines per entry def test_smallest_full_test_multiple_root_subject_types(self): rules = [ diff --git a/tests/python/pants_test/engine/util.py b/tests/python/pants_test/engine/util.py index c97ed4ae9c5..c29aea25d6f 100644 --- a/tests/python/pants_test/engine/util.py +++ b/tests/python/pants_test/engine/util.py @@ -6,6 +6,7 @@ unicode_literals, with_statement) import re +from types import GeneratorType from pants.binaries.binary_util import BinaryUtilPrivate from pants.engine.addressable import SubclassesOf, addressable_list @@ -13,11 +14,72 @@ from pants.engine.parser import SymbolTable from pants.engine.rules import RuleIndex from pants.engine.scheduler import WrappedNativeScheduler +from pants.engine.selectors import Get from pants.engine.struct import HasProducts, Struct from pants_test.option.util.fakes import create_options_for_optionables from pants_test.subsystem.subsystem_util import init_subsystem +def run_rule(rule, *args): + """A test helper function that runs an @rule with a set of arguments and Get providers. + + An @rule named `my_rule` that takes one argument and makes no `Get` requests can be invoked + like so (although you could also just invoke it directly): + ``` + return_value = run_rule(my_rule, arg1) + ``` + + In the case of an @rule that makes Get requests, things get more interesting: an extra argument + is required that represents a dict mapping (product, subject) type pairs to one argument functions + that take a subject value and return a product value. + + So in the case of an @rule named `my_co_rule` that takes one argument and makes Get requests + for product and subject types (Listing, Dir), the invoke might look like: + ``` + return_value = run_rule(my_co_rule, arg1, {(Listing, Dir): lambda x: Listing(..)}) + ``` + + :returns: The return value of the completed @rule. + """ + + task_rule = getattr(rule, '_rule', None) + if task_rule is None: + raise TypeError('Expected to receive a decorated `@rule`; got: {}'.format(rule)) + + gets_len = len(task_rule.input_gets) + + if len(args) != len(task_rule.input_selectors) + (1 if gets_len else 0): + raise ValueError('Rule expected to receive arguments of the form: {}; got: {}'.format( + task_rule.input_selectors, args)) + + args, get_providers = (args[:-1], args[-1]) if gets_len > 0 else (args, {}) + if gets_len != len(get_providers): + raise ValueError('Rule expected to receive Get providers for {}; got: {}'.format( + task_rule.input_gets, get_providers)) + + res = rule(*args) + if not isinstance(res, GeneratorType): + return res + + def get(product, subject): + provider = get_providers.get((product, type(subject))) + if provider is None: + raise AssertionError('Rule requested: Get{}, which cannot be satisfied.'.format( + (product, type(subject), subject))) + return provider(subject) + + rule_coroutine = res + rule_input = None + while True: + res = rule_coroutine.send(rule_input) + if isinstance(res, Get): + rule_input = get(res.product, res.subject) + elif type(res) in (tuple, list): + rule_input = [get(g.product, g.subject) for g in res] + else: + return res + + def init_native(): """Initialize and return a `Native` instance.""" init_subsystem(BinaryUtilPrivate.Factory)