Allow Covalent server to process workflows without requiring their deps #754
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use JSON-encodability to test if the object is a built-in type.
Same interface as `build_graph` except args and kwargs are assumed to comprise of TransportableObjects. TransportableObjects which encode fundamental types (determined by JSON-encodability) are deserialized before invoking the workflow function. It is assumed that only such arguments are used to determine control flow in the workflow (e.g. `for i in range(iterations)` where `iterations` is a lattice-level argument).
Also add `serialize_to_json()` method
Idempotent version of usual constructor: `make_transportable(make_transportable(obj)) == make_transportable(obj)`
Precisely, "collection" == list or dict Each item in the collection is assumed to be either a `TransportableObject` or a collection, which we deserialize recursively.
* Workflow inputs are encoded to `TransportableObjects`, and `build_graph` is replaced by `build_graph_encoded`. * Parameter type nodes are also encoded. * Dispatcher passes encoded inputs directly to `executor.execute()` * `wrapper_fn` decodes each arg and kwarg. Lists and dicts are recursively decoded. After the main function is run, the output is encoded before returning it to the dispatcher. * Handle collection-type inputs to electrons by modifying `to_electron_collection` to return a collection of decoded objects
List/dict structure is preserved.
Will be removed after _post_process starts running outside the dispatcher process.
* Add new result statuses PENDING_POSTPROCESSING and FAILED_POSTPROCESSING * Default to "local" (dispatcher process) preprocessing executor for now.
Potentially useful if the server lacks the dependencies to _post_process the workflow. Supercedes previous `get_decoded_result()`
Workflows willbe postprocessed using that executor. The special value "client" pauses the workflow before postprocessing. In this case, the client can retrieve the result object and perform postprocessing client-side using Result.post_process().
The same executor could be used for building sublattice graphs.
This gives one the option to postprocess client-side.
Store docstring in a attribute of TransportableObject to be accessed without deserializing the object
|
Need to adjust some tests to expect encoded outputs. |
Record "__name__" attr when serializing an object
These are the keys in the executor_plugins_map
Classes covered: * Deps * DepsCall * DepsBash * LocalExecutor * DaskExecutor Preparation for making `TransportGraph` JSON-serializable
scottwn
previously approved these changes
Jul 13, 2022
scottwn
previously approved these changes
Jul 14, 2022
scottwn
previously approved these changes
Jul 14, 2022
AlejandroEsquivel
previously approved these changes
Jul 15, 2022
Prasy12
previously approved these changes
Jul 15, 2022
Prasy12
approved these changes
Jul 15, 2022
cjao
requested review from
Prasy12 and
AlejandroEsquivel
and removed request for
AlejandroEsquivel
mshkanth
approved these changes
Jul 15, 2022
AlejandroEsquivel
approved these changes
Jul 15, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #748 (design doc in the link).