OPTIMIZATION: Support for omitting specific graph objects from file storage #404
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some entities and relationships do not need to be stored on the file system at all. We only need to store graph objects on the file system if we later intend to fetch the data from the job state or iterate over the `_type`. This PR introduces support for specifying metadata in each step that can be used by the `FileSystemGraphObjectStore` that will be used to omit specific data from being written to disk entirely. The data will still get uploaded.
austinkelleher
requested review from
aiwilliams,
ctdio,
ndowmon and
mknoedel
aiwilliams
previously approved these changes
Dec 27, 2020
| }); | ||
|
|
||
| if (indexable.length) { |
There was a problem hiding this comment.
Do you think it would be worthwhile to add a test for the scenario when there are no indexable types?
- @jupiterone/cli@5.3.0 - @jupiterone/integration-sdk-cli@5.3.0 - @jupiterone/integration-sdk-core@5.3.0 - @jupiterone/integration-sdk-dev-tools@5.3.0 - @jupiterone/integration-sdk-private-test-utils@5.3.0 - @jupiterone/integration-sdk-runtime@5.3.0 - @jupiterone/integration-sdk-testing@5.3.0
aiwilliams
approved these changes
Dec 27, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some entities and relationships do not need to be stored on the
file system at all. We only need to store graph objects on the file
system if we later intend to fetch the data from the job state or
iterate over the
_type. This PR introduces support for specifyingmetadata in each step that can be used by the
FileSystemGraphObjectStorethat will be used to omit specific data from being written to disk
entirely. The data will still get uploaded.
A longer term optimization would be actually leveraging the dependency graph to automatically generate this information. We can also make a follow-up improvement that removes the entire locking behavior during these cases because no step will rely on the unindexed data, so there is no reason to lock.