Add retrieve temporary list to CalcInfo datastructure #903
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For some calculation one wants to retrieve certain files for parsing but does not want to store them permanently in the repository because for example they may take up a lot of space. To facilitate for this use case we introduce the concept of the retrieve_temporary_list member in the CalcInfo datastructure. Implementations of a Calculation can opt to provide a list of directives, identical to that of the original retrieve_list member, that will instruct the exectution manager to retrieve those files to a local folder for parsing. The difference is that the files listed in the retrieve_temporary_list member will be stored in a FolderData node that will not be stored in the database and which will only be alive for the duration of the parsing. After parsing is completed the retrieved files are garbage collected.
Also deleted the link that was added to the retrieved_temporary_folder which was a left over from testing phase. Since the FolderData is never going to be stored, it also shouldn't have any links
The parser will not do anything, but if the 'retrieve_temporary_files' key was set in the 'template' input node, then it is expected that those files will be temporarily stored in an unstored FolderData node which should be passed in the 'retrieved' argument of the parse_with_retrieved call of the Parser class. In that case, the arguments are checked for the presence of the unstored FolderData node
The daemon integration test leverages the TemplatereplacerCalculation to launch some trivial calculations. We add the output filepath to the 'retrieve_temporary_files' entry of the template dictionary which will cause the TemplatereplacerParser class to check for an unstored FolderData node containing those files, which should be passed to the parse_with_retrieved function called by the execmanager
Also enhance the parser by explicitly verifying that all the files that are listed in the 'template' input node in the retrieve_temp_list key are present in the temporary folder
This means that the parsing of the output file can be moved from the daemon test to the actual parser itself
If no files were specified in the 'retrieve_temporary_list' of the calculations CalcInfo, we should not pass an empty FolderData to the parse_from_calc call of the Parser. Otherwise the parse_with_retrieved call will always get a retrieved_temporary_folder, even if it is empty. By setting retrieved_temporary_folder to None, unless some files have been specified in the retrieve_temporary_list member we maintain the original behavior of the execmanager calling the parser
giovannipizzi
approved these changes
Nov 9, 2017
|
I approved, note however that there is at least a file that needs to get the headers updated. |
|
I will do the headers when I will release the final version, along with contributors, version number etcetera |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This was already merged into
developwith PR's #892 and #894 but we are backporting it in the release