Migrate verdi to the click infrastructure
#1795
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A very common pattern for any verdi command will be an option or an argument that takes on or multiple values that is supposed to identify a specific orm entity. In AiiDA there are three types of identifier: * ID * UUID * STRING Most entities have both an ID as well as a UUID that identifies them. Some have an additional string based identifier, such as a label or a name. Here we define the class IdentifierParamType, which can be used to create reusable arguments and options that will load ORM instances based on the passed values, by deducing whether they are meant to repesent an ID, UUID or STRING. Note that since the type is lost and all values will be strings, one has to guess the type. The value will be attempted to be interpreted as an ID, UUID and then STRING, in that order. This approach can lead to ambiguities in some situations, where if an intended (partial) UUID can be interpreted as a valid ID. Or when a valid STRING also happens to be a valid ID. However, as long as the users do not use labels that are valid integers and use the full UUID in the worst case, chances for these clashes should be highly unlikely.
All ORM entities can be identified with an ID, UUID or optionally a
STRING like identifier. Any potential ambiguity between an ID and UUID,
for example when a partial UUID is also a valid ID, can always be solved
by including at least one dash in the UUID. The ambiguity when a STRING
identifier is also a valid ID or UUID is a little more tricky. We add the
convention here that if the last character of the identifier is an
exclamation point, it will always be interpreted as a STRING type like
identifier. Consider three entities with the following ID, UUID and STRING:
1 b6420d91-7577 'some_name'
2 d9c4c081-f6bb '1'
3 0cf82610-3ae8 'b6420d91-7577'
Trying to select either the second or third entity by their STRING identifier
will always result in the first entity being matched, given that their STRING
identifiers also match the ID and UUID, respectively of the first entity,
To break this one simply needs to use the identifier:
IdentifierParam.convert('1!')
IdentifierParam.convert('b6420d91-7577!')
The added exclamation mark will break the ambiguity by forcing the method to
interpret the identifier as a STRING type.
This behavior is implemented in the `convert` method of the IdentifierParam
parameter type and tests have been added that test the functionality.
This was necessary to allow to refactor the older load_node and load_group functions to use the same implementation as the OrmEntityLoader classes. These changes allow load_node and load_group to leverage the OrmEntityLoader class to do the heavy lifting while keeping the same function signature and functionality. The old query builder utility in `aiida.orm.utils` that these functions where relying on, which was reimplemented in a more general way in the OrmEntityLoader, was removed. The BaseTranslator of the REST API was relying on this function but has been reworked to be able to use the OrmEntityLoader as well.
The subclasses of the OrmEntityLoader have to define a base orm class for which any entity that is to be matched and loaded has to be a sub class of. However, sometimes a user may want to narrow the query set even more and make the requirement on the base class even more specific. To allow this, the get_query_builder and load_entity methods now have the optional sub_classes argument, which takes a tuple of ORM classes, each of which has to be a sub class of the base orm class. When valid, the query will be limited to entities which are a sub class of these specific orm sub classes. This now also allows the IdentifierParamType and its subclasses to, upon construction, define a set of entry points, whose classes will be passed as this sub_classes tuple when a value needs to be converted. The reason to taking entry point strings and not the actual ORM classes is to prevent to have to load the database environment upon import of the verdi commands in order to keep the autocompletion and interface quick and snappy.
To test this also implemented the CalculationParamType and DataParamType as well as their corresponding entity loaders.
#1607) The `append` method of the `QueryBuilder` now accepts a tuple, list or set of orm classes for the `cls` and `type` keyword argument, with the one restriction that all classes share a common base class. This allows the user to append a join for a set of classes with the same projection and filtering rules
Issue #1398 was fixed in a previous commit to allow adding a tuple of classes to the QueryBuilder through the append method, but it did not work for all ORM classes e.g. Group. This commit fixes the problem and enables the new approach for all ORM classes.
Migrates the existing `verdi export` commands to use the new `click` infrastructure. Also added tests for the `create` and `migrate` subcommands.
…1611) ## Missing (suggested to create separate issue & pull request) * Tests for `aiida.control.code.CodeBuilder` ## Added ### ConditionalOption `aiida.cmdline.optiontypes.conditional.py` A `click.Option` subclass, which is required only if a callback function returns True. This callback function recieves the command's context, which means the condition may include values of other parameters. Limitations: * does not work with `is_eager=True` * options with callbacks (and `is_eager=False` get executed last, but do not rely on a particular order between them. -> conditions should not depend on other options with callbacks ### InteractiveOption `aiida.cmdline.optiontypes.interactive.py` A `ConditionalOption` subclass, which * can prompt for missing values (not given as option on CLI) * can give a list of viable options (depends on parameter type) when "?" is entered at the prompt * can depend on other options (like `ConditionalOption`) * prompting for all `ConditionalOptions` of a command can be disabled by passing a `--non-interactive` flag ### multiline input `aiida.cmdline.utils.multi_line_input.py` Functions for opening an editor for the user to enter multi line text as input in the style of `git commit`. Available functionality: * `edit_pre_post`: open editor to create or change Prepend / Append bash scripts, combined, optionally displaying the configuration as a comment * `edit_comment`: open editor to attach comments to nodes, or change existing comments ### PluginParameterType A `click.ParameterType` for plugin names, optionally validates that the plugin loader can find the entry point name in the given category. Can print a list of available plugins on interactive prompt, tab-completion ready. ### CodeBuilder `aiida.control.code` A small utility class to simplify and standardize creating a code using the builder pattern. It is in the `control` module because it is primarily useful when working interactively or writing a tool. ### Reusable options A number of reusable options: `--non-interactive`, `--input-plugin` and others ## Modified * verdi code setup translated to click, using all of the above, introducing non-interactive mode * verdi code setup serves as a usage example for `InteractiveOption`, `PluginParameterType` and `edit_pre_post()`
The aiida.cmdline.params.arguments have been reorganized in similar fashion to aiida.cmdline.params.options. The module contains multiple files with variants of arguments and options, respectively. Any predefined arguments and options, using the Overridable class, are then defined in the __init__.py such that they are also automatically exposed in that level of the module.
Also changed the interface options of the command. The --class-name has been replaced by -e/--entry-point, since that is a more robust way of defining a class in AiiDA. Additionally, the --all option has been removed, since that would be implied if no identifier arguments would be specified but only an entry point.
Often one wants to define a command line parameter whose values should be valid entry points, but not necessarily restricted to one particular entry point group. Here we update PluginParamType to support definition of a tuple of valid entry point groups. This change does bring a complication with it. Since entry point names are only guaranteed to be unique within an entry point group, but not necessarily across groups, the situation may arise where a parameter is defined with support for one or more entry point groups that share an entry point with a common name. The conversion and autocompletion of entry points with a minimal format, i.e. only the entry point name and without the entry point group, may be ambiguous. In this case, the conversion should fail and auto-completion should always return the valid entry point strings in FULL format. In cases where there is no risk of ambiguity, the PluginParamType should support the user to pass all three types of entry point string format and adapt the auto-complete to the form of the incomplete value as much as possible.
Update verdi branch after merging of CI folders
The prefix was not used often and could easily be confused with the prefix that is actually prepended to the echo message. Also added the `echo_deprecated` message which can optionally be told to also call `sys.exit`. A `ExitCode` enum with three predefined exit codes is defined.
Merge develop into verdi
Click uses the timestamp to see if the file has been modified in `click.edit()` but on some filesystems the timestamp has a precision of 1 second
to comply with the logic we expect. Tests have also been adapted as they were not always correct. Also, tests on empty_ok substituted with tests on empty string defaults as empty_ok has been removed from the code.
…k that all parameters passed to a builder are used.
E.g. when a previous option decides if the option should be printed or not Also fixed the tests. NOTE: if the wrong number of interactive options is passed in a test, this will enter an infinite loop and will hang the tests! To decide if we want to have a way to prevent this (e.g. a max_prompt_count variable?)
* Converted verdi comment to use click yo Fairly straightforward change but 'comment' has been changed to an argument rather than an option i.e. it is specified by its position on the command line rather than having to use -c. Added use of secho to verdi user command (instead of ANSI color codes) * Updated files to make prospector and lint happy Bunch of scheduler and other related files * Fixed a couple of bugs with the user orm class * suppress linting for files that need deep refactoring * update exception type raised in get_detailed_jobinfo * mention refactorings in the CHANGELOG.md
* Starting to put verdi shell on click * Adding bpython to the AiiDA requirements, more progress to the verdi shell command * More progerss * Final cleaning * format code and fix shell command docstring * fix linter errors * made bpython an optional extra req (not added to 'all')
there was ambiguity between wf and cal pks in the output
* Starting to prepare the 'verdi setup' command. It is still incomplete, some options are missing, and the test infrastructure has not been adapted yet. * Almost finished the implementation of verdi computer setup Many tests already added for the non-interactive part. Still a few to add for the interactive part, and to adapt the tests. * Added interactive and mixed tests for 'verdi computer setup' Also, changed back the default question from 'disabled' to 'enabled' * Fixing the travis scripts to setup the computer * Minor fixes required in review of #1735 Also, removing two stale files that are not needed after non-interactive verdi code setup
* Migrate `verdi profile` to use the click infrastructure (#1591) * Added testcases for verdi profile command * Updated .pre-commit-config.yaml * Added doc strings in test profile command * Modified profile command for pre-commit * Solved pre-commit error for unused argument in profile command * Minor change: replaced echo_error with echo_critical * test commit for jenkins * tmp commit * Generated random names form dummy profiles in test_profile * Fixed postgres connection issue to delete profile * Updated delete profile command
…1742) * Bulk of static commands ported to click * First draft of commands. Added tests * Add tests * Finished non-interactive verdi computer functions and tests * Added with_dbenv decorators. Made computer delete test safer
* Data subcommand move to click: 1) create a folder for data click commands 2) place subcommands of verdi data in different files * data upf is moved to click * Put list-related functions to a separate file * data bands (not fully) transferred to click * Grouping the list option arguments * Removing not needed load_dbenv * data bands are fully transferred to click * More progress on cif * Remove label and description from data lavel * More progress on cif * data parameter are transferred to click * More progress on cif * Put export in a separate file * More progress on cif * More progress on cif * data remote is fully transferred to click * More progress on cif deposit * More progress on cif deposit * Ported verdi data structure to click * Transfer verdi data trajectory to click * Transfer verdi data array to click * Deposit function in a separate file * More progress on data cif * Small changes in data cif * Fixing various errors * More progress on tests * More progress on verdi data cif tests * Polish the verdi data code: - remove unnecessary imports - put show functionality in a separate file - put commonly used click options in isolated containers * More progress in tests * Transfer verdi data cif to click * Cleaning & correcting the code. * Cleaning tests, cif & general code, fixing problems in argument grouping * Adding tests for cif import * More progress on cif tests * Loading correctly db_env to not affect the tests * Cif tests should be OK (need some cleaning but are complete). We should see why and if we need to have so many flags that are not used. * Improving structure data listing. Starting structure data tests * More progress on verdi data tests * Structure listing test added * More progress on TrajectoryData list testing * Finished with array and bands tests * Add tests for verdi data parameter * some progress on verdi data remote tests * finish tests for verdi data remote * Trajectory data listing test OK * Verdi band listing tested more extensively, uses common listing arguments and has proper group support * verdi data cif list test uses also the common method for testing * Minor beautification of the code * finish with tests for verdi data upf * Remove --elements options where not needed * verdi data export tests for structure and trajectory too * More on verdi data trajectory export tests * verdi data cif export uses common testing mechanism with structure and trajectory * Add checks to verdi data cif,structure,trajectory * Remove redundant tests * Remove redundant test from the list of backend tests * Solve formatting problems * Adapt array.py and bands.py to python convention * Adapt cif.py to python format convention * Adapt deposit.py to python format convention * Adapt export.py to python format convention * Adapt list.py to python format convention * Adapt parameter.py to python format convention * Adapt remote.py to python format convention * Adapt show.py to python format convention * Adapt structure.py to python format convention * Adapt trajectory.py to python format convention * Adapt upf.py to python format convention * format changes at some random places * Fix remaining format changes
Removed several commands that were outdated and no longer in use * verdi devel query * verdi devel getresults The former was still under development and hardcoded Django queries and so was not usable across all backends. The latter was also in development stage and has now been replaced by the more apt `verdi calculation res` command.
) Updated the list of available commands and sub commands to recent changes. Also moved the location to `Working with AiiDA` and prefaced the list of commands with an introduction to general concepts that apply to the entire cli: * Structure and meaning of help strings * Parameters: difference between options and arguments * Profile: how default can be switched temporarily * Identifiers: logic of how identifier type is deduced and ambiguity caveats * Multi value options: peculiarities of greedy options and positional arguments
* commands/data: delay load_dbenv further
* commands/data/bands: use `finally` to close files
* commands/data: add `plugins` subcommand
* commands/data: rename `list.py` to avoid type collision
Otherwise even a specific import like
from aiida.cmdline.commands.data.list import _list
will then overload the type `list` to be the module list instead of the
type. So, expressions like `list([])` in the same scope (or below)
as the import will generate an error stating that a module is not callable.
Also rename the `export.py` for the same reason and to be consistent.
Furthermore, this way it is also clear that those files do not contain
standalone commands but rather helper functions.
* cmdline/utils/pluginable: fix list_commands
* commands/data: move sub-command import
We must ensure that the commands are defined at the point Pluginable's
`list_commands` is called, so they must be imported in global scope.
Since we delayed the `load_dbenv` further this is not a problem anymore.
* cmdline/baseclass: pass along to click in run()
For verdi subcommands this now runs the corresponding click command
instead of going through the list of available_subcommands. Via the
`Pluginable` class we therefore get proper plugin support until we can
drop the `VerdiSubcommand` class entirely.
* commands/data/cif: improve command description
Changes: * each transport now has a subcommand to configure Additions: * non-interactive: not given options are accepted with defaults (re-running changes only given options) * interactive: still ask for all keys, with defaults * new verdi computer configure show <computer> - shows current or default config - outputs in human readable or option string that can be passed for default config No-fixes: * transport/plugins/ssh.py should be added to pre-commit and fixed in another PR, in order not to block verdi click migration
Some of the `verdi` commands, such as `import` and `code` have names that clash with reserved python keywords or other often occurring variable names. To prevent import clashes, rename these files to be prefixed with `cmd_` to clearly distinguish them
Merge develop into verdi
Now that all `verdi` commands have been migrated to use `click`, the homebrew cli infrastructure can be ripped out and the entire stack is now powered by `click`. There were three commands, `setup` `quicksetup` and `run` that did not yet have their own command. They have been fully clickerized and hooked up now.
sphuber
changed the title
verdi to the click infrastructureverdi to the click infrastructure
DropD
approved these changes
Jul 25, 2018
This was referenced Jul 25, 2018
Closed
Codecov Report
@@ Coverage Diff @@
## develop #1795 +/- ##
===========================================
+ Coverage 57.12% 66.59% +9.46%
===========================================
Files 275 314 +39
Lines 33923 32318 -1605
===========================================
+ Hits 19379 21521 +2142
+ Misses 14544 10797 -3747
Continue to review full report at Codecov.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1697 #1638 #1092 #1483
This merge takes care of migrating all
verdicommands to use theclickinfrastructure. This renders the old home made cli library obsolete and it has been removed