Bash Programmable Completion for Spack

[adamjstewart]

Do you have trouble remembering the ordering for spack config edit compilers? Do you have to run spack list 'py-*' before installing any Python package because you can't remember the spelling of py-mysqldb1? Don't you wish you could just tab-to-complete everything? Allow me to introduce Bash Programmable Completion for Spack!

To add tab completion, simply source setup-env.sh from a Bash shell. Here is an example session. In the following examples, let the cursor be denoted by brackets, i.e. [], and assume that the user presses TABTAB at the cursor position:

$ spack []
activate      create        find          mirror        repo          url-parse
arch          deactivate    graph         module        restage       urls
bootstrap     dependents    help          package-list  spec          use
cd            diy           info          patch         stage         versions
checksum      doc           install       pkg           test          
clean         edit          list          providers     test-install  
compiler      env           load          purge         uninstall     
compilers     extensions    location      python        unload        
config        fetch         md5           reindex       unuse         
$ spack con[]
$ spack config []
edit  get   
$ spack config e[]
$ spack config edit []
compilers  mirrors    
$ spack config edit c[]
$ spack config edit compilers  # the full command

$ spack in[]
info     install  
$ spack ins[]
$ spack install []
Display all 320 possibilities? (y or n)  # lists all 320 available packages
$ spack install py-[]
py-astropy           py-mako              py-pmw               py-scikit-learn
py-basemap           py-matplotlib        py-pychecker         py-scipy
py-biopython         py-mock              py-pycparser         py-setuptools
py-blessings         py-mpi4py            py-pyelftools        py-shiboken
py-cffi              py-mx                py-pygments          py-sip
py-coverage          py-mysqldb1          py-pylint            py-six
py-cython            py-nose              py-pypar             py-sphinx
py-dateutil          py-numexpr           py-pyparsing         py-sympy
py-epydoc            py-numpy             py-pyqt              py-tappy
py-funcsigs          py-pandas            py-pyside            py-twisted
py-genders           py-pbr               py-pytables          py-urwid
py-gnuplot           py-periodictable     py-python-daemon     py-virtualenv
py-h5py              py-pexpect           py-pytz              py-wheel
py-ipython           py-phonopy           py-pyyaml            py-yapf
py-libxml2           py-pil               py-rpy2              
py-lockfile          py-pillow            py-scientificpython  
$ spack install py-numpy []
$ spack install --[] py-numpy  # whoops, forgot to add an option
--fake                 --ignore-dependencies  --keep-prefix          --no-checksum
--help                 --jobs                 --keep-stage           --verbose
$ spack install --v[] py-numpy
$ spack install --verbose[] py-numpy  # the complete command

[adamjstewart]

Let me know what you think about this particular implementation. There is still a lot of testing to do, and I'm not familiar with all of the Spack subcommands, so you'll have to help me out. One thing I would like to be able to do would be something like:

$ spack install hdf5 %[]
%gcc@4.4.7     %gcc@5.3.0     %intel@16.0.1  %nag@6.0       %pgi@15.10-0   
$ spack install hdf5 %gcc[]
%gcc@4.4.7     %gcc@5.3.0
$ spack install hdf5 %gcc@5.3.0 +[]
+cxx            +debug           +fortran     +mpi     +shared     +szip
+threadsafe     +unsupported
$ spack install hdf5 %gcc@5.3.0 +szip  # the full command

My current implementation strips out the flags and uses the remaining words to call a function with the same name (for example, _spack_install). But this breaks down when you want to go further because _spack_install_hdf5 does not exist. It also involves a lot of repeated code, since half of the subcommands should complete to available packages anyway.

Another idea I had, which I'm starting to realize might be a better solution, would be to use a single colossal case statement. That way I could group together all of the commands that should complete to a package spec. Thoughts on this method over the current method?

I don't use zsh, but if there is enough interest, I'm sure we can get this working for zsh too.

[adamjstewart]

test_vars and pretty_print are just testing functions. I will remove them once we decide on a final solution and they are no longer needed.

[tgamblin]

@adamjstewart: this is awesome! Thanks for doing it.

My worries about the current implementation are:

1. Maintenance. It looks like this stuff needs to be updated whenever the commands are updated, and I'm not sure how to enforce that through testing. We add commands frequently enough that it could get to be a pain.
2. Parsing. The bug you pointed out seems like it really needs spack to run a command to do the spec parsing and to offer completions. The way I would do that would be to modify the actual spec parser to provide suggestions. Spack could run a command that tells bash the possible completion, kind of like how spack cd uses spack location to figure out where to cd.

(2) kind of points to the need for a more context-sensitive spack parser. I had thought about making Spack's parser a bit smarter about distinguishing negative variants (-foo) from package names, since - is a legal part of a package. That may be more work than you want to take on for an initial implementation but I think it's what you have to, or you'll end up implementing a duplicate parser that needs to be maintained in bash.

Would it be better to auto-generate some or most the bash completion code? I think that would generally work better and would ensure that the completion stuff doesn't get out of sync with the rest of spack.

[adamjstewart]

@tgamblin: Those are valid concerns. When I was first writing this, I thought about modifying each subcommand to have an option to print all options, or to somehow import each subcommand package into a python one-liner and print out the arguments in subparser, but decided that it would be a lot of work for a first implementation. Yes, we are currently adding a lot of new options and commands, but as Spack matures, this should taper off. If you can think of an easier way to get Spack to provide suggestions, let me know.

As for using the Spack spec parser to provide suggestions, that would be nice. But I'm not sure where to get started.

Do you have any suggestions as to where I should take this next?

[tgamblin]

@adamjstewart: have you seen argcomplete? I was eyeing it as a first-cut solution for this a while ago.

I think it would allow you to generate most of the boilerplate stuff, and it also provides ways to have the python code suggest completions in certain contexts. I would be curious whether it could cover the cases you've covered so far, and we could then work together to see if it could suggest completions for the spec parser. It would also remove the burden of maintaining completion code :) Want to check it out?

[adamjstewart]

I'll check it out.

[adamjstewart]

From what I can tell, pretty much every file in lib/spack/spack/cmd would need to import argcomplete:

#!/usr/bin/env python
# PYTHON_ARGCOMPLETE_OK
import argcomplete, argparse
parser = argparse.ArgumentParser()
...
argcomplete.autocomplete(parser)
args = parser.parse_args()

This would make argcomplete a required dependency of Spack. Other than that hurdle, it seems like argcomplete would let us do exactly what we want. Thoughts?

[tgamblin]

Assuming argcomplete works well, I'm completely unopposed to adding it to lib/spack/external!

Do you need to add it to each command in cmd, or can it be added to the main parser in bin/spack? That's where the main parser setup is. Could you just add it there or does it have to be added to each sub-parser individually?

[adamjstewart]

Maybe? I'll have to try it out.