Rewrite command line parsing, add flags and expansions

[the-mikedavis]

This change rewrites the command line parsing code (discarding shellwords) and adds command flags and expansions. Unfortunately these features are not really independent - each feature influences the design of the parsing and completion code - so I think it's best to add them all together.

The syntax and design are roughly the same as what Kakoune is doing but with a few twists:

• %sh{..} expansions are recursive, enabling variables within shell commands like :noop %sh{gh browse "%{buffer_name}:%{cursor_line}" -c=%sh{git rev-parse HEAD}} for example from Command expansion v2 #11164 (comment). In contrast, Kakoune exposes $kak_<val_name> environment variables to the shell program.
• A command's signature can be written to treat the input as quoted after a given number of arguments. This is used now in :toggle-option for example to support toggling objects and arrays like :toggle rulers [10, 20] [30, 40]: after the first arg everything is parsed as a JSON stream instead.

The documentation in book/src/command-line.md explains the quoting rules and how to use flags and expansions.

Completion in the command mode prompt supports expansion names, variable names, and flags and it can be extended in the future for new expansions.

This PR focuses on the parsing and completion changes so it only adds one flag and a handful of variables. Adding new flags, variables and expansions should be quite minimal. Here are some good example changes:

• Add a --no-format flag to write commands: the-mikedavis@40f8da3
• Add a new variable %{theme}: the-mikedavis@967e237
• Add a %config{key} expansion: the-mikedavis@dc57b11

The user-facing changes are:

• :rsort is removed, replaced with :sort -r/:sort --reverse. (breaking)
• Shell commands like :sh or :pipe evaluate expansions but do not handle quotes. Aside from expansions the command line text is passed to the shell program as-is. (fixes sh command doesn't handle arguments with spaces #7453, fixes Remove escaping on shell commands  #10549)
• :set-option and :toggle-option now interpret the first argument according to normal rules but then use custom parsing for the rest of the input. :toggle-option can now toggle object and array values like rulers. (fixes Line numbers are now configurable via the gutters setting but apparently can't be changed with the :set command #2206, fixes :set-option cannot be used for structured options with spacing. #6171)
• In a similar fashion, :lsp-workspace-command now accepts JSON arguments (fixes Support lsp_workspace_commands with arguments #11079)
• New commands :echo and :noop. (:noop enables a workflow like Adds a run-shell-command-no-output command #3247)
• Some buggy completions on Unix have been resolved. (For example try :open a\<tab> in a directory with a file a b.txt on Unix.)

Review notes...

A good entrypoint looking at the new code is Args::parse. This is used when executing a command in helix_term::commands::typed::execute_command. Args reads Tokens from the Tokenizer, according to a typable command's declared Signature, expanding them when you hit <ret> in command mode. Signature tells Args how to parse the input: the range of positionals that should be expected, the command's flags and, optionally, after how many positionals the input should be consumed for further parsing within the command (Signature.raw_after, Tokenizer::rest).

The completion code in complete_command_args (typed.rs) is also worth a look to see how the Token type is reused outside of Args::parse.

Tokenizer closely follows Kakoune's CommandParser while Args somewhat follows Kakoune's ParametersParser.

shellwords has been discarded: the escaping rules were not very intuitive - generally backslash should be a convenience available to Unix and not at all required.

Fixes #5828
Fixes #10993
Closes #11164
Closes #12075
Closes #12624

[david-crespo]

Exciting PR. These are working for me on macOS:

C-b = ":sh gh browse %{buffer_name}:%{cursor_line} -c=%sh{latest_pushed_commit}"
C-B = ":echo %sh{git blame --date=short -L %{cursor_line},+1 %{buffer_name} | sed -E 's/[0-9]+).*//' | sed 's/(//g'}"

[nik-rev]

Some minor ideas here

[nik-rev]

When a variable is completed, we are left in this state:

:echo %{buffer_name

What would be nice is if it automatically inserted the}. This is a minor thing but It would slightly improve UX

[the-mikedavis]

Ideally that would be covered by auto-pairs within the command line but that will take further changes to how completions work in the Prompt component

[haras-unicorn]

thank you so much for this one! been waiting for quite a while for it :)

[rcorre]

I'm not sure if this is a bug, or working as intended, but it seems that noop waits synchronously for the command to complete, whereas :sh runs a command without blocking the editor.

[keys.normal.g]
i = ":echo %sh{gh browse '%{buffer_name}:%{cursor_line}' -c}"
b = ":noop %sh{gh browse '%{buffer_name}:%{cursor_line}' -c}"
B = ":sh gh browse '%{buffer_name}:%{cursor_line}' -c"

Both gi and gb cause helix to briefly hang while the command executes. gB doesn't block the editor, but shows the output in a hover window. So I'm not sure :noop quite replaces the need for a :run-shell-command-no-output as-is.

However, :sh gh browse '%{buffer_name}:%{cursor_line}' -c &>/dev/null works fine to run a command without blocking the editor and without showing output. So maybe there's not a missing feature here, but I'm not sure when I'd use :noop.

P.S. thank you for this feature, it's great!

[david-crespo]

@rcorre I'm confused about the output thing — gh browse doesn't have any output for me in helix — there is no popup even without the pipe to /dev/null. When I run it in the terminal, I see the Opening <url> in your browser message, and I confirmed it's going to stdout and not stderr. But I don't see it in helix when I run the command with :sh.

[rcorre]

@david-crespo sorry for not clarifying -- the output is coming from my browser itself. I use qutebrowser, which prints "opening in existing instance". Though you're right, gh also has output and I'm not sure why that doesn't show up.