Works so slow when I try to paste long string to terminal

[haoranpb]

I am just using main highlighter. At first, I thought if I just disable some styles, it will work fine. So I disabled all quoted-argument.

When I paste magnet:?xt=..t.nz after " into the terminal:

aria2c "magnet:?xt=urn:btih:90...er.fastcast.nz"

The highlight is gone, but it's still really slow!
I really love the highlighting, how can I make it faster?

[phy1729]

(Related to #240.)

Being slow with large inputs is a known issue. There seems to be two main causes:

1. Iteration over a string is quadratic.
2. Array appends are quadratic.

I've briefly looked at solving the first in zsh, but I haven't had time to make a complete patch.

With regards to disabling styles, setting a style to none doesn't disable the code that parses for that style. We could add a flag to disable _zsh_highlight_main_highlighter_highlight_argument, but that would disable

• single-hyphen-option
• double-hyphen-option
• history-expansion (sometimes)
• globbing
• path (and related styles)
• single-quoted-argument (and -unclosed)
• rc-quote
• double-quoted-argument (and -unclosed)
• dollar-double-quoted-argument
• back-double-quoted-argument
• dollar-quoted-argument (and -unclosed)
• back-dollar-quoted-argument
• back-quoted-argument (and -unclosed)

and once #512 is merged the highlighting of command and process substitution and their contents.

If that's acceptable, you can add return 0 as the first command in _zsh_highlight_main_highlighter_highlight_argument. We could add a flag to do the same, but I think long term it'd be preferable to fix the quadratic issues in zsh.

[haoranpb]

Thx a lot!

I didn't really find a func named _zsh_highlight_main_highlighter_highlight_argument, but I think you mean add return 0 in front of line 652 in main-highlighter.zsh

It did work a little faster

Tip in #86 is a good idea, I think I will just disable highlighting when I have a very large input

[phy1729]

Ah forgot the current release doesn't have that function. Where you added it will do similar, but may mess up the state in some cases. Don't think there's a single line change that'd work in the current release.

There's also the possibility to replace some loops with an while i=$arg[(ib:... style loop as in https://github.com/phy1729/zsh-syntax-highlighting/tree/ib-subscript . Haven't had much chance to test it, but I think it improved some cases and worsened others. It would be helpful to have a performance test suite to more rigorously test each change.

[haoranpb]

I viewed #86 and some of the source code again, came up with a temporary solution:

Add export ZSH_HIGHLIGHT_MAXLENGTH=60 in file .zshrc

Now it will just stop parse or highlight any long string, and this works well for me.

Thanks for your help!!

[danielshahaf]

@phy1729 Do you recall that make perf exists? I realise it has room for improvement.

Shall we open separate tickets for the two "X is quadratic" issues you identify?

[phy1729]

The issue with make perf is that it reuses the test suite data which is probably not reflective of a user's typical buffer. The performance for single quoted, double quoted, and non-quoted strings are all different. Depending on what mix is typical code changes may be performance gains or losses. E.g. changing the bracket highlighter's loop to
while pos=$BUFFER[(ib:pos+1:)[\{\}\(\)\[\]]]; (( pos <= buflen )); do
was faster if every third character or fewer was a bracket but slower if every character is a bracket (an unlikely case). For the main highlighter the change made it slower to highlight the master version of main-highlighter.zsh. I don't know if that result has any bearing on a typical user's experience.

Array appends are quadratic are discussed in #388. We could open an issue for string iteration being quadratic.

[Yriuns]

I encounter the similar issue when I copy curl command with long headers. I wonder if it is possible to disable the zsh-syntac-highlighting for specific prefix (like curl)?

[danielshahaf]

I'd recommend just using ZSH_HIGHLIGHT_MAXLENGTH as @ludanxer suggested. Alternatively, you could patch that part of z-sy-h's source to do something like if [[ $PREBUFFER == *curl* || $BUFFER == *curl* ]]; then return; fi.

[Yriuns]

Thanks I lot. I think I will use the former one.

[flakrat]

Thanks for sharing the ZSH_HIGHLIGHT_MAXLENGTH variable. Pasting blocks of script to the terminal was so slow I was about to disable the plugin.

[danielshahaf]

There are two separate possible causes to slowness:

1. Slowness could be due to the length of the buffer. If you try to fned a function at the prompt, for example, you'll see this, even though that involves no pasting. That's tracked as Slow with large inputs #240.

2. Slowness could be caused by z-sy-h being invoked repeatedly for each character in the paste, one at a time. This can happen on terminals that don't supported bracketed paste. For example, on my terminal, if I type in foo and paste bar then z-sy-h gets invoked four times: once each on f, fo, foo, and foobar; however, if I set zle_bracketed_paste[1]=$zle_bracketed_paste[2] first, then z-sy-h gets invoked on the intermediate states foob and fooba as well. If that's what you're running into, try enabling bracketed pasting in your terminal emulator.

If someone observes slowness that doesn't fall into one of these two buckets, please let us know. (And, naturally, help fixing #240 would be welcome, if anyone's interested.)

[chenshengzhi]

this gist works fine.

[danielshahaf]

Thanks!

Does the feature/redrawhook branch have this problem too? If it does, we should look into what bracketed-paste-magic does that makes a difference.

[forresthopkinsa]

In my experience, if you paste something into the terminal and it's taking a long time to draw, you can push any button on the keyboard (e.g. right arrow) and it'll give up highlighting and draw the text.

[Zedai00]

forresthopkinsa that worked thanks

[matthinea]

Another hack: don't paste heavily indented text. IME pasting resumes at normal speeds on newlines that have no indentation.

Worked better than any of the above for me.

[danielshahaf]

@matthinea Hi, sorry for the delay. Thanks for the lead. Could you please show the output of typeset -p ZSH_VERSION ZSH_PATCHLEVEL ZSH_HIGHLIGHT_REVISION ZSH_HIGHLIGHT_VERSION? I wonder if z-sy-h (or possibly zsh's ${(z)} implementation) should learn to optimize runs of whitespace.

[matthinea]

typeset ZSH_VERSION=5.3
typeset ZSH_PATCHLEVEL=zsh-5.3-0-g4cfdbdb

(I stopped using zsh-highlight-version when I thought it was slowing down pasting)

[danielshahaf]

``` typeset ZSH_VERSION=5.3 typeset ZSH_PATCHLEVEL=zsh-5.3-0-g4cfdbdb ``` (I stopped using zsh-highlight-version when I thought it was slowing down pasting)

Then cat the files `.version` and `.revision-hash` in the same directory as `zsh-syntax-highlighting.zsh`.

[smac89]

This comment fixed it for me: #295 (comment) and the issue is probably a duplicate of this one

[shqear93]

this should be added to the readme file or installation document

[Frederick888]

Does #835 fix both this and #295?

I added the gist in above comment to my .zshrc over two years ago. I just commented it out and tested it again. It's slightly slower but barely noticeable.

zstyle ':bracketed-paste-magic' active-widgets '.self-*' gave me about the same speed as the gist.

[danielshahaf]

Does #835 fix both this and #295?

Thanks for asking. It's hard to say, since this issue covers so many bases:

• zsh quadratic behaviour on indexing arrays [performance: Appends to region_highlight are quadratic? #388]
• zsh quadratic behaviour on indexing scalars [now spun off as zsh string iteration/indexing is quadratic #909]
• ZSH_HIGHLIGHT_MAXLENGTH [documented in docs/highlighters.md, so no further action needed]
• slowness on large buffers (whether or not pasted) [possibly resolved by Use zle-line-pre-redraw (the feature/redrawhook branch) #749]
• slowness specific to the pre-redrawhook codepath [resolved by Use zle-line-pre-redraw (the feature/redrawhook branch) #749]
• slowness related to bracketed-paste-magic [reported resolved]

Also, everyone might be using different versions of zsh and this plugin and other plugins.

At this point, I'll go ahead and close this issue. Please do not continue discussion on this ticket; open a new ticket for anything that remains.