Add new completion API

[predragnikolic]

NEW:

• completion kind symbols in autocomplete (symbols will probably be a subject of change)
  

• support deprecated flag in completion items(don't know any server that implements this, but here is an image of a deprecated completion item)

• this PR also fixes Incorrect additionalTextEdit applied when auto-importing PHP namespaces #757, because we now know exactly what completion item was chosen

I haven't been able to run any completion unit test, just to tell you.

There are some things in this PR, that I wish I had done better, but I was not able to(due to my limited knowledge, I will put some comments to mark those things)

[predragnikolic]

TODO:
remove completion_hint_type because it is no longer used
Edit: done

[predragnikolic]

Feel free to correct me, I don't know if I used the new API correctly :)

Here is my feedback:
I want to highlight the differences between the new api, and the code before it:

The code before it did this:

1. It copied the row, text, cursor position of each line, right before sending the request.
2. On selecting a completion item, it inserted back the text on each row.
   END RESULT: It brought the lines in a state like they were when the completions were requested.

The code in this commit does this.

1. It copies the change_id, change_region(the region passed to the transform_region_from), text and cursor position for each line, right before sending the request.
2. On selecting a completion item, the transform_region_from returns a transform_region. transform_region is used for removing and inserting back the previous text for each line.

 view.erase(edit, transform_region)
 view.insert(edit, transform_region.begin(), saved_line['text'])

// I could probably use view.replace instead of erase and insert

END RESULT: It brings the lines in a state like they were when the completions were requested.

You may have noticed that I did this a bit differently, than what John did in his gist.
https://gist.github.com/jskinner/4dfa6d86f848e1e1e583883507369b67#file-async_edits-py-L238-L244

    def fill_completions(self, edits):
        # The server has sent us a series of edits it wants to be made when
        # the completion command is run, however the user may have edited the
        # buffer since the request want sent. Adjust the response from the
        # server to account for these edits
        for c in self.changes_since_sent:
            edits = [(transform_region(r, c), text) for r, text in edits]

If I done the same thing as John did in his gist:
I would have to adjust the response from the server to account for all view edits that occurred,
and apply all TextChange-s to all completion items.
But that to me looks like a lot of work.

For me it looks easier to adjust the view state to the state it was when the request is send,
than to adjust the response from the server to account all the possible edits that may have had occurred.

Either way, I really want to hear the feedback from you all
And don't take me wrong with what I just said :)

[FichteFoll]

This is probably off topic for this ticket, so I'm more than happy to talk further about this elsewhere, but having " as a trigger character without a scope limiting it to punctuation.definition.string.begin is bananas. Personally, I'd say " must not trigger AC in comments, in strings (e.g., when the user types an escaped quote), or for a quote at the end of a string.

@jskinner, just for confirmation, the auto_complete_triggers selector is matched against the inserted char after it has been inserted? Because that would be different from the auto_complete_selector setting, which makes AC appear if the line break character at the end of the comment is not included in the comment scope.

[deathaxe]

Anything which prevents the line to be stored directly? Doing so would avoid converting region->tuple->region.

[FichteFoll]

Regions can't be serialized for transmission in run_command, which is required for the commit completion command.

However, you should be able to use tuple(region) now that regions are iterable. I also think this step should happen at the communication edge were it is necessary to do so and not for the internal storage to make this kind of relation more obvious. I'm still not seing where the use case I assumed actually occurs.

[deathaxe]

Guess the python 3.3 version of sublime.Region does not support iteration. This will be available as soon as we can move to python 3.8

[predragnikolic]

I didn't know you could do that tuple(region) with a Region. Cool :)

[predragnikolic]

'Region' object is not iterable.

    saved_line['change_region'] = tuple(saved_line['change_region'])
TypeError: 'Region' object is not iterable

Looks like the LSP plugin is still using python 3.3.

[rwols]

Some problems:

When I supplement the completion by typing more characters, the completion widget goes away.

---

I don't get a completion widget at all sometimes. In the gif below I expect to at least see a completion for the "time" variable:

I'm guessing this is due to incomplete completions. I'm not sure what the dynamic is here.

[jskinner]

@FichteFoll auto_complete_triggers is tested against the character just inserted, and auto_complete_selector is tested at the position before the inserted character (a change from ST3), but after it has been inserted

[rchl]

Noticed a bug with latest changes:

When, in Python, I'm typing False and getting an (only) suggestion with it, the suggestion popup doesn't disappear once I've typed whole word. It did before those changes.

[predragnikolic]

It did before those changes.

@rchl are you sure?
I just installed build 4068 and checkout commit(the one before adding the new API) b5c0de6

The autocomplete doesn't disapear there too when I finish typing False

[rchl]

Sorry, I didn't mean to say the latest changes specifically, just that I've noticed it with the latest changes included. :)

I've compared to master branch (7fb166d).

[deathaxe]

See the bug, too, using the LSP-json completions.

With regards to @jskinner's statements the triggers and selectors of the view are set to:

>>> view.settings().get("auto_complete_selector")
'- string - comment - constant | meta.main-key | meta.rule-key | meta.interpolation-key | meta.animation-key | meta.attributes-sequence - comment | meta.class-selector | meta.function-call.var meta.group'
>>> view.settings().get("auto_complete_character")
Unable to open /C/Apps/Sublime Text/Data/Packages/Default/Preferences.sublime-settings
>>> view.settings().get("auto_complete_triggers")
[{'characters': '<', 'selector': 'text.html'}, {'selector': 'punctuation.accessor'}, {'characters': '":', 'selector': 'source.json'}]

The string scope is explicitly excluded from completitions. The trigger ": is defined.

It looks like ST only uses the first character of the trigger, thus retriggering AC after committing a value which ends with ". Same happens after completing the "class" or just removing and retyping the final ". Both the json-key and json-value are string.quoted.double.

Not sure whether the trigger is intended to be of one character only by design or whether it is a bug.

[predragnikolic]

@rwols #866 (comment)
The LSP plugin doesn't trigger or hide the autocomplete programmatically, anymore.
It just registers the autocomplete triggers and then let sublime decide whether to show the AC or not.
I could try to reproduce it with a language server that returns incomplete completions items (LSP-intelephense supports that).

I want to say that this PR is getting quiet large :)

@rwols @rchl If you guys are happy with the changes till now,
can we merge this and create separate issues that should be addressed (like for the autocomplete triggers "forever" and other things) :)

[rwols]

Does tests/test_completion.py not crash for you? It segfaults SublimeText for me.

[rwols]

I'm okay with merging to st4000-exploration branch provided that the tests pass.

Outstanding issues:

• commit_completion infinite loop (Add new completion API #866 (comment))
• When, in Python, I'm typing False and getting an (only) suggestion with it, the suggestion popup doesn't disappear once I've typed whole word. It did before those changes. (Add new completion API #866 (comment)) (maybe fixed?)
• Review auto_complete_selector and auto_complete_triggers (Add new completion API #866 (comment))
• I'm not convinced the restore_lines.py file is needed now that we have view.transform_from_region. Think about how to use that method.
• When I supplement the completion by typing more characters, the completion widget goes away. (Add new completion API #866 (comment))
• I don't get a completion widget at all sometimes. In the gif below I expect to at least see a completion for the "time" variable (Add new completion API #866 (comment))

[predragnikolic]

No, they run fine:

----------------------------------------------------------------------
Ran 19 tests in 5.981s

OK

[rwols]

Strange, ST4 segfaults here when running UnitTesting: Test Current File 😕

[rwols]

OK, when I run all tests with UnitTesting: Test Current Package then ST4 doesn't segfault and all tests pass, so I'm okay with merging this :)

[rchl]

I won't claim that I've reviewed it but at least with my testing, I could not find more issues (than the one still remaining)

[predragnikolic]

Thanks for the response.
For me, it looks easier to split each issue,
and discuss each separately,
and make smaller PR that will fix those.
Than to do it all in this PR here :)

[predragnikolic]

@rwols I created issues for some of the things you wrote, but skipped these ones because I don't have enough information on what exactly needs to be done:

• Review auto_complete_selector and auto_complete_triggers
• I'm not convinced the restore_lines.py file is needed now that we have view.transform_from_region. Think about how to use that method.

@deathaxe if you don't have nothing against,
can you continue the topic you started about triggers and selectors in #933 :)