Sorted completions, improve completions marking and filtering and doc panel

[krassowski]

• implements sorting of completions solving point 4 of Completion improvements #495 by taking use of sortText if provided (jets tested and indirectly integration-tested)
• makes use of filterText if provided (jest tested)
• allows user to choose whether in presence of ties (i.e. two suggestions having the same match score) the kernel suggestions should go first or last (the latter by default) (integration tested)
• ensures that a full copy of item object gets created during filtering so that the documentation can still be retrieved dynamically (adds integration test for that)
• always re-renders the docpanel so that the markdown is always rendered (while waiting for Move documentation panel rendering to renderer jupyterlab/jupyterlab#9663 to be merged)
• allows to render labels that contain HTML tags (HTML/XML markup is removed from the completer label jupyterlab/jupyterlab#9790)

Code changes

• Use a custom completer model (LSPCompleterModel) and create a generic template GenericCompleterModel which will make the live much easier once upstreamed for 4.0 (Completer widget public API refactor jupyterlab/jupyterlab#9763); at this point it is experimental API to be upstreamed so if anyone depending on our codebase looks at it - please do not rely on this API just yet.
• adds a second pass of matchSumOfSquares to allow for matching on filterText which does not have to be a substring of label; there is a price to pay for the second match calculation on numpy completions, but the total cost is over an order of magnitude lower of the current limiting factor of SVG rendering and item rendering in general.

User-facing changes

TBD. So many nice things :)

Backwards-incompatible changes

• moved LazyCompletionItem to a file of its own (item.ts); better to make this change early
• added LSPCompleterModel

Chores

• linted
• tested
• documented
• changelog entry

[bollwyvl]

Awesome stuff. The sorting of search results is hilariously hard, getting mired in the whole field-based boosting, etc. but this is a very good step.

[bollwyvl]

It's probably worth a look through @lumino/algorithm, there are a number of (Big O-annotated) implementations that might be helpful, and look more like core.

[krassowski]

Yes, the code related to sorting is taken from core and changes are meant to be up-streamed for 4.0. I also modified the pyls sorting on my fork, please feel welcome to chime in and comment it: https://github.com/krassowski/python-language-server/pull/1/files#diff-66c341f45ce292941d6a15c7e287ce5faf7f7e7d10f35bb5919722995bd982c9R9-R36

As pyls is in a weird state where the spyder folks who were maintaining it (even though it was maintaining in a literal sense of literal life support and not much development) no longer have merge rights (which might take months to rectify) we should think about some solution. One solution would be a community fork, but lets see how things play out.

While my fork is not yet ready for general use (no CI and infrastructure set up) and I do not like the legacy baggage from pyls (like Python 2), it might be possible to refactor it to something usable and keep it compatible with community extensions.

An alternative would be to default to jedi-langauge-server as it is being still maintained and developed; it is nice because of the pygls but not nice because of a strictly limited scope. If it had the plugins/entry points mechanism as pyls it would be so great! (See pappasam/jedi-language-server#32).

A third alternative is using pyright. It is node-based (and MS) and controversial due to how pylance is being licensed.

Just mentioning this here, but it will probably deserve its own issue soon.

[bollwyvl]

Yeah, it's hard to have nice things, great work you're doing.

This is probably another JEP, not just a discussion on our repo. I feel like a number of things could happen:

• hard fork of pyls, i wouldn't hold my breath on the current copyright holders clearing anything up
  • despite the baggage, we know how it works, and the plugins work
  • would also require forking the plugins, or convincing them to migrate along with us
• get a robust plugin mechanism into pygls itself so that jedi-language-server wouldn't even know it was being extended 👿
  • the pygls pattern is a little weird, but actually pretty good for this
  • the pluggy route that pytest, etc. uses would actually be superb for this
• maybe with the above, create and maintain Yet Another Python Language Server
  • but demonstrate solid integration with the python kernels (and their derivative wrapper kernels), notebooks, etc.

I don't really see myself wanting to rely on any of the first-party MS stuff (especially if it's future-abandonware, and my feelings on requiring node to do python work are likely clear at this point), if anything they'll keep building their walled garden to get a slice of that sweet, sweet data science/ML mindshare and money.