Improve LSP support

[mattrunyon]

As a user, I want better LSP support so that the web IDE provides more niceties that I use in my desktop IDE.

We currently have support for Completion Request, but we could add support for several other items with Jedi. This list will be in the order that I find most useful/impactful, but all of them enhance the user experience.

• Completion request - We support this, but could improve to also include
  • CompletionItemKind - use LSP enum and then we will convert to monaco enum on client b/c monaco uses a different order for some reason
  • documentation in the response provides an arrow the user can click to see more docs. Or we can use this hack to show the documentation always. VSCode shows this documentation by default.
  • detail - this is often used to show the package or some potential item in the autocomplete item on the right side. VSCode for web stuff shows the package name if it is something that will be auto imported on autocomplete. Pycharm has some info about the package the item belongs to regardless of if it is imported already
  • deprecated - Not sure if we're marking anything as deprecated in the Python code, but would be a nice-to-have in the future if we start deprecating APIs. Shows the autocomplete item with strikethrough to indicate deprecation
• Signature help - Provides docstring and param info when you start calling a function
• Hover request - Provides docstring and other relevant info when hovering over a symbol
• LSP Diagnostics #3614
• Document formatting request - Would be nice to have the code follow some pep8 formatter when possible. The user should be able to disable formatting on the client. This would trigger formatting on save
• Document on Type Formatting Request - Auto indent is usually ok, but this might provide some more info
• Rename request - Useful for renaming a variable throughout a document

I'm sure there are others that would improve the user experience. The first 2 are major improvements IMO while the others are nice-to-haves.

[mofojed]

Much of this was completed with #3607

[mofojed]

We should also autocomplete the column name in .where clauses as well, to help people writing those filters. It should just suggest the column name, and not append any = or anything like that.

[dsmmcken]

https://github.com/astral-sh/ruff-lsp

We should get the rest of the LSP features like formatting via this ^

[devinrsmith]

Using ruff may depend on astral-sh/ruff#659, at least with regards to how we are using jedi today.

[devinrsmith]

Another option - looks like https://play.ruff.rs is WASM and accomplished completely client side; awesome! If we could move this logic completely to the client, that would be a big win IMO.

[dsmmcken]

I was looking at the source of this, since ruff can be WASM compiled... It actually looks like it would be relatively easy to get a python linter + a python formatter integrated client side. They integrated the formatter not how you actually would do it because they wanted it side by side. We should do it on the registerDocumentFormattingRequest or whatever monaco event.

https://github.com/astral-sh/ruff/blob/main/playground/src/Editor/SourceEditor.tsx
https://github.com/astral-sh/ruff/blob/09296e3e3cf18eeea1b6f6391fb43e50e49af00a/playground/src/Editor/Editor.tsx#L131

No official ruff wasm npm package yet, but there is a issue in the ruff repo to start publishing one. There is a community maintained package https://www.npmjs.com/package/@wasm-fmt/ruff_fmt and we could always build our own if necessary too. Maybe some complexity with the vite deploy of that, but that package has a vite example.

Anyways, this looks cheaper to implement then I expected, so in terms of effort vs payoff, I think it's actually pretty high.

Something to consider is how we handle the lint/formatting config. Is it a file in community? Is it a settings menu thing? What about in enterprise, is it server prop provided config? Can users override it?