Exception when imports are unsorted

[svenpanne]

python-lsp-ruff generates an exception "cattrs.errors.ClassValidationError: While structuring Fix (3 sub-exceptions)" (see pylsp_stderr.log) in the following simple scenario: Start with a Python file

import sys

and edit it to look like

import sys
import os

The imports are clearly unsorted and ruff detects that, but somehow this leads to the exception above. As additional info, I'm attaching the contents of the Emacs buffers related to pylsp and ruff: lsp-log_pylsp.log and lsp-log_ruff.log

Some more infos:

• python-lsp-ruff version 2.2.2
• ruff version 0.6.5
• the relevant snippet from pyproject.toml (full version here):

[tool.ruff.lint]
select = ["E", "F", "I", "W", "C90", "PL"]

• the simple Spacemacs configuration for the Python layer (hopefully understandable even by non-Spacemacs users):

(python :variables
        lsp-pylsp-plugins-mypy-enabled t
        lsp-pylsp-plugins-pylint-enabled t
        lsp-pylsp-plugins-ruff-format '("I")
        )

[jhossbach]

Hmm, I cannot reproduce this. Can you increase the verbosity of pylsp and save to a log file? See https://github.com/python-lsp/python-lsp-ruff?tab=readme-ov-file#debugging for inspiration (I don't know how to translate that to spacemacs though)

[svenpanne]

OK, here the pylsp_stderr.log when the verbosity has been set to -vvv, hope that helps.

BTW: In Spacemacs you just have to add the single line

lsp-pylsp-server-command '("pylsp" "-vvv")

to the LSP config mentioned above to raise the verbosity level..

Note that today we have ruff version 0.6.6 instead of 0.6.5, but its behaviour regarding the problem at hand seems to be identical.

[jhossbach]

Hmm okay, something is performing a ruff check and returns it to pylsp but with slightly different keys in the data field in a Diagnostic. For example, the Diagnostics returned by the emacs client all contain the codeDescription field but python-lsp-ruff does not use that. Also, the suggested edits should contain the keys content, location and end_location:

python-lsp-ruff/pylsp_ruff/ruff.py

Lines 11 to 15 in 041c695

	@dataclass
	class Edit:
	content: str
	location: Location
	end_location: Location

However, yours contain newText and a range. I just checked and python-lsp-ruff has never used these keys so I don't think that its a version error.
I am not sure where these diagnostics come from, but something messes with python-lsp-server. Can you make sure that absolutely no other ruff linter is present in your emacs setup?
I also found this where at least a similar diagnostic syntax is found: emacs-lsp/lsp-mode#4547

[svenpanne]

Hmmm, if one follows the LSP spec from textDocument/diagnostic through to Diagnostic, one ends up at LSPAny for data, which is not very helpful. But I can't find end_location anywhere in the LSP spec, so where is the Edit format specified?

(Disclaimer: I've got only some superficial knowledge about the whole LSP protocol, so I'm just guessing here)

[jhossbach]

That's not in the spec, but rather the edit format you get when calling ruff check --output-format=json test.py. It is up to the implementation of the lsp server to convert whatever linter to the required Edit, and that's also where your exception comes from AFAIK

[jhossbach]

I did find this: https://github.com/astral-sh/ruff/blob/03f3a4e85560333cf7175d44849f7250a6cdcb6a/crates/ruff_server/src/lint.rs#L40
It looks like your emacs editor is actually running ruff server somehow; You should be able to check whether this is the culprit by looking for any running ruff server processes when trying to perform a code action

[svenpanne]

Hmmm, this is how the process/thread tree for my Emacs looks like:

emacs
  |-node /home/sp/.emacs.d/.cache/lsp/npm/vscode-langservers-extracted/bin/vscode-json-language-server --stdio
  |   `-10*[{node}]
  |-pylsp /home/sp/git/check_mk/.venv/bin/pylsp
  |   |-python /home/sp/git/check_mk/.venv/lib/python3.12/site-packages/jedi/inference/compiled/subprocess/__main__.py /home/sp/git/check_mk/.venv/lib/python3.12/site-packages 3.12.6
  |   |-python /home/sp/git/check_mk/.venv/lib/python3.12/site-packages/jedi/inference/compiled/subprocess/__main__.py /home/sp/git/check_mk/.venv/lib/python3.12/site-packages 3.12.6
  |   `-17*[{pylsp}]
  |-ruff server
  |   `-20*[{ruff}]
  `-6*[{emacs}]

This is consistent with the Emacs code for the LSP mode: https://github.com/emacs-lsp/lsp-mode/blob/561788352cd566558d303f72821c160295089839/clients/lsp-ruff.el#L34

Should this structure look different somehow, e.g. ruff server being a subprocess of pylsp?

[jhossbach]

I think you are confusing python-lsp-ruff and ruff-lsp (or rather ruff server):
You should not need lsp-ruff.el at all, only lsp-pylsp.el, and ruff is used independently of the emacs lsp config. You should not see any ruff server process.

[svenpanne]

I think I finally understand what's going on: lsp-mode explicitly uses ruff as an add-on for other running Python LSP servers. This is why one sees pylsp (the main Python LSP server) and ruff server (the add-on Python LSP server) in the process tree above.

AFAICT, python-lsp-server is a plugin for pylsp, which explicitly & repeatedly executes ruff ... <some args> when needed, i.e. without using ruff server.

So when one uses lsp-mode, one doesn't need python-lsp-ruff at all, quite the opposite: Somehow they interfere, for whatever reason. So for my use case the solution is simple: Remove python-lsp-ruff from the virtual environment and let lsp-mode do its thing with 2 LSP servers for Python.

In a nutshell: Nothing seems to be wrong here, sorry for the noise, this ticket here can be closed. But OTOH, I think the Spacemcas docu needs some love, the LSP ecosystem for Python is quite confusing...