Integration with SublimeLSP stopped working

[zeux]

This used to work just fine, but after updating to a more recent version of luau-lsp (apologies, I haven't noted the last version that worked) I'm getting this:

Traceback (most recent call last):
  File "/home/zeux/.config/sublime-text/Installed Packages/LSP.sublime-package/plugin/core/transports.py", line 149, in invoke
  File "/home/zeux/.config/sublime-text/Installed Packages/LSP.sublime-package/plugin/core/sessions.py", line 1964, in on_payload
  File "/home/zeux/.config/sublime-text/Installed Packages/LSP.sublime-package/plugin/core/sessions.py", line 1952, in deduce_payload
TypeError: int() argument must be a string or a number, not 'NoneType'

The Python code that errors is

        elif "id" in payload:
            response_id = int(payload["id"])

I'm assuming that "id" is set to null in the JSON response... I'm going to try to bisect this but I wanted to file this first in case it's something obvious.

[zeux]

Actually, I synced all the way back to June and I'm hitting the same issue, so I'm now assuming this is on the SublimeLSP side. Will close for now, sorry for the noise!

[JohnnyMorganz]

id is set in the response payload, the only time this is set to null is when we return an error and we don't have a relevant request id to attach it too (this typically mean there is a bug in Luau LSP as it happens when we catch an unhandled exception).

Note that the specification mentions its valid to have a null id in the response https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#responseMessage, so it sounds like SublimeLSP is not following that part correctly.

[zeux]

id is set in the response payload, the only time this is set to null is when we return an error and we don't have a relevant request id to attach it too (this typically mean there is a bug in Luau LSP as it happens when we catch an unhandled exception).

Yup that's exactly what's going on here:

luau: trace level: "off"
luau: No definitions file provided by client
luau: No definitions file provided by client
luau: client does not allow didChangeWatchedFiles registration - automatic updating on sourcemap/config changes will not be provided
luau: failed to process response 1 - [json.exception.type_error.306] cannot use value() with null

[zeux]

Btw my bisection results were actually incorrect because I forgot I need to restart the LSP server each time :( But the issue is in requestConfiguration handler:

                    ClientConfiguration config = *configIt;
                    configStore.insert_or_assign(uri.toString(), config);

configIt here points to a JSON "null" object; the entire response for configuration is [null]. This might be an unrelated bug, because fixing this (by using default-constructed ClientConfiguration when *configIt is null) doesn't fix the underlying problem...

[JohnnyMorganz]

Do you mind sending over the LSP settings you are using?
I just booted up a fresh installation of Luau LSP on sublime using the latest released version and it seems to be working fine for me

[JohnnyMorganz]

Also, enabling "log_server": true, might give more insight into what request is causing the failure

[zeux]

// Settings in here override those in "LSP/LSP.sublime-settings"
{
  "show_diagnostics_panel_on_save": 0,
  "clients": {
    "luau": {
      "enabled": true,
      "command": ["/home/zeux/luau-lsp", "lsp"],
      "selector": "source.lua"
    }
  }
}

[JohnnyMorganz]

This might be an unrelated bug, because fixing this (by using default-constructed ClientConfiguration when *configIt is null) doesn't fix the underlying problem...

Fixed this bug. This was the only issue I could find, I couldn't reproduce the initial one. Let me know if it continues to occur

[zeux]

I'll need to spend more time on this tomorrow. It's fairly puzzling:

• I think the repro is just not very reliable. I need to pin this down more precisely, it sometimes works and sometimes doesn't
• After the fix above I no longer have any error output from luau-lsp (so this fixes the exception issue), but I still get the same stack trace from SublimeLSP. That however is not indicative of the issue itself - as in, I sometimes get the error stack trace in SublimeLSP and yet the error diagnostics work completely fine.

Unrelated, the only other message I get from luau-lsp now is "luau: unknown notification method: textDocument/didSave" - does luau-lsp produce some other method like didChange instead?

[JohnnyMorganz]

the only other message I get from luau-lsp now is "luau: unknown notification method: textDocument/didSave" - does luau-lsp produce some other method like didChange instead?

This is completely normal. Yeah, we listen to didChange instead. We just haven't registered a handler for didSave because we don't need one right now. We could probably silence the warning though

[zeux]

Ok thanks. Apologies if this one is unrelated - please bear with me, as I'm trying to basically identify every single thing that I know is guaranteed wrong, so that only spurious unidentified failures remain :) - but one other quirk is that sometimes (often) when I open the file for the first time in a new Sublime window, I don't get any diagnostics until making a dummy change to the file. Looking at the log, it looks like document/textOpen which triggers the diagnostics publish is called before workspace is configured, which throws a cancellation exception (see log below, "calling pushDiagnostics" is something I've added).

Under these circumstances, is the LSP client (Sublime) responsible for sending textOpen again?

luau: No documentation file given. Documentation will not be provided
:: --> luau initialize(1): {'initializationOptions': {}, 'capabilities': {'general': {'markdown': {'version': '3.2.2', 'parser': 'Python-Markdown'}, 'regularExpressions': {'engine': 'ECMAScript'}}, 'window': {'showMessage': {'messageActionItem': {'additionalPropertiesSupport': True}}, 'showDocument': {'support': True}, 'workDoneProgress': True}, 'workspace': {'configuration': True, 'symbol': {'tagSupport': {'valueSet': [1]}, 'dynamicRegistration': True, 'symbolKind': {'valueSet': [15, 7, 5, 2, 18, 19, 23, 25, 1, 14, 21, 9, 4, 6, 26, 8, 22, 11, 16, 3, 20, 17, 13, 24, 12, 10]}}, 'applyEdit': True, 'executeCommand': {}, 'inlayHint': {'refreshSupport': True}, 'semanticTokens': {'refreshSupport': True}, 'workspaceFolders': True, 'didChangeConfiguration': {'dynamicRegistration': True}, 'workspaceEdit': {'failureHandling': 'abort', 'documentChanges': True}, 'codeLens': {'refreshSupport': True}}, 'textDocument': {'completion': {'completionItem': {'tagSupport': {'valueSet': [1]}, 'insertTextModeSupport': {'valueSet': [2]}, 'deprecatedSupport': True, 'documentationFormat': ['markdown', 'plaintext'], 'resolveSupport': {'properties': ['detail', 'documentation', 'additionalTextEdits']}, 'labelDetailsSupport': True, 'snippetSupport': True}, 'insertTextMode': 2, 'dynamicRegistration': True, 'completionItemKind': {'valueSet': [12, 10, 7, 9, 22, 24, 17, 21, 11, 4, 16, 15, 5, 20, 8, 2, 18, 19, 14, 25, 6, 23, 1, 3, 13]}}, 'documentLink': {'tooltipSupport': True, 'dynamicRegistration': True}, 'signatureHelp': {'contextSupport': True, 'dynamicRegistration': True, 'signatureInformation': {'documentationFormat': ['markdown', 'plaintext'], 'parameterInformation': {'labelOffsetSupport': True}, 'activeParameterSupport': True}}, 'rename': {'dynamicRegistration': True, 'prepareSupport': True}, 'inlayHint': {'resolveSupport': {'properties': ['textEdits', 'label.command']}, 'dynamicRegistration': True}, 'selectionRange': {'dynamicRegistration': True}, 'hover': {'contentFormat': ['markdown', 'plaintext'], 'dynamicRegistration': True}, 'formatting': {'dynamicRegistration': True}, 'semanticTokens': {'tokenModifiers': ['static', 'async', 'declaration', 'modification', 'deprecated', 'definition', 'documentation', 'readonly', 'abstract', 'defaultLibrary'], 'overlappingTokenSupport': False, 'requests': {'range': True, 'full': {'delta': True}}, 'dynamicRegistration': True, 'tokenTypes': ['string', 'property', 'class', 'type', 'struct', 'macro', 'typeParameter', 'regexp', 'parameter', 'keyword', 'method', 'comment', 'enumMember', 'interface', 'number', 'namespace', 'modifier', 'operator', 'decorator', 'variable', 'event', 'function', 'enum'], 'augmentsSyntaxTokens': True, 'multilineTokenSupport': True, 'formats': ['relative']}, 'synchronization': {'didSave': True, 'willSave': True, 'dynamicRegistration': True, 'willSaveWaitUntil': True}, 'implementation': {'linkSupport': True, 'dynamicRegistration': True}, 'declaration': {'linkSupport': True, 'dynamicRegistration': True}, 'rangeFormatting': {'dynamicRegistration': True}, 'documentSymbol': {'tagSupport': {'valueSet': [1]}, 'hierarchicalDocumentSymbolSupport': True, 'dynamicRegistration': True, 'symbolKind': {'valueSet': [15, 7, 5, 2, 18, 19, 23, 25, 1, 14, 21, 9, 4, 6, 26, 8, 22, 11, 16, 3, 20, 17, 13, 24, 12, 10]}}, 'typeDefinition': {'linkSupport': True, 'dynamicRegistration': True}, 'definition': {'linkSupport': True, 'dynamicRegistration': True}, 'codeAction': {'disabledSupport': True, 'codeActionLiteralSupport': {'codeActionKind': {'valueSet': ['quickfix', 'refactor', 'refactor.extract', 'refactor.inline', 'refactor.rewrite', 'source.organizeImports']}}, 'dynamicRegistration': True, 'dataSupport': True, 'resolveSupport': {'properties': ['edit']}}, 'publishDiagnostics': {'tagSupport': {'valueSet': [2, 1]}, 'codeDescriptionSupport': True, 'versionSupport': True, 'relatedInformation': True, 'dataSupport': True}, 'documentHighlight': {'dynamicRegistration': True}, 'colorProvider': {'dynamicRegistration': True}, 'codeLens': {'dynamicRegistration': True}, 'references': {'dynamicRegistration': True}}}, 'rootPath': '/home/zeux/game-engine/Client/Luau', 'clientInfo': {'name': 'Sublime Text LSP', 'version': '1.18.0'}, 'processId': 3310314, 'workspaceFolders': [{'name': 'Luau', 'uri': 'file:///home/zeux/game-engine/Client/Luau'}], 'rootUri': 'file:///home/zeux/game-engine/Client/Luau'}
:: <-  luau window/logMessage: {'type': 2, 'message': 'No documentation file given. Documentation will not be provided'}
:: <<< luau 1: {'capabilities': {'definitionProvider': True, 'workspace': {'workspaceFolders': {'supported': True, 'changeNotifications': False}}, 'typeDefinitionProvider': True, 'implementationProvider': False, 'documentSymbolProvider': True, 'inlayHintProvider': True, 'textDocumentSync': {'didClose': {}, 'save': {}, 'didOpen': {}, 'change': {'syncKind': 2}}, 'semanticTokensProvider': {'range': False, 'full': True, 'legend': {'tokenModifiers': ['declaration', 'definition', 'readonly', 'static', 'deprecated', 'abstract', 'async', 'modification', 'documentation', 'defaultLibrary'], 'tokenTypes': ['namespace', 'type', 'class', 'enum', 'interface', 'struct', 'typeParameter', 'parameter', 'variable', 'property', 'enumMember', 'event', 'function', 'method', 'macro', 'keyword', 'modifier', 'comment', 'string', 'number', 'regexp', 'operator', 'decorator']}}, 'declarationProvider': False, 'signatureHelpProvider': {'triggerCharacters': ['(', ',']}, 'diagnosticProvider': {'interFileDependencies': True, 'workspaceDiagnostics': True, 'identifier': 'luau'}, 'documentLinkProvider': {'resolveProvider': False}, 'completionProvider': {'completionItem': {'labelDetailsSupport': True}, 'triggerCharacters': ['.', ':', "'", '"', '\n'], 'resolveProvider': False}, 'positionEncoding': 'utf-16', 'renameProvider': True, 'referencesProvider': True, 'hoverProvider': True}}
::  -> luau initialized: {}
luau: server initialized!
luau: trace level: "off"
::  -> luau textDocument/didOpen: {'textDocument': {'text': 'function find(needle, haystack: xxx)\n    for i, e in haystack do\n        if e == needle then return i end\n    end\nend\n\nfoo()\n\n', 'version': 0, 'languageId': 'lua', 'uri': 'file:///home/zeux/game-engine/Client/Luau/test.lua'}}
:: --> luau textDocument/documentLink(2): {'textDocument': {'uri': 'file:///home/zeux/game-engine/Client/Luau/test.lua'}}
:: <-  luau window/logMessage: {'type': 3, 'message': 'server initialized!'}
:: <-  luau window/logMessage: {'type': 3, 'message': 'trace level: "off"'}
luau: No definitions file provided by client
:: <-  luau window/logMessage: {'type': 2, 'message': 'No definitions file provided by client'}
luau: No definitions file provided by client
luau: client does not allow didChangeWatchedFiles registration - automatic updating on sourcemap/config changes will not be provided
:: <-  luau window/logMessage: {'type': 2, 'message': 'No definitions file provided by client'}
:: <-- luau client/registerCapability(0): {'registrations': [{'method': 'workspace/didChangeConfiguration', 'registerOptions': None, 'id': 'didChangeConfigurationCapability'}]}
:: >>> luau 0: None
:: <-- luau workspace/configuration(1): {'items': [{'section': 'luau-lsp'}, {'section': 'luau-lsp', 'scopeUri': 'file:///home/zeux/game-engine/Client/Luau'}]}
luau: calling pushDiagnostics; isConfigured=0
:: >>> luau 1: [None, None]
:: <-  luau window/logMessage: {'type': 2, 'message': 'client does not allow didChangeWatchedFiles registration - automatic updating on sourcemap/config changes will not be provided'}
:: <-  luau window/logMessage: {'type': 3, 'message': 'calling pushDiagnostics; isConfigured=0'}
luau: loaded configuration for 
luau: loaded configuration for file:///home/zeux/game-engine/Client/Luau
:: <<< luau 2: []
:: <-  luau window/logMessage: {'type': 3, 'message': 'loaded configuration for '}
:: <-  luau window/logMessage: {'type': 3, 'message': 'loaded configuration for file:///home/zeux/game-engine/Client/Luau'}
:: <-  luau window/showMessage: {'type': 1, 'message': "Failed to load sourcemap.json for workspace 'Luau'. Instance information will not be available"}

[JohnnyMorganz]

Yeah, I spotted this too, and you are correct - this is because the didOpen notification was sent before we received configuration.

(This check is mainly here to stopped the inverse happening - loads of false positive diagnostics in Roblox code because the default constructed config has Roblox support disabled)

The logic is somewhat flawed for push diagnostics. The server is mainly setup for pull diagnostics mode (which is the newer method), but Sublime does not yet support it, so we fall back to legacy push diagnostics.
The expectation would be that when we send the diagnostics cancellation, the client retriggers a textDocument/diagnostic request - but this only happens in pull mode.

didOpen should only be sent once. The issue is on the server side here. If we are in push diagnostics mode, what we should probably do is re-push all diagnostics for all currently opened files when the config changed callback is run (

luau-lsp/src/LanguageServer.cpp

Line 364 in b609ead

client->configChangedCallback = [&](const lsp::DocumentUri& workspaceUri, const ClientConfiguration& config)

)

(just noticed, we already do half of this, but only when whole workspace diagnostics is enabled)

[zeux]

After all these fixes I can't reproduce the problem anymore 😅 I'm going to wait until the next binary build so that I can confirm that on an executable built externally, and then hopefully close this.

The only identifiable problem that is left is the exception in Sublime LSP from the OP, which manifests on some errors, but that's a bug on their end that I'll submit a PR for.