Transition to rust-analyzer as our official LSP (Language Server Protocol) implementation

[nikomatsakis]

The RFC proposes a plan to adopt rust-analyzer as Rust's official LSP (Language Server Protocol) implementation. The transition to rust-analyzer will take place in a few stages:

• Feedback -- encourage people to use rust-analyzer and report problems
• Deprecation -- actively transition people from the RLS to rust-analyzer
• Final transition -- stop supporting the older RLS

As detailed below, one major concern with rust-analyzer as it stands today is that it shares very little code with rustc. To avoid creating an unsustainable maintenance burden, this RFC extracting shared libraries that will be used by both rustc and rust-analyzer ("library-ification"), which should eventually lead to rustc and rust-analyzer being two front-ends over a shared codebase.

Rendered.

Summary comments

If you're just joining the conversation, this summary comment summarizes all the points raised before it, and is a good starting point.

Thanks

Thanks to @matklad who co-authored the RFC.

[nikomatsakis]

I've chosen to tag @rust-lang/core for the team on this RFC, as it seems like a decision that impacts the project as a whole in a fairly substantial way. But I definitely think we want @rust-lang/compiler, @rust-lang/dev-tools, and @rust-lang/infra agreement with this RFC as well (I'm open to adding more teams, I just don't want to create excessive "rfcbot" overhead either).

[Diggsey]

For me, rust-analyzer has been the superior option for some time now. It's great 👍

However, there is a problem with stability: the "stable" channel still updates every week, and inevitably something (sometimes minor, sometimes quite major!) breaks with each release. If this is to become the "official IDE" then it might make sense to follow a similar release pattern to rustc, with "stable", "beta" and "nightly" channels.

Ideally it would be possible to have the "nightly" or "weekly" releases by default, but then when something breaks have the option to revert to the previous version until the next release (which at the moment is not possible, you can only switch channels).

[matklad]

but then when something breaks have the option to revert to the previous version until the next release (which at the moment is not possible, you can only switch channels).

@Diggsey it is possible to revert to an older weekly release today:

[nikomatsakis]

@Diggsey

However, there is a problem with stability: the "stable" channel still updates every week, and inevitably something (sometimes minor, sometimes quite major!) breaks with each release.

By this, you mean the stable channel of rust-analyzer, I presume? It's probably worth stating our expectations regarding distribution channels. I would presume that the current "weekly release" cycle will slow down as things get more stable, though I don't know when that would be. (Probably sometime in the "deprecation" period.)

One interesting question is whether it's necessary or desirable to (eventually) distribute rust-analyzer over rustup, so that everything is managed together.

With the save-analysis-based infrastructure, that was required because the RLS consumed the save-analysis output produced by rustc, and the format of save-analysis data was highly unstable and could change in every release.

rust-analyzer however is independent from rustc, and hence could change in principle be distributed at a different frequency and through different channels (as it is now).

When we get to the point where both rustc and rust-analyzer are built from the same core libraries, though, I can see some logic in having them upgrade and downgrade together, in that they will continue to support the same set of language features that way.

In any case, I'm not sure we have to answer this question now. It seems like a question that would be better answered when we're further along in the transition process. Perhaps it ought to be listed explicitly as an "unresolved question" or future possibility?

I know that @matklad and I discussed this a few times and I can't quite remember where we landed, so I'll let them weigh in.

[matklad]

In terms of stability as in "things do not break", I think that, de facto, the IDE story for Rust is in alpha stage, and would be there for some time. We should put more effort into making things more "stable" for the users, and beta/stable trains are a good instrument here, but we also should be careful not to advertise IDE integration as fixed until it actually is. So, something like rustup component +stable add rust-analyzer-preview is probably what we should aim at for the next couple of years. In retrospective, rust-analyzer's current usage of "stable" name for the weekly update channel is probably a mistake.

In terms of distribution, it is indeed the case that rust-analyzer can be distributed separately from the main toolchain (as we don't use nightly features) and that there's some benefit for developers in doing so (we can iterate faster), and maybe even for users (we get new features and bug fixes faster) (but we also get new bugs and regressions faster).

Additionally, distribution of binaries comes with security implications. The current distribution story (using github actions to publish github releases and marketplace extension) seems fine for a 3rd party tool, but would be underwhelming for an official project.

So, if/when we complete the transition period, we definitely want to ship rust-analyzer via rustup, just like any other official tool. Given that we want this eventually, we might as well start early, somewhere before the "deprecation" phase. There's a hope that the change of distribution mechanism would be transparent for the users, as it is the editor plugin which manages the set up of the server.

[pietroalbini]

I'd prefer for rust-analyzer to be shipped through rustup soon after the RFC is merged: being able to pull it like any other component would be great, especially for people who are not using VSCode.

In the short term, to avoid increasing the workload of the rust-analyzer and release teams, we could ship rust-analyzer-preview as a nightly-only component (like we do with miri). This way we don't have to deal with backporting bugfixes to a weeks-old beta or stable branch, and the maintenance work consists of just sending a PR bumping the sub{module,tree} once every week. Sending such PR could be even automated with their existing GHA setup.

[chris-morgan]

Please not “library-ification”. “librarification” or “librification”.

[kiljacken]

While the original wording does seem a little silly I perosnally find it way easier to read and understand than the suggested contracted versions. Maybe "Library Extraction" serves the purpose better?

[matu3ba]

Library-fication would be most exact, if they intend to describe the goal.
If they are doing it now its ying/y-ification.

[matu3ba]

@kiljacken the whole is becoming a collection of libraries, not only a part

[Xanewok]

Not sure how much we're invested in using the actual term but maybe modularization would be better? (It does remind me of Rust modules, though...)

[matu3ba]

Since Rust reads very nicely like a book, I prefer the book-related term. Arization comes from arize comes from arise which sounds like something pops up unexpected. ;-)

[nikomatsakis]

lol, I am for some odd reason quite fond of the "obviously incorrect" library-ification spelling, and it's definitely what we say more commonly when speaking out loud and typing informally, in contrast to librarification.

I don't like "modularization", even though I agree it is more correct, because it doesn't really communicate the "scale" to me. (As you said, @Xanewok, it sounds like Rust modules.)

In any case, I.. don't really want to debate this particular terminology right here, as it is hardly central to the RFC. Maybe we can come up with some clever pun based on the term crates though...

[joshtriplett]

I would similarly like to see rust-analyzer distributed in rustup as part of release channels, in the hopes that stable rust-analyzer will always parse stable rust, and nightly rust-analyzer will parse nightly rust.

Hopefully that'll be tested by test suites, as well.

That doesn't need to happen immediately, but long-term I'd like to see it riding the same trains.

[benbrittain]

I'd like to echo what @joshtriplett said here. We recently switched to only supporting rust-analyzer on the Fuchsia project mostly due to it's really awesome rust-project.json format (and our build system generates it natively now too!). It's pretty easy for VS Code users because of how the plugin system works, but it's onerous for everyone else.
We have a dedicated toolchain team and ensuring our users are using a known supported version is important.

[bstaletic]

As a maintainer of a LSP client that doesn't, and likely never will, support snippet based completion, switching the users of our client from RLS to rust-analyzer is impossible.

Currently rust-analyzer always claims that completionItemKind is 2, which corresponds to "snippet". In this case, our client simply fails an assertion.

The reason why we (the maintainers of ycmd) don't support snippets in completion items is because we consider them a terrible way to do completions.

In one conversation with @matklad, he said that implementing support for "snippetless clients" is not a priority, which I understood as "pull requests are welcome". However, I don't have enough rust experience to implement the feature myself.

Lastly, I have to point out that this behaviour is a violation of the protocol, since rust-analyzer doesn't respect client capabilities.

[oblitum]

@bstaletic you didn't give a logical reason, just a label.

As for me I find them absolutely great, specially that RA even provides configuration for the kind of expansion you may want for signatures, with or without placeholders, etc, I'd absolutely miss this.

[BurntSushi]

I think it would be wise to avoid taking this thread in a direction that discusses current missing features or shortcomings of rust-analyzer. As long as there is a path to those things being resolved, then I think that should be enough for the purposes of this discussion. I don't think we need to go off on a tangent debating different styles of completions, please.

Now if there are things that are impossible to achieve with rust-analyzer due to fundamental design differences caused by this library-ification, then I think those would definitely be worthwhile concerns to bring up!

[kinnison]

If the intent is to integrate rust-analyzer into the main release channels and handle it through rustup then at least some work will be needed in rustup (e.g. provision of proxy)

[bstaletic]

@oblitum

you didn't give a logical reason, just a label.

I thought "a protocol violation" was a reason enough.

rust-lang/rust-analyzer#2518 (comment)

As for me I find them absolutely great, specially that RA even provides configuration for the kind of expansion you may want for signatures, with or without placeholders, etc, I'd absolutely miss this.

And that's perfectly fine. I'm not saying "throw away snippets". I'm saying "respect client capabilities, including 'I-can't-do-snippets'" and "respect the protocol".

Violating the protocol is problematic as it defeats the point of the protocol and sometimes forces clients to be bug-compatible with vscode.

@BurntSushi

What I brought up is definitely not impossible, but then hardly anything is. Snippets are the only thing stopping my client from working with rust-analyzer, but respecting client capabilities in general is something rust-analyzer has not been paying attention to.

rust-lang/rust-analyzer#144

[oblitum]

I thought "a protocol violation" was a reason enough.

Guess I may have not expressed well enough. When you mentioned "reason", you were talking about why you don't provide them, and the only reason mentioned is because they're "terrible". I was not asking about the logical reason not having no-snippets is a problem, but the logical reason they're terrible for completion, besides the label "terrible".

[matklad]

@bstaletic thanks for raising the protocol conformance issue, I will give more details about it in the next comment.

@oblitum @bstaletic I strongly suggest moving the discussion about snippets in particular elsewhere.

rust-lang/rust-analyzer#2518 might be a good place to discuss implementation details in rust-analyzer specifically.

Not sure what's the right venue for discussion about whether snippet completion are a good or a bad feature, but it s definitely neither this RFC, nor rust-analyzer issue tracker.

[bstaletic]

@matklad

thanks for raising the protocol conformance issue, I will give more details about it in the next comment.

That would be very appreciated. Thanks!

I absolutely agree that opinions on snippets are offtopic.

[bstaletic]

Speaking of rust-analyzer in vim, ever since this happened, YouCompleteMe doesn't need any RA specific code to work correctly with it. All that is needed is something like

let g:ycm_language_server = [ { 'name': 'rust', 'filetypes': [ 'rust' ], 'cmdline': [ 'rust-analyzer' ], 'project_root_files': [ 'Cargo.toml' ] } ]

RA specific protocol extensions aren't supported.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

[eminence]

I do believe that maintaining vscode plugin in the same repository is beneficial. It significantly improves development workflow for maintainers. What is even more important, it lowers the contribution barrier for both the server and the client, as you can just run the two immediately after cloning the repository.

As someone who is very new to both RA and VS Code, I can confirm that all the efforts by the RA team to make this a smooth process have paid off. I was able to step through vscode typescript code and RA code in a debugger while knowing very little about either. This is some great tooling so kudos to the RA team for making this work so well.

Speaking as a user, there's also a small benefit to keeping the plugin and server in the same repo: it's one unified location for submitting bugs, and the RA team quickly triages issues into server problems or plugin problems. But with vim, for example: I'm currently having some problems with the automatic download feature, and it's not clear if this problem with coc, coc-rust-analyzer, or rust-analyzer (and so not clear where to report the issue)

[nikomatsakis]

Huzzah! This RFC has been merged. You can follow along at the tracking issue that @matklad created rust-analyzer/rust-analyzer#4224.

[mark-i-m]

Umm... isn't this supposed to be merged, rather than closed?

[gibfahn]

Umm... isn't this supposed to be merged, rather than closed?

Looks like it has landed:

b6fffdf
https://github.com/rust-lang/rfcs/blob/master/text/2912-rust-analyzer.md

Probably because it landed on master before the force-push to update the PR branch to the same commit.

[nikomatsakis]

Yeah I think I merged the PR but then realized that I was out of date with master and hence did a rebase atop master, or something like that.