merlin: add new contexts commands #10324

jchavarri · 2024-03-28T10:12:46Z

Fixes #10222.

This work is part of the multi-context support for vscode (see ocamllabs/vscode-ocaml-platform#1432).

Current version

This PR prepares Dune to support multi-context config retrieval from Merlin by doing the following:

Add an optional flag --context to dune ocaml-merlin: if passed, this flag will be used to return data to subsequent requests to the ocaml-merlin server
Add dune describe contexts subcommands, which returns the list of Dune contexts available (1 per line)
It also adds a new field generate_merlin_rules to the contexts declared in the workspace (originally implemented in merlin: introduce generate_merlin_rules #10328). This new field allows to generate merlin rules in any case, even if the context is not selected using (merlin). This addition is useful to dynamically change the context for Merlin via the SetContext command mentioned above.

Original version (now obsolete)

This PR adds two new commands to the ocaml-merlin API:

GetContexts: returns the list of contexts available in the workspace (only ctxt names)
SetContext(name): sets the current context to get merlin data from

It also adds a new field generate_merlin_rules to the contexts declared in the workspace (originally implemented in #10328). This new field allows to generate merlin rules in any case, even if the context is not selected using (merlin). This addition is useful to dynamically change the context for Merlin via the SetContext command mentioned above.

voodoos · 2024-04-03T13:21:20Z

I am wondering whether adding more feature to the "configuration server" is the correct way to proceed. This ad-hoc server was implemented before the landing of the more general Dune's RPC feature, and one future goal at some point was to move the ad-hoc configuration server logic to the RPC. Adding new features to dune ocaml-merlin would only make a transition to the RPC more complex.

Did you consider adding this feature to the RPC instead ? It feels like it would be a more correct way to proceed.
There would be one drawback however: only lsp-based editor plugins will be able to rely on it since, as of today, the old Merlin server does not speak with Dune's RPC. Since we might switch every plugins to using lsp in the future, and you seem to mostly target vscode, that might not be an issue ?

jchavarri · 2024-04-03T16:20:55Z

@voodoos Thanks for the feedback. I wasn't aware of how ocaml-lsp and Dune communicated through Dune's RPC feature, I understand you are referring to this Dune module?

I am not sure I understand how the suggested approach would work in terms of storing the information about the selected context. In the current implementation, the SetContext command will change the value of the current context:

dune/bin/ocaml/ocaml_merlin.ml

Line 215 in 7a5395e

| true -> selected_context := Custom ctxt_name)

which is stored as a ref that is initialized as soon as dune ocaml-merlin starts:

dune/bin/ocaml/ocaml_merlin.ml

Line 181 in 7a5395e

let selected_context = ref Commands.Default in

This is handy because this value can also be used by dune ocaml-merlin where the editor information is requested:

dune/bin/ocaml/ocaml_merlin.ml

Line 186 in 7a5395e

let* () = print_merlin_conf ~selected_context:!selected_context path in

As can be seen, this state handling happens only a few lines apart and doesn't spread to other modules outside Ocaml_merlin.

If the new commands were moved to Dune's RPC, then we would need to find a way to sync between the two processes (rpc and ocaml-merlin), because we would be writing the state through Dune's RPC, but we would still have to read it from the ocaml-merlin process. Or is there an alternative to avoid that?

Adding new features to dune ocaml-merlin would only make a transition to the RPC more complex.

Considering the above, I think this addition doesn't change much the current state of things, and moving all the Merlin configuration handling and state at once to the RPC will be easier than spreading state across processes. I would be happy to eventually work on transitioning everything to the RPC if/when time permits, but that sounds like a much larger task and scope than the current goal: to allow users to select the context used by print_merlin_conf.

jchavarri · 2024-04-03T16:57:50Z

Since we might switch every plugins to using lsp in the future, and you seem to mostly target vscode, that might not be an issue ?

Unfortunately, I would need to support users that don't use vscode / ocaml-lsp and query merlin directly via vim or emacs.

voodoos · 2024-04-04T08:16:47Z

test/blackbox-tests/test-cases/merlin/get-set-contexts.t

+Request config for file in alt context before calling SetContext
+
+  $ printf '(4:File%d:%s)' ${#FILE2} $FILE2 | dune ocaml-merlin | grep -i "$lib2" | sed 's/^[^:]*:[^:]*://'
+  No config found for file bar.ml. Try calling 'dune build'.))


I think we should improve the error messages now that the user might be using the wrong context since dune build won't solve that issue.

voodoos · 2024-04-04T08:16:54Z

I am not sure I understand how the suggested approach would work in terms of storing the information about the selected context.

Ah yes, you are right, unless we move everything to the RPC we cannot easily share the state here.

jchavarri · 2024-04-04T09:24:25Z

@voodoos Thanks for the review. I wonder what your thoughts are about the suggestion in ocamllabs/vscode-ocaml-platform#1432 (comment)? In particular, this part:

add --context CONTEXT option dune ocaml ocaml-merlin

Maybe that would make easier to transition to RPC in the future? We would just need to move the --context flag from the dune ocaml-merlin executable to the RPC one. Besides that, I find interesting about the suggestion that it removes the need to patch dot-merlin-reader, which I was preparing in ocaml/merlin#1743.

jchavarri · 2024-04-10T13:10:56Z

@voodoos I have updated the PR to follow our recent decision to use a --context flag rather than a new protocol command to set the context. I also added a new subcommand dune describe contexts to list the contexts, which I will use from ocaml-lsp later on to offer clients the list of contexts available (to show in e.g. vscode a menu to choose from).

The work on ocaml-lsp is still ongoing, but any feedback you might have on this one is already much appreciated. Thanks!

bin/ocaml/ocaml_merlin.ml

jchavarri · 2024-04-18T08:15:09Z

The downstream PRs that depend on this feature are ready for review:

The latter includes a short screen capture that helps visualize what the features added in this PR enable, from the user perspective.

jchavarri · 2024-04-23T08:54:44Z

@rgrinberg @voodoos Apologies for pinging, I wonder if you have any thoughts on this PR? Thanks.

anmonteiro

This looks good overall and my suggestions are mostly minor. The important ones are:

should we gate this on lang dune 3.16?
should we make the other commands in dune ocaml-merlin take --context too, not just print-conf?

bin/describe/describe_contexts.ml

anmonteiro · 2024-04-30T23:17:39Z

bin/ocaml/ocaml_merlin.ml

+      match ctx_name with
+      | None -> Default
+      | Some ctx_name ->
+        (match Context_name.of_string_opt ctx_name with


I was wondering what the valid names for contexts were and incidentally found a crash: #10472

anmonteiro · 2024-04-30T23:18:46Z

bin/ocaml/ocaml_merlin.ml

+      | None -> Default
+      | Some ctx_name ->
+        (match Context_name.of_string_opt ctx_name with
+         | None -> User_error.raise [ Pp.textf "Invalid context name %S" ctx_name ]


we could provide a hint (maybe it could even live in context_name.ml) about what's a valid context name here.

i agree. maybe this can be done in a separate PR (worth including a fix for your repro in #10472 as well).

bin/ocaml/ocaml_merlin.ml

doc/changes/10324.md

doc/reference/dune-workspace/context.rst

src/dune_rules/workspace.ml

jchavarri · 2024-05-02T07:22:10Z

@anmonteiro thanks for the review!

should we gate this on lang dune 3.16?
should we make the other commands in dune ocaml-merlin take --context too, not just print-conf?

I thought the answer to these 2 questions is yes, so just included them in the PR, pls lmk if I'm missing anything :)

bin/ocaml/ocaml_merlin.ml

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

Co-authored-by: Antonio Nuno Monteiro <anmonteiro@gmail.com> Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

Co-authored-by: Antonio Nuno Monteiro <anmonteiro@gmail.com> Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

Signed-off-by: Antonio Nuno Monteiro <anmonteiro@gmail.com>

anmonteiro · 2024-05-06T00:06:15Z

nevermind, I just messed up the rebase.

…in dump Signed-off-by: Antonio Nuno Monteiro <anmonteiro@gmail.com>

anmonteiro · 2024-05-06T01:13:59Z

I also just pushed a commit that:

removes Selected_context.t (Context_name.t and associated functions have all we need)
add the --context arg to dump too
reuse the context arg as Selected_context.arg

anmonteiro · 2024-05-06T01:30:19Z

Thank you, let's do it

@anmonteiro

CHANGES: ### Added - allow libraries with the same `(name ..)` in projects as long as they don't conflict during resolution (via `enabled_if`). (ocaml/dune#10307, @anmonteiro, @jchavarri) - `dune describe pp` now finds the exact module and the stanza it belongs to, instead of guessing the name of the preprocessed file. (ocaml/dune#10321, @anmonteiro) - Print the result of `dune describe pp` with the respective dialect printer. (ocaml/dune#10322, @anmonteiro) - Add new flag `--context` to `dune ocaml-merlin`, which allows to select a Dune context when requesting Merlin config. Add `dune describe contexts` subcommand. Introduce a field `generate_merlin_rules` for contexts declared in the workspace, that allows to optionally produce Merlin rules for other contexts besides the one selected for Merlin (ocaml/dune#10324, @jchavarri) - melange: add include paths for private library `.cmj` files during JS emission. (ocaml/dune#10416, @anmonteiro) - `dune ocaml-merlin`: communicate additional directives `SOURCE_ROOT`, `UNIT_NAME` (the actual name with wrapping) and `INDEX` with the paths to the index(es). (ocaml/dune#10422, @voodoos) - Add a new alias `@ocaml-index` that uses the `ocaml-index` binary to generate indexes that can be read by tools such as Merlin to provide project-wide references search. (ocaml/dune#10422, @voodoos) - merlin: add optional `(merlin_reader CMD)` construct to `(dialect)` stanza to configure a merlin reader (ocaml/dune#8567, @andreypopp) ### Changed - melange: treat private libraries with `(package ..)` as public libraries, fixing an issue where `import` paths were wrongly emitted. (ocaml/dune#10415, @anmonteiro) - install `.glob` files for Coq theories too (ocaml/dune#10602, @ejgallego) ### Fixed - Don't try to document non-existent libraries in doc-new target (ocaml/dune#10319, fixes ocaml/dune#10056, @jonludlam) - Make `dune-site`'s `load_all` function look for `META` files so that it doesn't fail on empty directories in the plugin directory (ocaml/dune#10458, fixes ocaml/dune#10457, @shym) - Fix incorrect warning for libraries defined inside non-existant directories using `(subdir ..)` and used by executables using `dune-build-info` (ocaml/dune#10525, @rgrinberg) - Don't try to take build lock when running `coq top --no-build` (ocaml/dune#10547, fixes ocaml/dune#7671, @lzy0505) - Make sure to truncate dune's lock file after locking and unlocking so that users cannot observe incorrect pid's (ocaml/dune#10575, @rgrinberg) - mdx: link mdx binary with `byte_complete`. This fixes `(libraries)` with foreign archives on Linux. (ocaml/dune#10586, fixes ocaml/dune#10582, @anmonteiro)

@anmonteiro

CHANGES: ### Added - allow libraries with the same `(name ..)` in projects as long as they don't conflict during resolution (via `enabled_if`). (ocaml/dune#10307, @anmonteiro, @jchavarri) - `dune describe pp` now finds the exact module and the stanza it belongs to, instead of guessing the name of the preprocessed file. (ocaml/dune#10321, @anmonteiro) - Print the result of `dune describe pp` with the respective dialect printer. (ocaml/dune#10322, @anmonteiro) - Add new flag `--context` to `dune ocaml-merlin`, which allows to select a Dune context when requesting Merlin config. Add `dune describe contexts` subcommand. Introduce a field `generate_merlin_rules` for contexts declared in the workspace, that allows to optionally produce Merlin rules for other contexts besides the one selected for Merlin (ocaml/dune#10324, @jchavarri) - melange: add include paths for private library `.cmj` files during JS emission. (ocaml/dune#10416, @anmonteiro) - `dune ocaml-merlin`: communicate additional directives `SOURCE_ROOT`, `UNIT_NAME` (the actual name with wrapping) and `INDEX` with the paths to the index(es). (ocaml/dune#10422, @voodoos) - Add a new alias `@ocaml-index` that uses the `ocaml-index` binary to generate indexes that can be read by tools such as Merlin to provide project-wide references search. (ocaml/dune#10422, @voodoos) - merlin: add optional `(merlin_reader CMD)` construct to `(dialect)` stanza to configure a merlin reader (ocaml/dune#8567, @andreypopp) ### Changed - melange: treat private libraries with `(package ..)` as public libraries, fixing an issue where `import` paths were wrongly emitted. (ocaml/dune#10415, @anmonteiro) - install `.glob` files for Coq theories too (ocaml/dune#10602, @ejgallego) ### Fixed - Don't try to document non-existent libraries in doc-new target (ocaml/dune#10319, fixes ocaml/dune#10056, @jonludlam) - Make `dune-site`'s `load_all` function look for `META` files so that it doesn't fail on empty directories in the plugin directory (ocaml/dune#10458, fixes ocaml/dune#10457, @shym) - Fix incorrect warning for libraries defined inside non-existant directories using `(subdir ..)` and used by executables using `dune-build-info` (ocaml/dune#10525, @rgrinberg) - Don't try to take build lock when running `coq top --no-build` (ocaml/dune#10547, fixes ocaml/dune#7671, @lzy0505) - Make sure to truncate dune's lock file after locking and unlocking so that users cannot observe incorrect pid's (ocaml/dune#10575, @rgrinberg) - mdx: link mdx binary with `byte_complete`. This fixes `(libraries)` with foreign archives on Linux. (ocaml/dune#10586, fixes ocaml/dune#10582, @anmonteiro) - virtual libraries: fix an issue where linking an executable involving several virtual libries would cause an error. (ocaml/dune#10581, fixes ocaml/dune#10460, @rgrinberg)

@anmonteiro

CHANGES: ### Added - allow libraries with the same `(name ..)` in projects as long as they don't conflict during resolution (via `enabled_if`). (ocaml/dune#10307, @anmonteiro, @jchavarri) - `dune describe pp` now finds the exact module and the stanza it belongs to, instead of guessing the name of the preprocessed file. (ocaml/dune#10321, @anmonteiro) - Print the result of `dune describe pp` with the respective dialect printer. (ocaml/dune#10322, @anmonteiro) - Add new flag `--context` to `dune ocaml-merlin`, which allows to select a Dune context when requesting Merlin config. Add `dune describe contexts` subcommand. Introduce a field `generate_merlin_rules` for contexts declared in the workspace, that allows to optionally produce Merlin rules for other contexts besides the one selected for Merlin (ocaml/dune#10324, @jchavarri) - melange: add include paths for private library `.cmj` files during JS emission. (ocaml/dune#10416, @anmonteiro) - `dune ocaml-merlin`: communicate additional directives `SOURCE_ROOT`, `UNIT_NAME` (the actual name with wrapping) and `INDEX` with the paths to the index(es). (ocaml/dune#10422, @voodoos) - Add a new alias `@ocaml-index` that uses the `ocaml-index` binary to generate indexes that can be read by tools such as Merlin to provide project-wide references search. (ocaml/dune#10422, @voodoos) - merlin: add optional `(merlin_reader CMD)` construct to `(dialect)` stanza to configure a merlin reader (ocaml/dune#8567, @andreypopp) ### Changed - melange: treat private libraries with `(package ..)` as public libraries, fixing an issue where `import` paths were wrongly emitted. (ocaml/dune#10415, @anmonteiro) - install `.glob` files for Coq theories too (ocaml/dune#10602, @ejgallego) ### Fixed - Don't try to document non-existent libraries in doc-new target (ocaml/dune#10319, fixes ocaml/dune#10056, @jonludlam) - Make `dune-site`'s `load_all` function look for `META` files so that it doesn't fail on empty directories in the plugin directory (ocaml/dune#10458, fixes ocaml/dune#10457, @shym) - Fix incorrect warning for libraries defined inside non-existant directories using `(subdir ..)` and used by executables using `dune-build-info` (ocaml/dune#10525, @rgrinberg) - Don't try to take build lock when running `coq top --no-build` (ocaml/dune#10547, fixes ocaml/dune#7671, @lzy0505) - Make sure to truncate dune's lock file after locking and unlocking so that users cannot observe incorrect pid's (ocaml/dune#10575, @rgrinberg) - mdx: link mdx binary with `byte_complete`. This fixes `(libraries)` with foreign archives on Linux. (ocaml/dune#10586, fixes ocaml/dune#10582, @anmonteiro) - virtual libraries: fix an issue where linking an executable involving several virtual libries would cause an error. (ocaml/dune#10581, fixes ocaml/dune#10460, @rgrinberg)

jchavarri mentioned this pull request Mar 28, 2024

merlin: add rules always, regardless the configuration of (merlin) field #10325

Closed

jchavarri force-pushed the merlin-add-contxts-cmds branch from 2b74737 to 7df18fb Compare March 28, 2024 14:13

This was referenced Mar 29, 2024

add get/set contexts commands ocaml/merlin#1743

Closed

Support --context flag of dune ocaml-merlin ocaml/ocaml-lsp#1238

Closed

jchavarri force-pushed the merlin-add-contxts-cmds branch from 6fd6463 to bbc7ac8 Compare April 2, 2024 06:28

jchavarri marked this pull request as ready for review April 2, 2024 08:06

jchavarri requested review from rgrinberg and anmonteiro April 2, 2024 08:06

jchavarri added the merlin label Apr 2, 2024

jchavarri mentioned this pull request Apr 2, 2024

merlin: introduce generate_merlin_rules #10328

Closed

jchavarri mentioned this pull request Apr 3, 2024

Feature: Allow users to choose which Dune context to use ocamllabs/vscode-ocaml-platform#1432

Open

voodoos reviewed Apr 4, 2024

View reviewed changes

jchavarri force-pushed the merlin-add-contxts-cmds branch from ac96149 to 8bb8ca4 Compare April 8, 2024 14:17

jchavarri requested a review from voodoos April 10, 2024 13:11

voodoos reviewed Apr 12, 2024

View reviewed changes

bin/ocaml/ocaml_merlin.ml Outdated Show resolved Hide resolved

jchavarri requested a review from voodoos April 12, 2024 14:20

This was referenced Apr 16, 2024

Show the context name for errors happening in "alt" contexts #10414

Merged

Allow selection of Dune context ocamllabs/vscode-ocaml-platform#1449

Open

anmonteiro reviewed Apr 30, 2024

View reviewed changes

anmonteiro reviewed May 2, 2024

View reviewed changes

bin/ocaml/ocaml_merlin.ml Outdated Show resolved Hide resolved

anmonteiro reviewed May 2, 2024

View reviewed changes

bin/ocaml/ocaml_merlin.ml Outdated Show resolved Hide resolved

jchavarri and others added 18 commits May 5, 2024 17:05

add docs and changelog

606d42f

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: roll back changes in default-based-context test

2bec4bb

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: update get-set-contexts test

83a9b6b

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

update changes

727d3a6

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

rename Standard to Default

cc27c7a

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: rename Nothing to Not_selected

6122054

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: fix tests

6653a03

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

describe: add contexts subcommand

094d1b9

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: replace get/set context with flag

8bc5679

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: cleanup

4482ad7

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

cleanup

603e123

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: remove context check

a7436ab

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

apply suggestions from code review

b41a254

Co-authored-by: Antonio Nuno Monteiro <anmonteiro@gmail.com> Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

gate generate_merlin_rules to 3.16

8981216

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: add --context to dump_dot_merlin

c894fa1

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

apply suggestions from code review

44f10c9

Co-authored-by: Antonio Nuno Monteiro <anmonteiro@gmail.com> Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

merlin: add Select_context.conv

e76a082

Signed-off-by: Javier Chávarri <javier.chavarri@gmail.com>

fix: promote tests after rebase

9ddb4c3

Signed-off-by: Antonio Nuno Monteiro <anmonteiro@gmail.com>

anmonteiro force-pushed the merlin-add-contxts-cmds branch from 1ee8fe2 to 9ddb4c3 Compare May 6, 2024 00:05

refactor: remove Selected_context.t, reuse the context arg, use it …

3295d2f

…in dump Signed-off-by: Antonio Nuno Monteiro <anmonteiro@gmail.com>

anmonteiro merged commit 0471f45 into ocaml:main May 6, 2024
26 of 27 checks passed

jchavarri deleted the merlin-add-contxts-cmds branch May 6, 2024 05:50

gridbugs mentioned this pull request Jun 5, 2024

[new release] dune (15 packages) (3.16.0~alpha1) ocaml/opam-repository#26022

Closed

emillon mentioned this pull request Jun 17, 2024

[new release] dune (15 packages) (3.16.0) ocaml/opam-repository#26101

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

merlin: add new contexts commands #10324

merlin: add new contexts commands #10324

jchavarri commented Mar 28, 2024 •

edited

Loading

voodoos commented Apr 3, 2024

jchavarri commented Apr 3, 2024

jchavarri commented Apr 3, 2024

voodoos Apr 4, 2024

voodoos commented Apr 4, 2024

jchavarri commented Apr 4, 2024

jchavarri commented Apr 10, 2024

jchavarri commented Apr 18, 2024

jchavarri commented Apr 23, 2024

anmonteiro left a comment

anmonteiro Apr 30, 2024

anmonteiro Apr 30, 2024

jchavarri May 2, 2024

jchavarri commented May 2, 2024 •

edited

Loading

anmonteiro commented May 6, 2024

anmonteiro commented May 6, 2024

anmonteiro commented May 6, 2024

merlin: add new contexts commands #10324

merlin: add new contexts commands #10324

Conversation

jchavarri commented Mar 28, 2024 • edited Loading

Current version

Original version (now obsolete)

voodoos commented Apr 3, 2024

jchavarri commented Apr 3, 2024

jchavarri commented Apr 3, 2024

voodoos Apr 4, 2024

Choose a reason for hiding this comment

voodoos commented Apr 4, 2024

jchavarri commented Apr 4, 2024

jchavarri commented Apr 10, 2024

jchavarri commented Apr 18, 2024

jchavarri commented Apr 23, 2024

anmonteiro left a comment

Choose a reason for hiding this comment

anmonteiro Apr 30, 2024

Choose a reason for hiding this comment

anmonteiro Apr 30, 2024

Choose a reason for hiding this comment

jchavarri May 2, 2024

Choose a reason for hiding this comment

jchavarri commented May 2, 2024 • edited Loading

anmonteiro commented May 6, 2024

anmonteiro commented May 6, 2024

anmonteiro commented May 6, 2024

jchavarri commented Mar 28, 2024 •

edited

Loading

jchavarri commented May 2, 2024 •

edited

Loading