Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to nREPL 0.4 #540

Merged
merged 6 commits into from
Aug 4, 2018
Merged

Migrate to nREPL 0.4 #540

merged 6 commits into from
Aug 4, 2018

Conversation

bbatsov
Copy link
Member

@bbatsov bbatsov commented Jul 16, 2018

An initial stab at updating the middleware for nREPL 0.4.

That's going to be part of the 0.19 release (or at least that's the current plan).

Any volunteers to test with using https://github.com/nrepl/lein-nrepl? (use lein install to build the artefact locally)

@bbatsov bbatsov added the wip label Jul 16, 2018
@bbatsov bbatsov requested review from arrdem and SevereOverfl0w July 16, 2018 20:15
@arrdem
Copy link
Contributor

arrdem commented Jul 16, 2018

LGTM, will try to make time to test it locally.

@codecov
Copy link

codecov bot commented Jul 17, 2018

Codecov Report

Merging #540 into master will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #540   +/-   ##
=======================================
  Coverage   73.33%   73.33%           
=======================================
  Files          36       36           
  Lines        2306     2306           
  Branches      136      136           
=======================================
  Hits         1691     1691           
  Misses        479      479           
  Partials      136      136
Impacted Files Coverage Δ
src/cider/nrepl/middleware/slurp.clj 61.11% <ø> (ø) ⬆️
src/cider/nrepl/middleware/content_type.clj 33.33% <ø> (ø) ⬆️
src/cider/nrepl/middleware/refresh.clj 89.71% <ø> (ø) ⬆️
src/cider/nrepl/middleware/track_state.clj 93.45% <ø> (ø) ⬆️
src/cider/nrepl/middleware/version.clj 87.5% <ø> (ø) ⬆️
src/cider/nrepl/middleware/test.clj 58.15% <ø> (ø) ⬆️
src/cider/nrepl/version.clj 100% <ø> (ø) ⬆️
src/cider/nrepl/middleware/profile.clj 59.55% <ø> (ø) ⬆️
src/cider/nrepl/middleware/debug.clj 56.2% <ø> (ø) ⬆️
src/cider/nrepl/middleware/pprint.clj 92.15% <ø> (ø) ⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 270400f...26f3f0f. Read the comment docs.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 17, 2018

The changes seem trivial, but for some reason this doesn't work right. When I try evaluation with nREPL 0.4 alone everything is fine, when I add the middleware I can connect to the server but afterwards evaluation locks up.

So far I couldn't find what exactly is going wrong, so I'd certainly appreciate some help in testing this.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 17, 2018

  id         "7"
  op         "out-subscribe"
  session    "0826418d-8cc2-44b9-9e20-15f917e6417c"
  time-stamp "2018-07-17 11:25:30.601503000"
)
(-->
  id           "8"
  op           "init-debugger"
  session      "0826418d-8cc2-44b9-9e20-15f917e6417c"
  time-stamp   "2018-07-17 11:25:30.615373000"
  print-length 10
  print-length 10
  print-level  10
  print-level  10
)
(<--
  id            "7"
  session       "0826418d-8cc2-44b9-9e20-15f917e6417c"
  time-stamp    "2018-07-17 11:25:30.689880000"
  out-subscribe "0826418d-8cc2-44b9-9e20-15f917e6417c"
  status        ("done")
)
(-->
  id           "9"
  op           "eval"
  session      "0826418d-8cc2-44b9-9e20-15f917e6417c"
  time-stamp   "2018-07-17 11:26:18.215710000"
  code         "10"
  column       7
  content-type "true"
  file         #("*cider-repl localhost(clj)*" 12 21 (ivy-index 0))
  line         45
  ns           "user"
)

That's what happens for me. Everything in the init sequence seems fine. My only guess is that init-debug might be causing some lockup but I haven't checked this yet.

Such a mystery...

@bbatsov
Copy link
Member Author

bbatsov commented Jul 17, 2018

After a bit of trial and error and I've reduced the problem to "something's wrong with track-state". Disabling this middleware fixes the hang and everything else works fine. Now the question is what's wrong with track-state, as everything seems fine to me. Maybe @dpsutton would have some insight, as I recall he spent some "quality" time a while ago debugging it. :-)

@bbatsov
Copy link
Member Author

bbatsov commented Jul 17, 2018

Actually it seems that some things continue to be broken even with track-state disabled (e.g. eldoc). That's one hell of a mystery!

@bbatsov
Copy link
Member Author

bbatsov commented Jul 17, 2018

Likely last update from me for today, as this got me extremely pissed off - seemed like some problem with info, but I can't figure out what exactly's going on. Requests are just timing out, no errors are raised from the middleware and I have been unable to debug it, as the damn lein plugin I wrote is throwing today some bizarre class not found error, when I try to run it outside its project. What was supposed to be an easy tasks managed to frustrate me on an epic scale...

@dpsutton
Copy link
Contributor

One lesson that i learned from the cemerick -> cider transition for piggieback:

Let's make a release of the new nrepl under the new namespaces that is identical in all respects except for the namespaces. This makes sure that everyone can move to the new nrepl and the tooling doesn't need to case on one or the other. that was a pain.

@dpsutton
Copy link
Contributor

sorry just noticed you've done exactly that. just getting up to speed on the nrepl changes. sorry.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

No problem. Obviously dealing with this is not my dream in life, but it's the only way to fix the fundamental problems. Otherwise we can just pack up shop even now, because I'm done dealing with the limitations of the legacy nREPL and I'd rather just abandon the work on CIDER completely instead of wasting my time.

Btw, if you look at the version in the branch you'll see it's 0.19, instead of 0.18 because I want to work on this in parallel with 0.18 and give people so time to adjust other related packages.

I'm well aware there'll be some frustration after this release, but I didn't think the frustration would be not being able to even update the middlewares. :-)

@dpsutton
Copy link
Contributor

[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-apropos, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-classpath, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-complete, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-debug, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-enlighten, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-format, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-info, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-inspect, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-macroexpand, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-slurp, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-ns, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-out, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-content-type, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-slurp, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-pprint, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-pprint-fn, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-profile, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-refresh, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-resource, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-spec, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-stacktrace, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-test, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-trace, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-tracker, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-undef, see clojure.tools.middleware/set-descriptor!
[WARNING] No nREPL middleware descriptor in metadata of #'cider.nrepl/wrap-version, see clojure.tools.middleware/set-descriptor!
nREPL server started on port 38431 on host :: - nrepl://:::38431
Jul 18, 2018 12:15:49 AM clojure.tools.logging$eval429$fn__432 invoke
SEVERE: Unhandled REPL handler exception processing message {:op out-subscribe, :session 7c5592c7-43f3-4aff-9f23-04acbe3c8faf, :id 7}
java.lang.IllegalAccessError: newline? is not public, compiling:(clojure/tools/reader/reader_types.clj:1:1)

lots of info in the *nrep-server fizzbuzz* window for me.

@dpsutton
Copy link
Contributor

and that last line is very strange:
java.lang.IllegalAccessError: newline? is not public, compiling:(clojure/tools/reader/reader_types.clj:1:1)

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

@dpsutton Are you start nREPL 0.2 or 0.4? You'd be seeing such warnings with 0.2, but not with 0.4.

@dpsutton
Copy link
Contributor

i'm not positive. i'm trying to lein install nREPL 0.4.1 and the cider-nrepl with this patch applied. easy to get lost in this stuff :)

I had to modify CIDER's version string to 0.19-snapshot so it would ask for that installed version of cider-nrepl which would use the new nREPL.

Quite possible i have a missing link in there somewhere.

@dpsutton
Copy link
Contributor

ok yes i am seeing tools nrepl 0.2.x showing up in m2. not sure where i've gone wrong

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

@dpsutton I think it's easiest to use https://github.com/nrepl/lein-nrepl to start nREPL 0.4. I do:

  • lein install for cider-nrepl 0.19-SNAPSHOT (the branch of this PR)
  • add it to the deps of some random project
  • do $ lein nrepl :middleware "['cider.nrepl/cider-middleware]" there
  • finally cider-connect to the running server

You'll get a warning in CIDER about the version mismatch, but obviously you can safely ignore that.

@dpsutton
Copy link
Contributor

[dan@fedora fizzbuzz]$ lein nrepl :middleware "['cider.nrepl/cider-middleware']"
(:middleware ['cider.nrepl/cider-middleware'])
Exception in thread "main" java.lang.ClassNotFoundException: leiningen.nrepl, compiling:(/tmp/form-init7845473687617269713.clj:1:73)

not sure what's up with that. i'll have to look in the morning. its quite late now. I'm gonna dig into this with you @bbatsov . This sucks its being so strange.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

not sure what's up with that.

Seems like a bug in the plugin I've created, although I can't figure this one out as well. It's just a super standard one file lein plugin.

@@ -110,7 +110,7 @@

(defn flag-tooling
"Walk the call stack from top to bottom, flagging frames below the first call
to `clojure.lang.Compiler` or `clojure.tools.nrepl.*` as `:tooling` to
to `clojure.lang.Compiler` or `nrepl.*` as `:tooling` to
distinguish compilation and nREPL middleware frames from user code."
[frames]
(let [tool-regex #"^clojure\.lang\.Compiler|^clojure\.tools\.nrepl|^cider\."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the clojure\.tools\.nrepl in the regex also be changed to just nrepl?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it should be changed.

@reedho
Copy link

reedho commented Jul 18, 2018

I got same error as above testing with https://github.com/nrepl/lein-nrepl

 /p/t/randos $ lein with-profile production nrepl
nil
Exception in thread "main" java.lang.ClassNotFoundException: leiningen.nrepl, compiling:(/private/var/folders/vt/k15mc3ln5vx1n7_9x9mqh7q40000gn/T/form-init14967222619861060167.clj:1:126)

Exploring other option, using clj deps (without lein) seems there is no problem at all, can connect and repl-ing normally, midleware work as expected. The gist i put here: https://gist.github.com/reedho/3d3aeb805f20fc6f3aec03881c74cab4

My best guess is that lein always injecting old org.clojure/tools.nrepl for non production profile.

@dpsutton
Copy link
Contributor

I managed to get the contents of that temp file that is throwing the error:

(.deleteOnExit (java.io.File. "/tmp/form-init2271818862159530970.clj")) (do (set! *warn-on-reflection* nil) nil (leiningen.nrepl/start-nrepl {:middleware [(quote cider.nrepl/cider-middleware')]}))

@dpsutton
Copy link
Contributor

to contrast, other tmp files have

(try (clojure.core/require (quote cider.nrepl)) (catch java.lang.Throwable t__11809__auto__ (clojure.core/println "Error loading" (clojure.core/str (quote cider.nrepl) ":") (clojure.core/or (.getMessage t__11809__auto__) (clojure.core/type t__11809__auto__)))))

in them. not sure why these have a created require message in them but that would explain the "what the heck is leiningen.repl" message we are seeing

@dpsutton
Copy link
Contributor

Inside the eval-in-project call the project's own classpath will be active and Leiningen's own internals and plugins will not be available.

https://github.com/technomancy/leiningen/blob/master/doc/PLUGINS.md#evaluating-in-project-context

this seems to be an abuse of eval in project apparently.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

@dpsutton Yeah, I also figured this out in the morning (and not just me nrepl/lein-nrepl#1). I really wished this produced a more meaningful error, though. Now I'm wondering what's the best course of action. Probably it's best to add this dep if it's not in the project deps, but I'm not a Lein expert and I don't know what's the best strategy there.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

My best guess is that lein always injecting old org.clojure/tools.nrepl for non production profile.

@reedho Wow! You're probably right! Lein has a hard dep on the old nREPL, but I didn't think it could mess something up because of the different namespaces. Likely, however, that's not completely true.

@SevereOverfl0w
Copy link
Contributor

I didn't think it could mess something up because of the different namespaces. Likely, however, that's not completely true.

I'm not sure what else could collide here? I suppose this can easily be verified via the clj tools.

@SevereOverfl0w
Copy link
Contributor

❯ clj -Sdeps '{:deps {nrepl {:mvn/version "0.4.1"} org.clojure/tools.nrepl {:mvn/version "0.2.13"}}}' -m nrepl.cmdline --interactive
network-repl
Clojure 1.9.0
user=> (+ 1 1)
2

Nope, you can have new & old nrepl alongside one another.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

I'm not sure what else could collide here? I suppose this can easily be verified via the clj tools.

@SevereOverfl0w Yeah, that's what I plan to do when I find some time. But I did notice yesterday I can't jack in the nREPL project, so I guess something goes sideways when the two jars nREPLs are present.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

@SevereOverfl0w For me the problems started only after adding the cider-nrepl middleware (and not everything was broken, just some things). I guess you should also load some middleware and see if var-info is working.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

clj -Sdeps '{:deps {cider/cider-nrepl {:mvn/version "0.19.0-SNAPSHOT"} }}' -e '(require (quote cider-nrepl.main)) (cider-nrepl.main/init ["cider.nrepl/cider-middleware"])' works like a charm. All the middleware are fine.

Guess @reedho is right and something's getting messed up because of lein's nREPL dep. At least we're making progress. Hopefully with Boot there will be no such problems (I'll leave it to @arichiardi to test this). Now we have to figure out exactly what's going wrong with lein nrepl.

@SevereOverfl0w
Copy link
Contributor

I don't think it is because of the nrepl dep:

❯ clj -Sdeps '{:deps {nrepl {:mvn/version "0.4.1"} org.clojure/tools.nrepl {:mvn/version "0.2.13"} cider/cider-nrepl {:mvn/version "0.17.0"}}}' -e '(require (quote cider-nrepl.main)) (cider-nrepl.main/init ["cider.nrepl/cider-middleware"])'
nREPL server started on port 35557 on host 0:0:0:0:0:0:0:0 - nrepl://0:0:0:0:0:0:0:0:35557

@bbatsov
Copy link
Member Author

bbatsov commented Jul 18, 2018

Well, for some reason it works fine like this and it doesn't work with lein nrepl - what else can I think? :-)

@reedho
Copy link

reedho commented Jul 19, 2018

Just checked this pull (4bf683b) against latest nrepl/lein-nrepl@366bbea, things works fine.

 /p/t/cider-nrepl (CIDER-NREPL-019) $ java -version                                                                                                         java version "10.0.1" 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)

 /p/t/cider-nrepl (CIDER-NREPL-019) $ git log -1 --oneline
4bf683b (HEAD -> CIDER-NREPL-019, origin/nrepl-0.4) Bump the nREPL dep to 0.4.2

 /p/t/cider-nrepl (CIDER-NREPL-019) $ lein install
Created /private/tmp/cider-nrepl/target/cider-nrepl-0.19.0-SNAPSHOT.jar
Wrote /private/tmp/cider-nrepl/pom.xml
Installed jar and pom into local repo.

---

 /p/t/lein-nrepl $ git log -1 --oneline
366bbea (HEAD -> master, origin/master, origin/HEAD) Kill a debug println
 /p/t/lein-nrepl $ lein install
Created /private/tmp/lein-nrepl/target/lein-nrepl-0.1.0-SNAPSHOT.jar
Wrote /private/tmp/lein-nrepl/pom.xml
Installed jar and pom into local repo.

---

 /p/t/randos $ cat project.clj
(defproject randos "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.9.0"]
                 [nrepl "0.4.2"]]
  :plugins [[nrepl/lein-nrepl "0.1.0-SNAPSHOT"]
            [cider/cider-nrepl "0.19.0-SNAPSHOT"]])

 /p/t/randos $ lein nrepl :middleware "['cider.nrepl/cider-middleware]"
nREPL server started on port 63727 on host localhost - nrepl://localhost:63727

---

M-x cider-browse-ns-all
All loaded namespaces
  cljs-tooling.complete
  cljs-tooling.info
  cljs-tooling.util.analysis
  cljs-tooling.util.misc
  clojure.core
  clojure.core.protocols
  clojure.core.server
  clojure.core.specs.alpha
  clojure.edn
  clojure.instant
  clojure.java.browse
  clojure.java.classpath
  clojure.java.io
  clojure.java.javadoc
  clojure.java.shell
  clojure.main
  clojure.pprint
  clojure.reflect
  clojure.repl
  clojure.set
  clojure.spec.alpha
  clojure.spec.gen.alpha
  clojure.stacktrace
  clojure.string
  clojure.template
  clojure.test
  clojure.tools.logging
  clojure.tools.logging.impl
  clojure.tools.namespace.dependency
  clojure.tools.namespace.file
  clojure.tools.namespace.find
  clojure.tools.namespace.parse
  clojure.tools.namespace.track
  clojure.tools.reader
  clojure.tools.reader.default-data-readers
  clojure.tools.reader.impl.commons
  clojure.tools.reader.impl.errors
  clojure.tools.reader.impl.inspect
  clojure.tools.reader.impl.utils
  clojure.tools.reader.reader-types
  clojure.uuid
  clojure.walk
  compliment.context
  compliment.core
  compliment.sources
  compliment.sources.class-members
  compliment.sources.keywords
  compliment.sources.local-bindings
  compliment.sources.namespaces-and-classes
  compliment.sources.ns-mappings
  compliment.sources.resources
  compliment.sources.special-forms
  compliment.utils
  dynapath.defaults
  dynapath.dynamic-classpath
  dynapath.util
  leiningen.nrepl.core
  nrepl.ack
  nrepl.bencode
  nrepl.core
  nrepl.middleware
  nrepl.middleware.interruptible-eval
  nrepl.middleware.load-file
  nrepl.middleware.pr-values
  nrepl.middleware.session
  nrepl.misc
  nrepl.server
  nrepl.transport
  orchard.classloader
  orchard.classpath
  orchard.eldoc
  orchard.info
  orchard.inspect
  orchard.java
  orchard.meta
  orchard.misc
  orchard.namespace
  orchard.spec
  profile.core
  user

@bbatsov
Copy link
Member Author

bbatsov commented Jul 19, 2018

Yep, seems we're all good. I haven't tested only the boot support, but probably everything's fine there as well.

I'll write some notes about using this with CIDER and I'll leave this open for now, until 0.18 is released. @vspinu asked us to postpone the release a bit. In the mean time I'll update piggieback (likely to support both nREPLs) and ping @thheller and @bhauman to add some support for nREPL 0.4 as well. The changes they'll have to make to support both nREPLs should be small.

@bbatsov
Copy link
Member Author

bbatsov commented Jul 19, 2018

Big thanks to everyone who helped with this!

In case someone's curious what the problem turned out to be - the short story is that I wrote a very buggy lein plugin. :-) I really need to read up on Leiningen one of those days...

@arichiardi
Copy link
Contributor

arichiardi commented Jul 20, 2018

Can confirm it works with nrepl-server and clojure-emacs/cider#2379 applied.

Support with tools.nrepl is kept for now with the assumption
that tools.nrepl namespaces would be loaded automatically by
`lein repl` or `boot repl`.

Support for tools.nrepl will likely be removed after a couple of
releases (once mainstream Lein and Boot switch to nREPL 0.4).
@bbatsov bbatsov merged commit 15d8774 into master Aug 4, 2018
@bbatsov bbatsov deleted the nrepl-0.4 branch August 4, 2018 07:51
@bbatsov
Copy link
Member Author

bbatsov commented Aug 6, 2018

The actual work went into 0.18 (15d8774)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants