resolve-missing doesn't find newly defined symbols due to cache

[plexus]

I'm using resolve-missing a lot from clj-refactor.el to add missing require/refer statements to the namespace declaration.

I noticed that when defining new functions and using them from another namespace I had to require them manually, refactor-nrepl would not find them. It would work fine after restarting the Clojure process.

I finally traced it down to the fact that in refactor-nrepl.ns.slam.hound.regrow the symbols are cached, so anything defined/loaded later on gets ignored.

I worked around this by disabling the cache, and now it works great.

(add-hook 'nrepl-connected-hook
          (lambda ()
            (cider-nrepl-request:eval
             "(alter-var-root #'refactor-nrepl.ns.slam.hound.regrow/*cache* (constantly nil))"
             (lambda (x)))))

I would like to help to come up with a general solution, but I'm not sure what the best course of action is. Dropping the caching is one option, but I assume it's there for a reason. Alternatively it would need some kind of invalidation, maybe marking namespaces as "dirty" when evaluating forms in them.

Thoughts?

[expez]

I've noticed the same and I tried taking a look at the cache code. After a quick glance I couldn't figure out why it doesn't work because it checks the cache first and then falls back to whatever it was gonna do without the cache in place.

When we copied this stuff from slamhound the cache was disabled and I only recently enabled it. I did this because it takes around 20s to scan for stuff on the clojure project at work, without caching.

If you want to investigate why 'plan b', when there's a cache miss, doesn't work as it should, that would be great.

[plexus]

I had a closer look. Candidates are found by calling (symbols->ns-syms) which returns a map of Symbol => set of namespace names, e.g.

{'join #{'clojure.set 'clojure.string}
 ,,,}

This whole map gets cached and reused the next time, the only "cache miss" is when this function has never been called before.

As a stopgap we could recompute this when there are no candidates found. This wouldn't work when using a var name that already exists in another namespace, but it would certainly be an improvement.

Ideally we would detect when new vars are defined and amend or expire the cache, but I don't suppose there's a var-defined-hook, that would be too easy :)

[expez]

As a stopgap we could recompute this when there are no candidates found. This wouldn't work when using a var name that already exists in another namespace, but it would certainly be an improvement.

Yeah, that sounsd like a good idea! Care to take a stab at it?

[plexus]

Alright, I'll give it a shot. I still feel this is a very imperfect solution, as the few cases where it doesn't work will really puzzle people.

It seems nREPL in the end delegates to clojure.main/repl, we could wrap it and mark the corresponding namespaces as dirty.

(def dirty-ns (atom #{}))

(defn wrap-repl [f]
  (fn [& args]
    (swap! dirty-ns conj (ns-name *ns*))
    (apply f args)))

(alter-var-root #'clojure.main/repl wrap-repl)

What do you think?

[expez]

What do you think?

If this works, it looks like a good way to invalidate certain parts of the cache 👍

[benedekfazekas]

@malcolmsparks want to reopen this or file a new one so I don't forget about it? ;)

[malcolmsparks]

Hi @benedekfazekas - I'm not sure this issue is the issue I'm having - so I'll raise a new one now.