REPL: script directory-relative requires fail with 'cannot find module'

[nolanw]

If I have hi.coffee in the current directory, I can't use require './hi' in CoffeeScript's REPL to load it. However, it works with node's REPL. For example:

$ touch hi.coffee
$ coffee
coffee> require './hi'
Error: Cannot find module './hi'
    [snip]
$ node
> require("coffee-script"); require("./hi");
{}

[TrevorBurnham]

Looks like this worked under 0.9.4 but hasn't worked since 0.9.5. I'm trying git bisect to isolate the specific commit that broke it, but there are related issues in nearby commits...

[TrevorBurnham]

[Edit: As satyr points out below, this post actually provides a fix to a different bug, not the REPL one.]

Found it. Ah, I'm a little embarrassed—it looks like I introduced this bug (when fixing eval, which wasn't working at all at the time). One can fix it by changing the line in coffee-script.coffee from

root.filename = fs.realpathSync options.fileName or '.'

to

root.filename = if options.fileName then fs.realpathSync(options.fileName) else '.'

which is an awfully inelegant way of saying "Let's not run '.' through fs.realpathSync," which I'd assumed was a good thing... apparently it isn't.

[nolanw]

Right on, thanks Trevor!

[satyr]

root.filename = fs.realpathSync options.fileName or '.'

This is from .run right? I thought REPL uses .eval.

[TrevorBurnham]

Ah, quite right satyr—what I posted does fix a bug, but it's a different bug! Try this:

touch hi.coffee
coffee -e "require './hi'"

On the current master, that gives you an error ("Cannot find module './hi'"); change the line I suggested, and it runs fine.

Still investigating the REPL bug (which, it seems, has been around forever)...

[TrevorBurnham]

So here's the problem: When you run input in the REPL, it goes into this function:

exports.eval = (code, options) ->
  __filename = options.fileName
  __dirname  = path.dirname __filename
  eval compile code, options

If it weren't for these __filename and __dirname changes, then they'd point to wherever the CoffeeScript libs are located; instead, they're set to where the REPL is running.

That's great—but, of course, these lines really just create new variables named __filename and __dirname which shadow the globals. And when Node.js sees require './hi', it looks for a file named hi relative to its own record of __dirname. Even changing the global __dirname doesn't affect it (I tried). So it may be impossible to fix this without either making significant changes to the way CoffeeScript runs eval or accessing some special Node.js API.

In the meantime, nolanw, here's a workaround:

requireLocal = (filename) -> require path.join fs.realpathSync('.'), filename
requireLocal 'hi'

[nolanw]

Trevor, that works great. As a next step I'm trying to make requireLocal available whenever I start the REPL. So far I've tried:

coffee -r ~/.coffeerc
coffee -i ~/.coffeerc

Neither of which seem to run ~/.coffeerc, which consists entirely of console.log 'updog'. (My plan was to get this working, put requireLocal in .coffeerc, then alias coffee -i to whatever's needed to run this script first.)

This isn't all that surprising, as I suppose the -i option could mean "ignore any given scripts and just run a REPL", while -r could mean "require this script iff we're compiling something" (and apparently -r only works on .js files?). Unless I'm doing this wrong.

And that's all good, in which case I ask: is there a way to make requireLocal available in the REPL without my defining it every time or putting it in a file in NODE_PATH and requiring it that way? Just as a workaround for now.

[TrevorBurnham]

Good point, nolanw—it looks like there's no way to do that now. I might raise that as a separate issue...

I got an answer on the Node mailing list, and it turns out there's a pretty easy fix to the original bug: module.filename is used by require to determine the base path, so it just needs to be set equal to __filename.

I've combined this and the other fix mentioned above as a patch here.

(Interestingly, a comment in command.coffee alludes to this—"If evaluating the script directly sets __filename, __dirname and module.filename to be correct relative to the script's path."—but apparently the code has gotten out of sync with the comment, because module.filename isn't referenced anywhere else in the CoffeeScript code base...)

[TrevorBurnham]

I've posted a pull request as issue 1045 for the other issue you pointed out, nolanw. If that patch is merged into master, it'll be possible to require hi and enter the REPL by simply writing

coffee -r ./hi

[nolanw]

Very cool Trevor, you're a star!

[jashkenas]

Thanks, Trevor, well done ... I've merged in your branches here:

7815138

[blaenk]

I previously deleted my comment to allow myself time to read through this issue list. It seems that this has been fixed before, but I'm experiencing the same exact issue in version 1.1.1, so I don't really know what the deal is.

I'll provide any additional information you guys may need.

[michaelficarra]

Re-opening because @blaenk claims this issue has regressed.

[sbowman]

I'm new to both CoffeeScript and Node, but I'm having the same issue with 1.1.1/0.4.6. I don't know if this will help, but the coffee REPL doesn't seem to know about module.paths. Is this just a by-product of the problem, or could it indicate a cause?

coffee> console.log module.paths
undefined

Whereas node REPL seems to know what's going on:

> console.log(module.paths)
[ '/Users/sbowman/Projects/experimental/tutorial/repl/node_modules',
  '/Users/sbowman/Projects/experimental/tutorial/node_modules',
  '/Users/sbowman/Projects/experimental/node_modules',
  '/Users/sbowman/Projects/node_modules',
  '/Users/sbowman/node_modules',
  '/Users/node_modules',
  '/node_modules' ]

If I write a little .coffee script, it knows about node_modules.

paths.coffee

console.log module.paths

When I run paths.coffee, I get the correct module.paths:

sbowman$ coffee paths.coffee
[ '/Users/sbowman/Projects/experimental/tutorial/repl/node_modules',
  '/Users/sbowman/Projects/experimental/tutorial/node_modules',
  '/Users/sbowman/Projects/experimental/node_modules',
  '/Users/sbowman/Projects/node_modules',
  '/Users/sbowman/node_modules',
  '/Users/node_modules',
  '/node_modules' ]

[blaenk]

If you look at the last commit that fixed this: 7815138

You will see that the variables have changed since then, I don't know if that was to mirror changes in the node.js architecture, but I imagine the problem stems from those changes.

[balupton]

Having this issue myself - #1485

[TrevorBurnham]

Sorry for not looking at this sooner. The REPL went through some major changes for 1.1.0 (mostly for the better—thanks Michael), but a few things broke along the way. Here's what went wrong:

When you call require, it uses module.filename to look up relative paths. That's why my patch that set module.filename fixed this issue before. So you might think the solution is to just add

sandbox.module.filename = ...

to the eval function. But, it's a little trickier than that: Normally, require and module are objects that Node.js provides to you. Every file gets its own require and its own module; the two are invisible linked. So setting module.filename only affects require if you're using the module and require provided to the same file. The culprit, therefore, is this code:

sandbox =
require: require
module : { exports: {} }

The sandbox's require object is linked to the module in whatever file it was defined in (repl.coffee if you're using the REPL), and, weirdly, there's no way to access a require object's internal module link.

The solution: Use a real module object. Either reuse the one from the file in which sandbox is defined (and modify its filename), which is how the old code worked; or create a new module and require using Node's internal API. This feels cleaner, though it runs the risk of breaking under future Node versions. I'm creating a pull request that takes the latter approach...

[michaelficarra]

@TrevorBurnham: wow, I've been having some serious coincidences today. I was actually just working on a fix for this issue and coming up with virtually the same solution. I'll take a look at your pull request right now.

[TrevorBurnham]

Ah, sorry for the duplicated effort! Pull request posted at #1487.

[michaelficarra]

Fixed by fff4c9c

[jashkenas]

Looks good -- with @satyr's revisions. Following the Node REPL's example is always progress.

[vitaly]

so.. when will it be released? im still having this problem on 1.1.1/0.4.9

[michaelficarra]

so.. when will it be released?

@vitaly: As soon as @jashkenas packages and tags 1.1.2 (and who knows when that will be at this point). You can just clone master if you want it right now.