require can't find module by name if it's a symlink anymore

[Songworks]

• Node.js Version: 8.11.1
• OS: Linux Mint 17.3 (based on Ubuntu 14.04)
• Scope (install, code, runtime, meta, other?): code/runtime, I think?
• Module (and version) (if relevant):

I have method that worked fine a couple month ago:

// [...]
resolvePathTo(pathTo) {
  let result = process.cwd();

  try {
   result = path.dirname(require.resolve(`${pathTo}${path.sep}package`));
  } catch (error) {
   if (error.code !== 'MODULE_NOT_FOUND') {
     throw error;
   }
  }

  return result;
}

// Example usage:
resolvePathTo('anything'). // Returns value from process.cwd()
resolvePathto('@vendor/module') // Returns path to installed module '@vendor/module'

It'd basically return current working directory path (the module calling the method) by default or the directory path to a module with the name from the methods parameter, if it is installed somewhere in the paths require.resolve looks in.

Unfortunately the code doesn't work anymore for looking up a module that is a symlink (i.e. setup with npm link). It works fine if the module is an actual directory and not a symlink though.

Example directory setup:

stefan@Work-Stefan  /srv/projects/tests/songworks/module $ ls -la node_modules/@songworks/
total 52K
drwxr-xr-x   4 stefan stefan 4.0K Apr 14 05:38 ./
drwxr-xr-x 848 stefan stefan  36K Apr 14 05:25 ../
drwxr-xr-x   3 stefan stefan 4.0K Apr 14 03:23 testsuite/
drwxr-xr-x   3 stefan stefan 4.0K Apr 14 03:23 util/
lrwxrwxrwx   1 stefan stefan   31 Apr 14 05:38 module -> /srv/projects/songworks/module/

require.resolve can resolve the module @songworks/testsuite, but not @songworks/module (anymore).

I can still make the method work if I give require.resolve an absolute path to the module symlink, with f.e. path.resolve('node_modules', pathTo, 'package'). But that kinda defeats the purpose of using require.resolve to begin with.

Also, the code worked fine in at least node 8.9.3, as that's the version I last worked on this project with.

Any idea what I'm doing wrong or why it doesn't work with symlinked modules anymore? What can I do to figure this out?

Thanks in advance!

[ryzokuken]

cc @nodejs/modules

[ljharb]

How are you running node? There's a --preserve-symlinks option that may be relevant.

[Songworks]

@ljharb Through webpack. My code is part of a small module that returns a Promise resolving to a webpack configuration.

Now that you mentioned that. I did update webpack from 3 to 4. That may be part of the issue.
I'll write a small script for testing the code independent of any other module and will be back with the results. :)

[ljharb]

I would be willing to bet that webpack 4 changed its default symlink behavior; but if something happens when using webpack, it's definitely a webpack issue and not a node one :-)

[Songworks]

Thanks for the idea!

I tried it out in a small script* and it properly resolves symlinks. Seems to be issue with Webpack and will follow up there.

Additionally, I also couldn't figure out why require.resolve.paths() was undefined. But the function works fine outside of Webpack.

* Test script:

/* eslint-disable no-console */
const path = require('path');

function resolvePathTo(pathTo) {
  let result = process.cwd();

  try {
    console.log('Resolving', `${pathTo}${path.sep}package`);
    result = path.dirname(require.resolve(`${pathTo}${path.sep}package`));
  } catch (error) {
    if (error.code !== 'MODULE_NOT_FOUND') {
      throw error;
    }
  }
  return result;
}

console.log('resolves to cwd', resolvePathTo('anything'));
console.log('resolves directory-module', resolvePathTo('directory-module'));
console.log('resolves symlink-module', resolvePathTo('symlink-module'));
console.log('resolves original-module', resolvePathTo('original-module'));
console.log(require.resolve.paths('symlink-module'));

Directory structure (each containing a package.json):

stefan@Work-Stefan /srv/projects/tests/symlinks $ ll node_modules/
total 16K
drwxr-xr-x 4 stefan stefan 4.0K Apr 14 19:30 ./
drwxr-xr-x 3 stefan stefan 4.0K Apr 14 19:21 ../
drwxr-xr-x 2 stefan stefan 4.0K Apr 14 19:20 directory-module/
drwxr-xr-x 2 stefan stefan 4.0K Apr 14 19:21 original-module/
lrwxrwxrwx 1 stefan stefan   15 Apr 14 19:21 symlink-module -> original-module/
stefan@Work-Stefan /srv/projects/tests/symlinks $ 

[Songworks]

After further testing I found the 'error'. Not in webpack or node but on my misunderstanding of how node/require walks up/through directories.

Anyway, two small modules to illustrate why it doesn't and shouldn't have ever worked:

[...]/module-a/index.js:

/* eslint-disable no-console */
const moduleB = require('module-b');

console.log(module.paths);
try {
  console.log('resolved', require.resolve('module-b'));
} catch (e) {
  console.log('failed resolve module-b');
}

console.log(moduleB());

[...]/module-b/index.js:

/* eslint-disable no-console */

module.exports = () => {
  console.log(module.paths);
  try {
    // module.paths = module.paths.concat(module.parent.paths);
    console.log('resolved', require.resolve('module-b'));
  } catch (e) {
    console.log('failed resolve module-b');
  }

  return 'module-b result';
}

Symlink is: [...]/module-a/node_modules/module-b -> ../../module-b

Here's the output of running module-a:

stefan@Work-Stefan /srv/projects/issues/resolve-within-symlink/module-a $ node index.js 
module-a
[ '/srv/projects/issues/resolve-within-symlink/module-a/node_modules',
  '/srv/projects/issues/resolve-within-symlink/node_modules',
  '/srv/projects/issues/node_modules',
  '/srv/projects/node_modules',
  '/srv/node_modules',
  '/node_modules' ]
resolved /srv/projects/issues/resolve-within-symlink/module-b/index.js
module-b
[ '/srv/projects/issues/resolve-within-symlink/module-b/node_modules',
  '/srv/projects/issues/resolve-within-symlink/node_modules',
  '/srv/projects/issues/node_modules',
  '/srv/projects/node_modules',
  '/srv/node_modules',
  '/node_modules' ]
failed resolve module-b
module-b result

The issue is that module-b is looking for itself in the completely wrong directory tree. However that's only the case if it's a symlink. If it were an actual directory like a properly installed module, require would start to look in [...]/module-a/node_modules/module-b/node_modules and recursively continue up the tree, resolving it in the parent.

So now I understand why it doesn't work and my best guess to why it worked before is that at the time each of my projects had their node_modules directory as a symlink to the same "master node_modules". So naturally my "module-b" could resolve itself in there.
Eventually I stopped doing the whole 'all linking to the same node_modules' but I can't remember if I was still working on the specific project afterwards...

Anyway, wasted a few hours on this but at least I learned something.

PS: To whomever it may concern: Thanks for creating node!