Dependencies not installed in production

[adamreisnz]

Ref: #761
Moving this particular issue from 761 into it's own issue.

Build is failing on Heroku, but only for production environment. Caching is disabled.

-----> Node.js app detected
-----> Creating runtime environment
       
       NPM_CONFIG_LOGLEVEL=error
       NPM_CONFIG_PRODUCTION=true
       NODE_ENV=production
       NODE_MODULES_CACHE=true
-----> Installing binaries
       engines.node (package.json):  ^7.2.1
       engines.npm (package.json):   unspecified (use default)
       
       Resolving node version ^7.2.1 via semver.io...
       Downloading and installing node 7.2.1...
       Using default npm version: 3.10.10
       Resolving yarn version (latest) via semver.io...
       Downloading and installing yarn (0.18.1)...
       Installed yarn 0.18.1
-----> Restoring cache
       Loading 2 from cacheDirectories (default):
       - node_modules
       - bower_components (not cached - skipping)
-----> Building dependencies
       Installing node modules (yarn)
       yarn install v0.18.1
       [1/4] Resolving packages...
       [2/4] Fetching packages...
       warning fsevents@1.0.15: The platform "linux" is incompatible with this module.
       info "fsevents@1.0.15" is an optional dependency and failed compatibility check. Excluding it from installation.
       [3/4] Linking dependencies...
       [4/4] Building fresh packages...
       error /tmp/build_c86802dccae94b3fb074d3b88f3f63f2/9512deeed23c0eca48d68fb2c8850a28f76692ea/node_modules/bcrypt: Command failed.
       Exit code: 1
       Command: sh
       Arguments: -c node-pre-gyp install --fallback-to-build
       Directory: /tmp/build_c86802dccae94b3fb074d3b88f3f63f2/9512deeed23c0eca48d68fb2c8850a28f76692ea/node_modules/bcrypt
       Output:
       module.js:472
       throw err;
       ^
       
       Error: Cannot find module 'abbrev'
       at Function.Module._resolveFilename (module.js:470:15)
       at Function.Module._load (module.js:418:25)
       at Module.require (module.js:498:17)
       at require (internal/module.js:20:19)
       at Object.<anonymous> (/tmp/build_c86802dccae94b3fb074d3b88f3f63f2/9512deeed23c0eca48d68fb2c8850a28f76692ea/node_modules/nopt/lib/nopt.js:10:14)
       at Module._compile (module.js:571:32)
       at Object.Module._extensions..js (module.js:580:10)
       at Module.load (module.js:488:32)
       at tryModuleLoad (module.js:447:12)
       at Function.Module._load (module.js:439:3)
       info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
-----> Build failed
       
       We're sorry this build is failing! You can troubleshoot common issues here:
       https://devcenter.heroku.com/articles/troubleshooting-node-deploys
       
       Some possible problems:
       
       - A module may be missing from 'dependencies' in package.json
       https://devcenter.heroku.com/articles/troubleshooting-node-deploys#ensure-you-aren-t-relying-on-untracked-dependencies
       
       - This module may be specified in 'devDependencies' instead of 'dependencies'
       https://devcenter.heroku.com/articles/nodejs-support#devdependencies
       
       Love,
       Heroku
       
 !     Push rejected, failed to compile Node.js app.
 !     Push failed

Setting NODE_MODULES_CACHE=false didn't help either.

Here is the dependency tree:

├─┬ babel-cli@6.18.0
│ └─┬ chokidar@1.6.1
│   └─┬ fsevents@1.0.15
│     └─┬ node-pre-gyp@0.6.29
│       └─┬ nopt@3.0.6
│         └── abbrev@1.0.9 
├─┬ google-cloud@0.45.1
│ └─┬ @google-cloud/bigquery@0.6.0
│   └─┬ @google-cloud/common@0.9.1
│     └─┬ grpc@1.0.1
│       └─┬ node-pre-gyp@0.6.31
│         └─┬ nopt@3.0.6
│           └── abbrev@1.0.9 
└─┬ istanbul@1.0.0-alpha.2
  └── abbrev@1.0.9 

I think the problem is the deep dependency via google-cloud. That is a production module, and both babel-cli and instanbul are dev only.

Moreover, when I use yarn why abbrev, it fails to pick up the google-cloud and babel-cli parent dependents:

yarn why v0.18.1
[1/4] 🤔  Why do we have the module "abbrev"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
info Reasons this module exists
   - "istanbul" depends on it
   - "istanbul#nopt" depends on it

As observed by @SimenB istanbul#nopt also looks wrong in the why output, and that seems likely to be the core of this problem. It seems to think that nopt is part of istanbul package, instead of google-cloud and/or babel-cli, and maybe that's why it's not installing it for production environments, because istanbul is not a prod dependency.

Full package.json to reproduce:

"dependencies": {
    "bcrypt": "^1.0.1",
    "bluebird": "^3.4.6",
    "body-parser": "^1.15.2",
    "chalk": "^1.1.3",
    "compression": "^1.6.2",
    "cookie-parser": "^1.4.3",
    "cors": "^2.8.1",
    "express": "^4.14.0",
    "glob": "^7.1.1",
    "google-cloud": "^0.45.1",
    "handlebars": "^4.0.6",
    "html-pdf": "^2.1.0",
    "http-as-promised": "^1.1.0",
    "meanie-express-error-handling": "git+https://github.com/meanie/express-error-handling#2.0.0",
    "meanie-express-github-service": "^2.0.2",
    "meanie-express-jwt-service": "^1.0.2",
    "meanie-express-raven-service": "^1.0.1",
    "meanie-mail-composer": "^1.2.0",
    "meanie-mongoose-only-id": "^1.0.1",
    "meanie-mongoose-set-properties": "^1.0.1",
    "meanie-mongoose-to-json": "^1.1.0",
    "meanie-multer-mime-types-filter": "^1.0.1",
    "meanie-passport-refresh-strategy": "^1.1.2",
    "moment": "^2.17.1",
    "mongoose": "^4.7.3",
    "morgan": "^1.7.0",
    "multer": "^1.1.0",
    "passport": "^0.3.2",
    "passport-http-bearer": "^1.0.1",
    "passport-local": "^1.0.0",
    "phantomjs-prebuilt": "2.1.14",
    "sendgrid": "^4.7.1",
    "sendgrid-mailer": "^1.0.7",
    "socket.io": "^1.7.2",
    "yargs": "^6.5.0"
  },
  "devDependencies": {
    "babel-cli": "^6.16.0",
    "babel-preset-es2015": "^6.18.0",
    "chai": "^3.5.0",
    "chai-as-promised": "^6.0.0",
    "dirty-chai": "^1.2.2",
    "eslint": "^3.12.1",
    "express-simulate-latency": "0.0.2",
    "istanbul": "^1.0.0-alpha.2",
    "mocha": "^3.2.0",
    "mocha-clean": "^1.0.0",
    "nodemon": "^1.11.0",
    "sinon": "^1.17.6",
    "sinon-as-promised": "^4.0.0",
    "sinon-mongoose": "^1.3.0"
  }

[chrisdarroch]

Could be another case of #2263?

[adamreisnz]

Looks like they might be related yes 👍

[blexrob]

Oops, posted my analysis of this issue in the #761 thread... #761 (comment)

[adamreisnz]

🎉