Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Newrelic + Fastify: Function.prototype.apply was called on #<Object>, which is an object and not a function #2155

Closed
gevalo1 opened this issue Apr 22, 2024 · 18 comments · Fixed by #2168
Assignees
Labels

Comments

@gevalo1
Copy link
Contributor

gevalo1 commented Apr 22, 2024

Description

We're converting our project from CJS to ESM, and are encountering some issues with Newrelic & its Fastify instrumentation.
Not sure if relevant, but we also integrate @apollo/server with fastify using https://www.npmjs.com/package/@as-integrations/fastify.

Troubleshooting or NR Diag results

{
  "error.message": "Function.prototype.apply was called on #<Object>, which is an object and not a function",
  "error.stack": "TypeError: Function.prototype.apply was called on #<Object>, which is an object and not a function\n    at wrappedFastifyModule (/.../node_modules/.pnpm/newrelic@11.15.0/node_modules/newrelic/lib/instrumentation/fastify.js:99:37)\n    at createFastifyInstance (file:///.../dist/src/utils/fastify/fastify.js:27:17)\n    at start (file:///.../dist/src/server.js:45:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
}

The above error is the main error message, but we're also seeing the following message since we migrated to ESM.

{
  "message": "New Relic for Node.js in worker_threads is not officially supported. Not starting! To bypass this, set `config.worker_threads.enabled` to true in configuration.",
}

How can it be that we didn't get this message before?

Steps to Reproduce

We encountered this when migrating our project from CJS to ESM. Everything worked fine before.

node --experimental-loader newrelic/esm-loader.mjs -r newrelic ./index.js

If/when I'm able to create a reproduction I can share, I will link it here.

Your Environment

Node version: 20.11.1
Docker Image: node:20.11.1-alpine
Newrelic 11.15.0 (same error on 11.14.0)
Fastify: 4.26.2
@fastify/cors: 9.0.1
@fastify/helmet: 11.1.1
@fastify/type-provider-typebox: 4.0.0
@apollo/server: 4.10.2
@as-integrations/fastify: 2.1.1
@newrelic/apollo-server-plugin: 5.1.0

newrelic.cjs

module.exports.config = {
  agent_enabled: process.env.NEW_RELIC_LICENSE_KEY !== undefined,
  app_name: process.env.NEW_RELIC_APP_NAME,
  application_logging: {
    forwarding: {
      enabled: false,
    },
    local_decorating: {
      enabled: true
    },
  },
  distributed_tracing: {
    enabled: true,
  },
  ignore_server_configuration: true,
  labels: process.env.NEW_RELIC_LABELS,
  license_key: process.env.NEW_RELIC_LICENSE_KEY,
  logging: {
    level: 'warning',
    filepath: 'stdout',
  },
  slow_sql: {
    enabled: true,
  },
  transaction_tracer: {
    enabled: true,
    record_sql: 'obfuscated',
  },
};

Additional context

@workato-integration
Copy link

@mrickard
Copy link
Member

We're sorry to hear you're having trouble. A reproduction case would help us pin down what you're seeing, since there are a few moving parts (CJS-to-ESM, Fastify, Apollo).

@gevalo1
Copy link
Contributor Author

gevalo1 commented Apr 23, 2024

Hi @mrickard,

Here is a reproduction case for both error messages: https://github.com/gevalo1/node-newrelic-issue-2155.

Using fastify at all seems to cause the main error: TypeError: Function.prototype.apply was called on #<Object>, which is an object and not a function.

While the logging configuration in newrelic.cjs seems to cause the New Relic for Node.js in worker_threads is not officially supported. Not starting! To bypass this, set config.worker_threads.enabled to true in configuration. error.

@gevalo1
Copy link
Contributor Author

gevalo1 commented Apr 23, 2024

I also added a GitHub Action in that repository to show both errors: https://github.com/gevalo1/node-newrelic-issue-2155/actions.

@jsumners-nr
Copy link
Contributor

You stated that you are seeing this problem with ESM. The provided example seems to be TypeScript. Where is the ESM reproduction?

@gevalo1
Copy link
Contributor Author

gevalo1 commented Apr 23, 2024

I'm not sure if it's an ESM specific problem or not, but we started seeing it when we migrated to ESM.

The example I provided is indeed TypeScript, but it's also ESM. Either way I simplified the example even more by removing TypeScript entirely: https://github.com/gevalo1/node-newrelic-issue-2155.

I've kept the TypeScript code under the typescript branch as well.

@jsumners-nr
Copy link
Contributor

Thank you for the clear reproduction case. It helped greatly with triaging the problem.

@gevalo1
Copy link
Contributor Author

gevalo1 commented Apr 29, 2024

Hi @jsumners-nr & @bizob2828,

I just want to follow-up quickly regarding the reproduction, was the message

New Relic for Node.js in worker_threads is not officially supported. Not starting! To bypass this, set `config.worker_threads.enabled` to true in configuration.

expected there?

@bizob2828
Copy link
Member

This application must spawn worker threads. We do not properly function in worker threads because we do not propagate asynchronous context. In fact we don't want to make the decision of propagating context as executing in worker threads is heavily dependent on your application logic. You can opt in to loading the agent in worker threads but keep in mind that mileage may vary.

@gevalo1
Copy link
Contributor Author

gevalo1 commented Apr 29, 2024

The replication doesn't do anything with worker threads though? Unless either New Relic or Fastify does something with them by default?

@gevalo1
Copy link
Contributor Author

gevalo1 commented May 7, 2024

Hi,

I've tried the latest version (11.16.0), and we are still getting the following warning:

New Relic for Node.js in worker_threads is not officially supported. Not starting! To bypass this, set `config.worker_threads.enabled` to true in configuration.

The replication above also throws this warning but doesn't use any worker threads. Can I safely ignore this warning and will the agent still work as expected?

@mrickard
Copy link
Member

mrickard commented May 7, 2024

@gevalo1 If you take a look at the code path that logs this message, you'll see that it's logged when the agent isn't running on the main thread. As New Relic doesn't support instrumentation for applications not running on the main thread, the agent won't start.

If you ignore the message, the agent won't provide instrumentation. If you override, there's no guarantee of the agent working in the same way that it works with main-thread instrumentation.

@gevalo1
Copy link
Contributor Author

gevalo1 commented May 8, 2024

Hi @mrickard, I understand that the agent doesn't support worker threads.

However my question is: What is causing there to be any worker threads at all?
To demonstrate what I mean I simplified the above replication to only show this warning: https://github.com/gevalo1/node-newrelic-issue-2155/tree/worker-threads.
And you can see the output and resulting warning of running this code here: https://github.com/gevalo1/node-newrelic-issue-2155/actions/runs/9001197998/job/24726926905#step:5:18

What in my reproduction is causing this warning to be thrown? There is practically no code there and still the agent doesn't want to start?
(Ignoring the fact that I didn't push a license key in that repository of course)
What should I change in that replication to make the agent able to start?

@jsumners-nr
Copy link
Contributor

Given the code provided below (the referenced replication), I can run the application like so:

$ node -r newrelic index.js

And get 0 lines referencing threads:

$ cat newrelic_agent.log | jq -rc 'pick(.component,.module,.msg)' | rg thread
# no results

Do you have the NEW_RELIC_WORKER_THREADS environment variable set? How are you starting your application?


index.js
const timeout = 5000;

const start = async () => {
  console.log(`Starting, running for ${timeout/1000} seconds`);

  await new Promise(function (resolve, _reject) {
    setTimeout(function () {
      console.log('Stopping, timeout expired');

      resolve();
    }, timeout);
 });
};

start().catch((error) => {
  console.log('Error', { error });

  process.exit(1);
});
package.json
{
  "dependencies": {
    "newrelic": "^11.16.0"
  }
}
newrelic.js
'use strict'
exports.config = {
  app_name: ['jsumners-playground'],
  license_key: 'redacted',
  logging: {
    level: 'trace'
  },
  
  /**
   * When true, all request headers except for those listed in attributes.exclude
   * will be captured for all traces, unless otherwise specified in a destination's
   * attributes include/exclude lists.
   */
  allow_all_headers: true,
  attributes: {
    /**
     * Prefix of attributes to exclude from all destinations. Allows * as wildcard
     * at end.
     *
     * NOTE: If excluding headers, they must be in camelCase form to be filtered.
     *
     * @name NEW_RELIC_ATTRIBUTES_EXCLUDE
     */
    exclude: [
      'request.headers.cookie',
      'request.headers.authorization',
      'request.headers.proxyAuthorization',
      'request.headers.setCookie*',
      'request.headers.x*',
      'response.headers.cookie',
      'response.headers.authorization',
      'response.headers.proxyAuthorization',
      'response.headers.setCookie*',
      'response.headers.x*'
    ]
  }
}

@gevalo1
Copy link
Contributor Author

gevalo1 commented May 8, 2024

I see, when I run node -r newrelic ./src/server.js there are indeed no warnings, however with node --experimental-loader newrelic/esm-loader.mjs -r newrelic ./src/server.js the warnings are back.

The documentation contains a step to use the newrelic esm-loader: https://docs.newrelic.com/docs/apm/agents/nodejs-agent/installation-configuration/es-modules/#add-loader.
Is this no longer necessary?

@bizob2828
Copy link
Member

bizob2828 commented May 8, 2024

You will definitely need that if you're using ESM. My guess is you're seeing that because the loader runs in a separate thread. I would just ignore that warning

@jsumners-nr
Copy link
Contributor

Node.js's loader API spawns threads to handle module loading. The -r newrelic gets passed to every thread started by the process. So this warning can be safely ignored.

@gevalo1
Copy link
Contributor Author

gevalo1 commented May 31, 2024

Just to clarify still, with ignoring the warning do you mean I need to set config.worker_threads.enabled?

@bizob2828 bizob2828 moved this to Done: Issues recently completed in Node.js Engineering Board Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
5 participants