Missing postgres query spans in Remix app

[finack]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/remix

SDK Version

8.37.1

Framework Version

React 18.2.0

Link to Sentry event

https://blaide.sentry.io/performance/trace/3eb527c0f79f145a3d7c59e8d52ade3f/?pageEnd&pageStart&project=4508270748368896&source=traces&statsPeriod=1h&timestamp=1731368996.911

Reproduction Example/SDK Setup

You can find a repo to run this at finack/sentry-report.

You will need to

1. set DATABASE_URL and SENTRY_DSN in .env
2. npx drizzle-kit push to initialize database
3. npm run dev:sentry to run the dev server
4. visit localhost:5173/user

I tried upgrading the @opentelemetry/instrumentation-pg to the latest but this did not change anything.

I added a bunch of console.log to the opentelemetry/instrumentation-pg and could see the pg-pool wrappings fire, but not the pg ones. I suspect this is an issue with opentelemetry not detecting the pg module but have not verified that.

Steps to Reproduce

1. Create a request to the database to the user by visiting localhost:5173/user

Expected Result

In the trace I expect to see the postgres queries themselves. Not the query plan, but just the queries being fired and how long they take.

Actual Result

I am seeing the pg-pool connect span, but not any of the postgres query spans.

Debug output

This was ran from npm run build && npm run start

❯ npm run start

> start
> NODE_OPTIONS='--import ./instrumentation.server.mjs' remix-serve ./build/server/index.js

Sentry Logger [log]: Initializing Sentry: process: 63119, thread: main.
Sentry Logger [log]: Integration installed: InboundFilters
Sentry Logger [log]: Integration installed: FunctionToString
Sentry Logger [log]: Integration installed: LinkedErrors
Sentry Logger [log]: Integration installed: RequestData
Sentry Logger [log]: Integration installed: Console
Sentry Logger [log]: Integration installed: NodeFetch
Sentry Logger [log]: Integration installed: OnUncaughtException
Sentry Logger [log]: Integration installed: OnUnhandledRejection
Sentry Logger [log]: Integration installed: ContextLines
Sentry Logger [log]: Integration installed: LocalVariablesAsync
Sentry Logger [log]: Integration installed: Context
Sentry Logger [log]: Integration installed: ProcessAndThreadBreadcrumbs
Sentry Logger [log]: Integration installed: Modules
Sentry Logger [log]: Integration installed: Express
Sentry Logger [log]: Integration installed: Fastify
Sentry Logger [log]: Integration installed: Graphql
Sentry Logger [log]: Integration installed: Mongo
Sentry Logger [log]: Integration installed: Mongoose
Sentry Logger [log]: Integration installed: Mysql
Sentry Logger [log]: Integration installed: Mysql2
Sentry Logger [log]: Integration installed: Redis
Sentry Logger [log]: Integration installed: Postgres
Sentry Logger [log]: Integration installed: Nest
Sentry Logger [log]: Integration installed: Hapi
Sentry Logger [log]: Integration installed: Koa
Sentry Logger [log]: Integration installed: Connect
Sentry Logger [log]: Integration installed: GenericPool
Sentry Logger [log]: Integration installed: Kafka
Sentry Logger [log]: Integration installed: Amqplib
Sentry Logger [log]: Integration installed: LruMemoizer
Sentry Logger [log]: Integration installed: Http
Sentry Logger [log]: Integration installed: Remix
Sentry Logger [log]: Integration installed: _postgresIntegration
Sentry Logger [log]: Running in CommonJS mode.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for diag v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for trace v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for context v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for propagation v1.9.0.
Sentry Logger [debug]: @sentry/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'http' }
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'http' }
Sentry Logger [debug]: @sentry/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'https' }
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'https' }
Sentry Logger [debug]: @opentelemetry/instrumentation-express Applying instrumentation patch for module on require hook {
  module: 'express',
  version: '4.21.1',
  baseDir: '/Users/peter/src/blaide/sentry/node_modules/express'
}
Sentry Logger [debug]: @opentelemetry/instrumentation-pg Applying instrumentation patch for module on require hook {
  module: 'pg-pool',
  version: '3.7.0',
  baseDir: '/Users/peter/src/blaide/sentry/node_modules/pg-pool'
}
[remix-serve] http://localhost:3000 (http://192.168.7.216:3000)
Manual DB Request -> Hello world!
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [log]: [Tracing] Starting sampled root span
  op: < unknown op >
  name: remix.request
  ID: 3a8df69be412ded7
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for LOADER root: true
Sentry Logger [log]: [Tracing] Starting sampled span
  op: < unknown op >
  name: LOADER root
  ID: bcd5d5678631dbc1
  parent ID: 3a8df69be412ded7
  root ID: 3a8df69be412ded7
  root op: http.server
  root description: remix.request user
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for LOADER routes/user: true
Sentry Logger [log]: [Tracing] Starting sampled span
  op: < unknown op >
  name: LOADER routes/user
  ID: e69f404d347a0329
  parent ID: 3a8df69be412ded7
  root ID: 3a8df69be412ded7
  root op: http.server
  root description: remix.request user
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for pg-pool.connect: true
Sentry Logger [log]: [Tracing] Starting sampled span
  op: < unknown op >
  name: pg-pool.connect
  ID: d338d045337f804b
  parent ID: e69f404d347a0329
  root ID: 3a8df69be412ded7
  root op: http.server
  root description: remix.request user
Sentry Logger [log]: [Tracing] Finishing "loader.remix" span "LOADER root" with ID bcd5d5678631dbc1
Sentry Logger [log]: SpanExporter has 1 unsent spans remaining
Sentry Logger [log]: [Tracing] Finishing "< unknown op >" span "pg-pool.connect" with ID d338d045337f804b
Sentry Logger [log]: SpanExporter has 2 unsent spans remaining
Sentry Logger [log]: [Tracing] Finishing "loader.remix" span "LOADER routes/user" with ID e69f404d347a0329
Sentry Logger [log]: SpanExporter has 3 unsent spans remaining
Sentry Logger [log]: [Tracing] Finishing "http.server" root span "remix.request user" with ID 3a8df69be412ded7
Sentry Logger [log]: SpanExporter exported 4 spans, 0 unsent spans remaining
GET /user 200 - - 57.930 ms
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [debug]: Instrumentation suppressed, returning Noop Span
Sentry Logger [log]: [Tracing] Not injecting trace data for url because tracing is suppressed.
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for outgoing requests
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http https instrumentation outgoingRequest
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http http.ClientRequest return request
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for incoming request
Sentry Logger [log]: [Tracing] Inheriting remote parent's sampled decision for remix.request: true
Sentry Logger [log]: [Tracing] Starting sampled root span
  op: < unknown op >
  name: remix.request
  ID: 8866be0935269805
  parent ID: 862d3f93ec95fd95
Sentry Logger [log]: [Tracing] Finishing "http.server" root span "remix.request user" with ID 8866be0935269805
Sentry Logger [log]: SpanExporter exported 1 spans, 0 unsent spans remaining
Sentry Logger [debug]: Instrumentation suppressed, returning Noop Span
Sentry Logger [log]: [Tracing] Not injecting trace data for url because tracing is suppressed.
Sentry Logger [debug]: @sentry/instrumentation-http http instrumentation for outgoing requests
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http https instrumentation outgoingRequest
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http http.ClientRequest return request
GET /__manifest?p=%2Fuser&version=dc502acf 200 - - 2.744 ms
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http outgoingRequest on response()
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http outgoingRequest on end()
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http outgoingRequest on request close()
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http outgoingRequest on response()
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http outgoingRequest on end()
Sentry Logger [debug]: @opentelemetry_sentry-patched/instrumentation-http outgoingRequest on request close()

[chargome]

hey @finack, thanks for the detailed reproduction. Might be just an oversight in the repo but did you actually call the integration?

// in the reproduction repo you do this: integrations: [Sentry.postgresIntegration],
// instead you should do:
integrations: [Sentry.postgresIntegration()],

[finack]

Hi @chargome! Oops. Good catch.

Originally I did not specify any integrations and added this (incorrectly configured). I just tried it with integrations: [Sentry.postgresIntegration()] and without.

Both yield the same outcome with only seeing pg-pool getting wrapped and not seeing the postgres query spans.

@opentelemetry/instrumentation-pg Applying instrumentation patch for module on require hook {
  module: 'pg-pool',
  version: '3.7.0',
  baseDir: '/Users/peter/src/blaide/sentry/node_modules/pg-pool'
}

[chargome]

Ok so sadly no quick fix 😅 – I'll check out your repro today.

[chargome]

@finack I had a go at this today and was able to reproduce it but sadly couldn't pin down the issue exactly. What I did find out using debug: true in the Sentry server config is that somehow the app is running in CommonJs mode (means a global require function is defined) and my suspicion is that this is the reason for the other pg modules not to be patched.

I was able to fix this though by setting the build output in the remix vite plugin to:

remix({
      serverModuleFormat: "cjs",
      serverBuildFile: "index.cjs",
      future: {
        v3_fetcherPersist: true,
        v3_relativeSplatPath: true,
        v3_throwAbortReason: true,
        v3_singleFetch: true,
        v3_lazyRouteDiscovery: true,
      },
    }),

and updating the start command to NODE_OPTIONS='--require ./instrumentation.cjs' remix-serve ./build/server/index.cjs

I hope that unblocks you for now until we have found what the underlying issue is!

[finack]

@chargome I briefly tried this on our production app but I am getting a bunch of build and runtime errors with libraries we are using (eg react-markdown). I will find some time in the next day or so to poke at this and see if I can get our prod app running under cjs.

[bnussman]

Seeing the same behavior in Bun using @sentry/bun@8.38.0

Only pg-pool.connect is being recorded. Not seeing any queries being recorded from the pg client itself

Sentry Logger [log]: Integration installed: Postgres
Sentry Logger [log]: Running in CommonJS mode.
....
Sentry Logger [debug]: @opentelemetry/instrumentation-pg Applying instrumentation patch for module on require hook {
   module: "pg-pool",
   version: "3.7.0",
   baseDir: "/home/banks/Development/beep/node_modules/pg-pool",
}
...
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for pg-pool.connect: true
Sentry Logger [log]: [Tracing] Starting sampled span
  op: < unknown op >
  name: pg-pool.connect
  ID: 88ff5e1a266d5962
  parent ID: 88bd34910ee5d145
  root ID: 1f53d55b9aa325f1
  root op: http.server
  root description: GET /beep.beepsCount

I also tried using a pnpm resolution to pin @opentelemetry/instrumentation-pg to the latest version 0.47.1, but this did not make a difference.

[chargome]

@bnussman thanks for reporting, we'll try to find out if the issue is linked.

[bnussman]

This may not help @finack , but I was able to fix my bun issue by simplily changing

import { Pool } from "pg";

to

const { Pool } = require("pg");

bnussman/beep#110

[onurtemizkan]

Opened a PR upstream that should resolve this: open-telemetry/opentelemetry-js-contrib#2563