[BUG] pgvecto.rs: IPC connection is closed unexpected.

[mglants]

The bug

[Nest] 8  - 12/20/2023, 1:24:29 AM   ERROR [JobService] QueryFailedError: pgvecto.rs: pgvecto.rs: IPC connection is closed unexpected.
ADVICE: The error is raisen by background worker errors. Please check the full Postgresql log to get more information.
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async SmartInfoRepository.upsertEmbedding (/usr/src/app/dist/infra/repositories/smart-info.repository.js:110:9)
    at async SmartInfoRepository.upsert (/usr/src/app/dist/infra/repositories/smart-info.repository.js:103:9)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:103:9)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)

The OS that Immich Server is running on

Kubernetes, 1.27

Version of Immich Server

v1.91.4

Version of Immich Mobile App

v1.0.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "8"
    meta.helm.sh/release-name: immich-microservices
    meta.helm.sh/release-namespace: apps
    reloader.stakater.com/auto: "true"
  creationTimestamp: "2023-11-07T23:53:01Z"
  generation: 9
  labels:
    app.kubernetes.io/component: main
    app.kubernetes.io/instance: immich-microservices
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: immich-microservices
    helm.sh/chart: app-template-2.0.3
    helm.toolkit.fluxcd.io/name: immich-microservices
    helm.toolkit.fluxcd.io/namespace: apps
  name: immich-microservices
  namespace: apps
  resourceVersion: "277312664"
  uid: b1aa4cb1-9e7e-4e70-ba5a-c2010d7c5661
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app.kubernetes.io/component: main
      app.kubernetes.io/instance: immich-microservices
      app.kubernetes.io/name: immich-microservices
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: main
        app.kubernetes.io/instance: immich-microservices
        app.kubernetes.io/name: immich-microservices
    spec:
      automountServiceAccountToken: true
      containers:
      - args:
        - start-microservices.sh
        env:
        - name: DB_DATABASE_NAME
          valueFrom:
            secretKeyRef:
              key: dbname
              name: immich-pgsql-pguser-immich
        - name: DB_HOSTNAME
          valueFrom:
            secretKeyRef:
              key: host
              name: immich-pgsql-pguser-immich
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: immich-pgsql-pguser-immich
        - name: DB_PORT
          value: "5432"
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              key: user
              name: immich-pgsql-pguser-immich
        - name: DISABLE_REVERSE_GEOCODING
          value: "true"
        - name: IMMICH_MEDIA_LOCATION
          value: /usr/src/app/upload
        - name: LOG_LEVEL
          value: log
        - name: MICROSERVICES_PORT
          value: "3002"
        - name: REDIS_HOSTNAME
          valueFrom:
            configMapKeyRef:
              key: REDIS_HOSTNAME
              name: immich-configmap
        - name: REDIS_PORT
          valueFrom:
            configMapKeyRef:
              key: REDIS_PORT
              name: immich-configmap
        - name: STAKATER_IMMICH_CONFIGMAP_CONFIGMAP
          value: 0cd40a1ba7a4c36cbfd8a510784480b5a1973ea3
        image: ghcr.io/immich-app/immich-server:v1.91.4
        imagePullPolicy: IfNotPresent
        name: main
        resources:
          limits:
            memory: 8000Mi
          requests:
            cpu: 100m
            memory: 512Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/src/app/.reverse-geocoding-dump
          name: geocoding-dump
        - mountPath: /usr/src/app/node_modules/local-reverse-geocoder/geonames_dump
          name: geoname-dump
        - mountPath: /usr/src/app/upload
          name: library
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 568
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 568
        runAsUser: 568
        supplementalGroups:
        - 44
        - 109
        - 65542
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: immich-microservices
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
      volumes:
      - emptyDir: {}
        name: geocoding-dump
      - emptyDir: {}
        name: geoname-dump
      - name: library
        persistentVolumeClaim:
          claimName: immich-nfs
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-11-07T23:53:01Z"
    lastUpdateTime: "2023-12-20T01:15:12Z"
    message: ReplicaSet "immich-microservices-68ccc9ccf4" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-12-20T01:23:56Z"
    lastUpdateTime: "2023-12-20T01:23:56Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 9
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

### Your .env content

```Shell
- name: DB_DATABASE_NAME
          valueFrom:
            secretKeyRef:
              key: dbname
              name: immich-pgsql-pguser-immich
        - name: DB_HOSTNAME
          valueFrom:
            secretKeyRef:
              key: host
              name: immich-pgsql-pguser-immich
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: immich-pgsql-pguser-immich
        - name: DB_PORT
          value: "5432"
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              key: user
              name: immich-pgsql-pguser-immich
        - name: DISABLE_REVERSE_GEOCODING
          value: "true"
        - name: IMMICH_MEDIA_LOCATION
          value: /usr/src/app/upload
        - name: LOG_LEVEL
          value: log
        - name: MICROSERVICES_PORT
          value: "3002"
        - name: REDIS_HOSTNAME
          valueFrom:
            configMapKeyRef:
              key: REDIS_HOSTNAME
              name: immich-configmap
        - name: REDIS_PORT
          valueFrom:
            configMapKeyRef:
              key: REDIS_PORT
              name: immich-configmap

Reproduction steps

1. Run microservices, any task running is errored with  pgvecto.rs:

Additional information

pgvecto.rs = 0.1.11

[bo0tzz]

This sounds like it's probably a pgvecto.rs issue. Can you post the postgres log?

[VoVAllen]

I'm the developer of pgvecto.rs and sorry to hear that. May I ask what's your system version and kernel version? And does this error happen immediately after upgrade?

[mathieugfortin]

FYI, yesterday I did a fresh reinstall of version of 1.93.3, nuked the DB, nuked the immich photo directory and had the same pgvecto and postgres errors. Asset import worked were not impacted.

I just did a compose down/update stack and now the face detection job runs normally

[YentlFrickx]

I have the same issue, restarted db, was successful, but following error was shown:

"[2024-02-01T18:28:24Z ERROR vectors::bgworker] Panickied. Info: PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Error(\"unknown variant `Cosine`, expected one of `L2`, `Cos`, `Dot`\", line: 1, column: 67)), location: Location { file: \"crates/service/src/utils/file_atomic.rs\", line: 51, col: 52 }, can_unwind: true, force_no_backtrace: false }. Backtrace: disabled backtrace."

After that deleted immich-server pod, that fails to start and then following logs appear from postgres:

"2024-02-01 18:28:51.940 UTC [38] ERROR:  pgvecto.rs: pgvecto.rs: IPC connection is closed unexpected."
"\tADVICE: The error is raisen by background worker errors. Please check the full Postgresql log to get more information."
"2024-02-01 18:28:51.940 UTC [38] STATEMENT:  COMMIT"
"2024-02-01 18:28:51.981 UTC [38] WARNING:  there is no transaction in progress"

Small update: reverting to previous version (v1.93.3) works, so seems to be something in immich (didn't even restart the db pod)

[nothing2obvi]

I'm getting this error on v1.105.1. What do I do?