Add support for mocking Redis when in development mode.

[lemiesz]

When developing features locally, I am forced to either connect to my hosted redis cluster, or run redis locally. For my purposes this is unnecessary, and adds additional setup time. Would be nice to include a flag for just using a in memory queue for development and testing.

const devQueue = new Queue('devQueue', {useDevelopmentQueue: true})

This would implement the same interface as Redis, but allow for storing events in process memory.

[lemiesz]

Real talk though, how do I do run unit tests without something like this. My CI pipelines dont allow external connections.

        const redis = new RedisMock();
            return new Queue(queueName, {
                connection: redis,
            });

This apparently doesn't work. Queue will still attempt to connect to a external Redis instnace.

[eric-hc]

Do you think something like ioredis-mock would work?

[manast]

no, it won't.

[eric-hc]

The implementation would have to require a change to bullmq? Just trying to find a solution and have also seen this module, but I just don't have too much experience working with Redis

[manast]

None of the mocks available support lua scripts. The one that you propose for example: yeahoffline/redis-mock#121

[eric-hc]

@lemiesz this could help for now

[lemiesz]

@convenientstop
Thanks so actually I tried using the ioredis-mock library but it was causing issue because I was doing it before this change dd466b3

I think it should work now thanks

[eric-hc]

Interesting, I was still running into issues and thought I had updated to a version that has that change. Would you mind sharing your implementation or an example here?

[phrozen]

I'll probably have to drop BullMQ from our project, the thing is as much as you want it to work with tests, whenever some check fails with the connection you provide (mocked or otherwise), it uses it's own IORedis to connect to defaults.

If I provide an IORedis connection and you don't like it somehow, or it doesn't work, please fail gracefully, throw an error or something instead of going rogue and silently try to succeed with your own means behind the user's control.

[lemiesz]

Interesting, I was still running into issues and thought I had updated to a version that has that change. Would you mind sharing your implementation or an example here?

I forgot and never actually tried to implement this. Maybe I was wrong and it does not work.

[prithvin]

Not sure if this helpful, but heres a mock we are using for bullmq 1.4.2. We aren't extensively unit testing things like attempts / exponential backoff, etc. so only using this mock for the barebones unit tests:

const EventEmitter = require('events');

const bullEmitter = new EventEmitter();

type JobDataT = Object;

export class Worker {
  constructor(queueName: string, callback: ({ name: string, data: JobDataT }) => {}) {
    bullEmitter.on(queueName, (eventType, data) => {
      callback({ name: eventType, data });
    });
  }
}

export class Queue {
  queueName: string;

  constructor(queueName: string) {
    this.queueName = queueName;
  }

  async add(
    eventType: string,
    payload: JobDataT,
    jobMetadata: { delay: null | void | number, ... },
  ) {
    if (jobMetadata.delay != null) {
      const jobDelay = jobMetadata.delay;
      await new Promise(r => setTimeout(r, jobDelay));
    }
    bullEmitter.emit(this.queueName, eventType, payload);
  }
}

[boredland]

So we would just need to make sure bullmq assumes ioredis-mock is a valid redis instance,, I think. This currently works for me just fine:

import * as bullUtils from 'bullmq/dist/utils';

jest.mock('ioredis', () => require('ioredis-mock/jest'));
const bullMock = jest.spyOn(bullUtils, 'isRedisInstance');

bullMock.mockImplementation((_obj: unknown) => {
  return true;
});

If ioredis-mock is missing ioredis-features that we need to test bullmq properly, we should raise that upstream with them, thats imo nothing bullmq should solve.

[manast]

Highly unlikely they will implement lua scripting in ioredis-mock though...

[ccollie]

@manast according to https://github.com/stipsan/ioredis-mock/blob/HEAD/compat.md evalsha is supported

[kasir-barati]

Guys I have nausea after trying to mock BullMQ, it is just working irrationally. I mean I have it working perfectly fine in another project and now I just copied and pasted the same code to another proj and it is not. I also checked their versions and they are exactly the same, here is the freaking error message which does not make sense to me:

console.error
    Error: connect ECONNREFUSED 127.0.0.1:6379
        at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1615:16) {
      errno: -111,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '127.0.0.1',
      port: 6379
    }

      at Worker.emit (../node_modules/bullmq/src/classes/queue-base.ts:121:17)
      at Worker.emit (../node_modules/bullmq/src/classes/worker.ts:323:18)
      at RedisConnection.<anonymous> (../node_modules/bullmq/src/classes/queue-base.ts:67:56)
      at EventEmitter.RedisConnection.handleClientError (../node_modules/bullmq/src/classes/redis-connection.ts:105:12)
      at EventEmitter.silentEmit (../node_modules/ioredis/built/Redis.js:482:30)
      at Socket.<anonymous> (../node_modules/ioredis/built/redis/event_handler.js:196:14)

[manast]

@kasir-barati it simply seems that it is trying to connect to a Redis instance that is not running in your local machine.

[kasir-barati]

@manast Thanks for the quick reply. TBH I am cracked this nut and learned that this is all because of NestJS doing some magic behind the scene. But what makes me wonder if something is missing in the BullMQ or NestJS is the fact that I have a Redis instance up and running as I explained it here in dev.to. You can also see my repo here: https://github.com/kasir-barati/bugs/tree/bullmq-worker-needs-connection

So long story short @Process decorator should have been able to see my Redis instance which I just ran using Testcontainers.

[manast]

@kasir-barati that error is quite clear, there is no Redis instance running in localhost at port 6379.

[kasir-barati]

First thanks for the reply, second I have added a new test case just to prove that we have a Redis instance up and running: https://github.com/kasir-barati/bugs/blob/6a0a8a1129d42b935ba6e8c0f683830a066e2d0f/test/cart.e2e-spec.ts#L9-L30

I am not sure if you read it or not but in my original comment I also mentioned it, I have a Redis instance running.

[manast]

Ok, so you have a Redis instance running locally but BullMQ fails to connect to it, so what's your theory? my guess is as good as yours, but one thing I know, this is not a bug in BullMQ.

[manast]

Locking this conversation as it is off topic.