incorrect benchmark ? #25

jjmr · 2015-05-05T10:12:09Z

Hi,

I have ran the benchmarks in my local machine with nodejs 0.12.2, node_redis 0.12.1 and the master branch of ioredis and I have got the following results:

$ npm run bench
> ioredis@1.2.0 bench /tmp/ioredis
> matcha benchmarks/*.js

child_process: customFds option is deprecated, use stdio instead.

                  simple set
     109,742 op/s » ioredis
     105,823 op/s » node_redis

                  simple get
      88,683 op/s » ioredis
     102,495 op/s » node_redis

                  simple get with pipeline
      10,600 op/s » ioredis
      13,012 op/s » node_redis

                  lrange 100
      68,533 op/s » ioredis
      72,331 op/s » node_redis

 Suites:  4
 Benches: 8
 Elapsed: 53,973.10 ms

I have ran the benchmark with and without hiredis having similar performance.

Those results show better or equal performance between node_redis and ioredis. However, in the documentation, the performance of ioredis is quite better than node_redis. Any idea?

The text was updated successfully, but these errors were encountered:

luin · 2015-05-05T11:17:43Z

I've run the benchmark in my computers and our servers, and the results are similar. However there are many factors that can impact the results, so it's hard to say why yours are different. I just submit a commit to print system info as well when running benchmark, and here's the result on my laptop(MacBook Pro, Retina, 15-inch, Late 2013):

ioredis git:master ❯ npm run bench                ✹

> ioredis@1.2.0 bench /Users/luin/opensource/ioredis
> matcha benchmarks/*.js

child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.2.0
node_redis: 0.12.1
CPU: 8
OS: darwin x64
==========================

                      simple set
          75,846 op/s » ioredis
          43,185 op/s » node_redis

                      simple get
          75,463 op/s » ioredis
          41,240 op/s » node_redis

                      simple get with pipeline
          12,329 op/s » ioredis
           4,146 op/s » node_redis

                      lrange 100
          58,273 op/s » ioredis
          45,782 op/s » node_redis


  Suites:  4
  Benches: 8
  Elapsed: 61,455.94 ms

Could you pull the latest commit and run the benchmark again?

dirkbonhomme · 2015-05-05T11:29:48Z

I'm also seeing great results:

nodejs: 0.10.25
ioredis: 1.2.0
os: ubuntu x86_64 (Vagrant)

> ioredis@1.2.0 bench /vagrant/
> matcha benchmarks/*.js


                      simple set
         104,896 op/s » ioredis
          55,791 op/s » node_redis

                      simple get
          95,655 op/s » ioredis
          54,011 op/s » node_redis

                      simple get with pipeline
          19,481 op/s » ioredis
           7,032 op/s » node_redis

                      lrange 100
          80,833 op/s » ioredis
          46,986 op/s » node_redis


  Suites:  4
  Benches: 8
  Elapsed: 57,291.53 ms

jjmr · 2015-05-05T12:52:14Z

My becnhmark updated:

 $ npm run bench

> ioredis@1.2.0 bench /tmp/ioredis
> matcha benchmarks/*.js

child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.2.0
node_redis: 0.12.1
CPU: 8
OS: linux x64
==========================

                      simple set
         102,374 op/s » ioredis
         100,401 op/s » node_redis

                      simple get
          69,507 op/s » ioredis
          90,876 op/s » node_redis

                      simple get with pipeline
          10,670 op/s » ioredis
          11,413 op/s » node_redis

                      lrange 100
          57,448 op/s » ioredis
          67,558 op/s » node_redis


  Suites:  4
  Benches: 8
  Elapsed: 54,293.49 ms

My OS is Linux Mint.

If I run the same benchmark more than once I can see different values and in general ioredis and node_redis have similar performance in each suite (more or less)

luin · 2015-05-05T13:04:30Z

Which redis-server version are you using? Is Redis running on the same machine as the client?

jjmr · 2015-05-05T13:40:54Z

redis server is in the same machine and the version is 2.8.4

lpinca · 2015-05-05T15:25:12Z

Node.js v0.12.2, early 2009 MacBook Pro, redis-server 3.0.0 running on the same machine:

macbook:ioredis luigi$ npm run bench

> ioredis@1.2.0 bench /Users/luigi/Desktop/repos/ioredis
> matcha benchmarks/*.js

child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.2.0
node_redis: 0.12.1
CPU: 2
OS: darwin x64
==========================

                      simple set
          23,289 op/s » ioredis
          27,762 op/s » node_redis

                      simple get
          22,474 op/s » ioredis
          26,978 op/s » node_redis

                      simple get with pipeline
           4,312 op/s » ioredis
           2,758 op/s » node_redis

                      lrange 100
          18,529 op/s » ioredis
          22,116 op/s » node_redis


  Suites:  4
  Benches: 8
  Elapsed: 67,757.50 ms

AVVS · 2015-05-06T22:47:20Z

Vitalys-MacBook-Pro:mservice-smtp vitaly$ redis-cli -v
redis-cli 2.8.19

Vitalys-MacBook-Pro:ioredis vitaly$ npm run bench

> ioredis@1.2.0 bench /Users/vitaly/projects/ioredis
> matcha benchmarks/*.js

==========================
ioredis: 1.2.0
node_redis: 0.12.1
CPU: 8
OS: darwin x64
==========================

                      simple set
          86,129 op/s » ioredis
          75,235 op/s » node_redis

                      simple get
          82,437 op/s » ioredis
          74,842 op/s » node_redis

                      simple get with pipeline
          15,923 op/s » ioredis
           8,383 op/s » node_redis

                      lrange 100
          61,766 op/s » ioredis
          54,674 op/s » node_redis


  Suites:  4
  Benches: 8
  Elapsed: 56,919.29 ms

Vitalys-MacBook-Pro:ioredis vitaly$ node -v
v0.10.38

luin · 2015-05-07T01:07:19Z

I added a link to this issue to the benchmark section in README, and will leave this issue open util we know more about the reason of the difference of results.
At a guess, ioredis has better performance when the server is able to handle higher concurrency.

IncSW · 2015-06-24T23:21:33Z

redis-cli 2.8.17
debian jessie

> ioredis@1.5.4 bench /home/tests/ioredis
> matcha benchmarks/*.js

child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.5.4
node_redis: 0.12.1
CPU: 8
OS: linux x64
==========================

                      simple set
         133,532 op/s » ioredis
         156,371 op/s » node_redis

                      simple get
         133,728 op/s » ioredis
         135,891 op/s » node_redis

                      simple get with pipeline
          17,511 op/s » ioredis
          15,210 op/s » node_redis

                      lrange 100
          90,610 op/s » ioredis
         106,528 op/s » node_redis


  Suites:  4
  Benches: 8
  Elapsed: 53,750.60 ms

AVVS · 2015-06-24T23:25:32Z

Updated for mac / iojs 2.2.1. Redis 3.0.1

Vitalys-MacBook-Pro:ioredis vitaly$ npm run bench

> ioredis@1.5.4 bench /Users/vitaly/projects/ioredis
> matcha benchmarks/*.js

child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.5.4
node_redis: 0.12.1
CPU: 8
OS: darwin x64
==========================

                      simple set
         113,225 op/s » ioredis
          88,521 op/s » node_redis

                      simple get
          98,778 op/s » ioredis
          80,953 op/s » node_redis

                      simple get with pipeline
          14,946 op/s » ioredis
           7,907 op/s » node_redis

                      lrange 100
          86,316 op/s » ioredis
          81,688 op/s » node_redis


  Suites:  4
  Benches: 8
  Elapsed: 58,897.17 ms

Vitalys-MacBook-Pro:ioredis vitaly$ node -v
v2.2.1

fantasticsoul · 2015-08-18T06:51:51Z

你好，我本地机器是win7系统，使用npm install ioredis命令下载ioredis到了本地，我本地有redis-server,版本为2.6.12,请问怎样跑这个ioredis/benchmarks/single_node.js文件？直接在ioredis/benchmarks/目录下输入 node single_node.js会报错：
ReferenceError: suite is not defined
at Object. (C:\Users\Administrator\node_modules\ioredis\benchmark
s\single_node.js:30:1)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)
at node.js:814:3

luin · 2015-08-18T07:00:16Z

npm run bench. 文档里有写 https://github.com/luin/ioredis#benchmark

fantasticsoul · 2015-08-18T07:09:24Z

输入npm run bench后回车，依然不成功，控制打印如下：

ioredis@1.7.5 bench C:\Users\Administrator\node_modules\ioredis
matcha benchmarks/*.js

'matcha' 不是内部或外部命令，也不是可运行的程序
或批处理文件。

npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs
\node_modules\npm\bin\npm-cli.js" "run" "bench"
npm ERR! node v0.12.0
npm ERR! npm v2.5.1
npm ERR! code ELIFECYCLE
npm ERR! ioredis@1.7.5 bench: matcha benchmarks/*.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the ioredis@1.7.5 bench script 'matcha benchmarks/.js'.
npm ERR! This is most likely a problem with the ioredis package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! matcha benchmarks/.js
npm ERR! You can get their info via:
npm ERR! npm owner ls ioredis
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! C:\Users\Administrator\node_modules\ioredis\benchmarks\npm-debug.lo
g

然后我npm i matcha后，再次输入npm run bench后回车，还是不成功，控制打印如下：

ioredis@1.7.5 bench C:\Users\Administrator\node_modules\ioredis
matcha benchmarks/*.js

child_process: customFds option is deprecated, use stdio instead.
module.js:338
throw err;
^
Error: Cannot find module 'C:\Users\Administrator\node_modules\ioredis\benchmark
s*.js'
at Function.Module._resolveFilename (module.js:336:15)
at Function.require.resolve (module.js:388:19)
at C:\Users\Administrator\node_modules\ioredis\node_modules\matcha\bin_matc
ha:80:18
at Array.map (native)
at Object.runSuite (C:\Users\Administrator\node_modules\ioredis\node_modules
\matcha\bin_matcha:79:17)
at Program.parse (C:\Users\Administrator\node_modules\ioredis\node_modules\m
atcha\node_modules\electron\lib\electron\program.js:262:27)
at Object. (C:\Users\Administrator\node_modules\ioredis\node_modu
les\matcha\bin_matcha:36:9)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)

npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs
\node_modules\npm\bin\npm-cli.js" "run" "bench"
npm ERR! node v0.12.0
npm ERR! npm v2.5.1
npm ERR! code ELIFECYCLE
npm ERR! ioredis@1.7.5 bench: matcha benchmarks/*.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the ioredis@1.7.5 bench script 'matcha benchmarks/.js'.
npm ERR! This is most likely a problem with the ioredis package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! matcha benchmarks/.js
npm ERR! You can get their info via:
npm ERR! npm owner ls ioredis
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! C:\Users\Administrator\node_modules\ioredis\benchmarks\npm-debug.lo
g

这是肿么了

luin · 2015-08-18T07:15:19Z

npm run 时会设置系统环境变量保证 devdeps 里面的依赖会作为全局依赖。Windows 会有问题。可以直接用 npm i -g matcha

fantasticsoul · 2015-08-18T07:19:43Z

然后再次执行npm run bench

这是为什么呢~~~~(>_<)~~~~

luin · 2015-08-18T07:24:37Z

直接执行 matcha single_node.js 吧

fantasticsoul · 2015-08-18T07:27:56Z

非常感谢，已经ok了，接下来我去centOS跑跑看看结果

zhuangya · 2015-10-26T04:12:41Z

node 4.2.1:

version: pull-request #184

12:09:59 nos:~/Projects/Github/ioredis 13 (redis-dev-dep)
~> npm run bench

> ioredis@1.10.0 bench /Users/nos/Projects/Github/ioredis
> matcha benchmarks/*.js

(node) child_process: options.customFds option is deprecated. Use options.stdio instead.
==========================
ioredis: 1.10.0
node_redis: 2.2.5
CPU: 8
OS: darwin x64
==========================

                      simple set
         111,917 op/s » ioredis
          87,705 op/s » node_redis

                      simple get
         107,242 op/s » ioredis
          86,911 op/s » node_redis

                      simple get with pipeline
           9,105 op/s » ioredis
           8,371 op/s » node_redis

                      lrange 100
          10,572 op/s » ioredis
           8,917 op/s » node_redis

                      publish
         108,210 op/s » ioredis
          87,479 op/s » node_redis

                      subscribe
          48,086 op/s » ioredis
          46,994 op/s » node_redis


  Suites:  6
  Benches: 12
  Elapsed: 89,436.12 ms

shershams · 2016-01-15T23:02:23Z

Here are results with Mid 2014 MBPro 2.5 GHz i7, 16GB RAM, NVIDIA GeForce GT 750M 2GB. Running against Redis 3.0.0 on the same machine. Any special configs needed for Redis to optimize it for ioredis?

> matcha benchmarks/*.js
(node) child_process: options.customFds option is deprecated. Use options.stdio instead.
==========================
ioredis: 1.14.0
node_redis: 0.12.1
CPU: 8
OS: darwin x64
node version: v5.3.0
current commit: 581524d
==========================
                      simple set
          34,464 op/s » ioredis
          49,481 op/s » node_redis
                      simple get
          33,857 op/s » ioredis
          40,399 op/s » node_redis
                      simple set
          36,601 op/s » ioredis
          52,078 op/s » node_redis
                      simple get
          41,298 op/s » ioredis
          51,527 op/s » node_redis
                      simple get with pipeline
           2,769 op/s » ioredis
           9,503 op/s » node_redis
                      lrange 100
          34,479 op/s » ioredis
          65,115 op/s » node_redis
                      publish
          40,704 op/s » ioredis
          49,343 op/s » node_redis
                      subscribe
          30,397 op/s » ioredis
          40,692 op/s » node_redis
  Suites:  6
  Benches: 12
  Elapsed: 96,849.92 ms

curtiszimmerman · 2016-02-15T04:35:32Z

I'm running Redis via Docker (something very close to docker run -d --name=redis -p 127.0.0.1:6379:6379 redis which automagically pulls the stock Redis image and then starts it listening locally). I'm posting because my results are much more in favor of ioredis than node_redis.

Redis server v=3.0.7

==========================
ioredis: 1.15.0
node_redis: 2.4.2
CPU: 8
OS: linux x64
node version: v4.3.0
current commit: 13ac1c4

==========================

                      simple set
          99,635 op/s » ioredis
          55,776 op/s » node_redis

                      simple get
          86,711 op/s » ioredis
          53,505 op/s » node_redis

                      simple get with pipeline
           8,082 op/s » ioredis
           7,081 op/s » node_redis

                      lrange 100
          14,140 op/s » ioredis
           9,686 op/s » node_redis

                      publish
          92,252 op/s » ioredis
          56,567 op/s » node_redis

                      subscribe
          41,925 op/s » ioredis
          33,334 op/s » node_redis


  Suites:  6
  Benches: 12
  Elapsed: 92,940.75 ms

luin mentioned this issue Oct 26, 2015

You should redo the benchmark with latest node_redis #186

Closed

LewisJEllis mentioned this issue Dec 8, 2015

change from node_redis to ioredis bee-queue/bee-queue#21

Closed

luin closed this as completed Jun 2, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

incorrect benchmark ? #25

incorrect benchmark ? #25

jjmr commented May 5, 2015

luin commented May 5, 2015

dirkbonhomme commented May 5, 2015

jjmr commented May 5, 2015

luin commented May 5, 2015

jjmr commented May 5, 2015

lpinca commented May 5, 2015

AVVS commented May 6, 2015

luin commented May 7, 2015

IncSW commented Jun 24, 2015

AVVS commented Jun 24, 2015

fantasticsoul commented Aug 18, 2015

luin commented Aug 18, 2015

fantasticsoul commented Aug 18, 2015

luin commented Aug 18, 2015

fantasticsoul commented Aug 18, 2015

luin commented Aug 18, 2015

fantasticsoul commented Aug 18, 2015

zhuangya commented Oct 26, 2015

shershams commented Jan 15, 2016

curtiszimmerman commented Feb 15, 2016

incorrect benchmark ? #25

incorrect benchmark ? #25

Comments

jjmr commented May 5, 2015

luin commented May 5, 2015

dirkbonhomme commented May 5, 2015

jjmr commented May 5, 2015

luin commented May 5, 2015

jjmr commented May 5, 2015

lpinca commented May 5, 2015

AVVS commented May 6, 2015

luin commented May 7, 2015

IncSW commented Jun 24, 2015

AVVS commented Jun 24, 2015

fantasticsoul commented Aug 18, 2015

luin commented Aug 18, 2015

fantasticsoul commented Aug 18, 2015

luin commented Aug 18, 2015

fantasticsoul commented Aug 18, 2015

luin commented Aug 18, 2015

fantasticsoul commented Aug 18, 2015

zhuangya commented Oct 26, 2015

shershams commented Jan 15, 2016

curtiszimmerman commented Feb 15, 2016