Performance issue

[Thomasdezeeuw]

When logging allot of items (> 50.000) my application starts to slow down quite alot. It starts to use 100% of the cpu and starts eating memory. I use it to log some basic information about every request. When benchmarking my application my memory usage skyrockets to 500 ~ 600 mb, without logging anything it stays below 80 mb. It processes about 100.000 requests every minute.

As for my configuration:

var config = {
    // Production transports
    transports: [
      // Error log
      new winston.transports.File({
        name: 'error.log',
        filename: './logs/error.log',
        level: 'warn',
        maxsize: 2 * 1024 * 1024 // 2mb
      }),
      // Debug log
      new winston.transports.File({
        name: 'debug.log',
        filename: './logs/debug.log',
        maxsize: 20 * 1024 * 1024 // 20mb
      })
    ],
    exceptionHandlers: [
      new winston.transports.File({
        filename: 'logs/exceptions.log'
      })
    ]
  };

var logger = new winston.Logger(config);

I've found a issue that might be related, but it's rather old #296, aswell as a possible solution #376, but that issue also doesn't seem very active.

[3rd-Eden]

I don't want to be captain obvious here.. But of course your application is going to be slower when you start logging 50K lines vs nothing.

[Thomasdezeeuw]

I realize that but adding 0.5GB in memory is a little bit much isn't it?

[Thomasdezeeuw]

This node issues might be related.

[indexzero]

Duplicate of #288.

[Thomasdezeeuw]

I don't think so, that issue is about data loss, this one is about to much memory usage. I not 100% about the details but I don't think any data was lost.

[indexzero]

@Thomasdezeeuw I think they are all related to flushing / buffering. Data lost & too much data buffered are in the same boat to me.

I reference issues explicitly like this because #288 will be where the winston team goes to find all the issues related to this hard-to-track-down performance issue. Your input is still important 👍