Emit levels in STRING instead of integers

[marcellodesales]

Hi there,

Is there any support to emitting the records as STRING values instead of integers??? We use Splunk to aggregate the logs without any pre-processing step like logstash... Splunk indexes json events and our requirement is to print the String representation to be aligned with other systems...

The requirement is that the console output must be using the "bunyan" format, while the console.log format is the "long", human-readable one... This requirement is only when writing to the rotate-file stream type...

From

• level: 20

{...,"level":20,"msg":"Missing property 'logs.file.dir'....}

To

• level: ERROR

{...,"level":ERROR,"msg":"Missing property 'logs.file.dir'....}

I found the method _emit (https://github.com/trentm/node-bunyan/blob/master/lib/bunyan.js#L778), but it is emitting for all the types :( Any help greatly appreciated!

thanks
Marcello

[olsonpm]

I was just looking at the code just now for the same functionality. There is no support for that - so my question is whether that's a design decision or if they wouldn't mind me writing support for it.

You could either add functionality to the resolveLevel function, or you could add another function e.g. "mapLevelToName". This should accompany new exported name variables - preferably in an object and not all separate like levels are currently.

In the meantime, go ahead and use this helper function:

function mapLevelToName(level) {
    var res = '';
    switch (level) {
        case bunyan.TRACE:
            res = 'DEBUG';
            break;
        case bunyan.DEBUG:
            res = 'DEBUG';
            break;
        case bunyan.INFO:
            res = 'INFO';
            break;
        case bunyan.WARN:
            res = 'WARN';
            break;
        case bunyan.ERROR:
            res = 'ERROR';
            break;
        case bunyan.FATAL:
            res = 'FATAL';
            break;
    }
    return res;
};

// Example stream definition usage
var MyStream = function() {};
MyStream.prototype.write = function(rec) {
  console.log('Level: %s', mapLevelToName(rec.level))
};

[marcellodesales]

@olsonpm Thanks a lot for the directions... But I can't get that suggestion yet... :( Here's what I have... I defined the console stream... I would need a writer function for the record when the format is "bunyan", which triggers this https://github.com/thlorenz/bunyan-format/blob/master/lib/format-record.js#L370

  var consoleStream = {
    name: "console",
    level: "trace",
    stream: bformat({ outputMode: "bunyan" }),
    write: function(rec) {
      rec.level = mapLevelToName(rec.level);
    }
  };

I still can't get it to work...

$ NODE_ENV=preprd node app{"name":"responsive-experience","hostname":"ubuntu","pid":25169,"component":"ux-topics","appVersion":"0.0.1","level":30,"msg":"newrelic config={\"app_name\":[\"ux-topics-preprd\"],\"license_key\":\"af2c03eeb0226a4908450d11f007af1532a68893\",\"logging\":{\"level\":\"info\"},\"proxy\":\"\"}","time":"2014-12-01T17:02:33.053Z","src":{"file":"/home/mdesales/dev/icode/responsive-experience-sp-logging/newrelic.js","line":30},"v":0}

[olsonpm]

I'm a little confused at what your write function is doing. In your above code - all it's going is setting the local rec.level and doesn't actually write anything.

[marcellodesales]

@olsonpm Sorry, I was trying to get something late night :) I pushed a change to the module I use for formatting. I updated the examples and everything works as expected! Thanks for the suggestions and I added a note in the patch about your suggestion!

[marcellodesales]

Let me know how this patch would apply to Bunyan formatter and I will be glad to submit a pull request here.

[olsonpm]

Glad you got it working, and i can definitely relate to the late night mishaps.

[trentm]

@marcellodesales I got it working here: https://gist.github.com/trentm/8f16f630ddd73d3f87d4

Not in bunyan core, but it does require a change to bunyan to export the RotatingFileStream class, which I've done in commit b9e3a0d. Note that this is a bit inefficient in that it does the JSON.stringify again (internally Bunyan will already be doing this in its _emit).

I don't think I want explicit support for this in Bunyan core because it would basically mean an option that results in emiting log records that are almost Bunyan-spec'd log records, but not quite. That said, it is very laborious to customize a stream to do something like this. I hope that with Bunyan 2.x plans (no timeline for that at all) to make bunyan stream handling a lot cleaner, it would be easy to compose the built in rotating file stream and a small object stream that did the level translation.

Please re-open if I missed something.

[olsonpm]

I understand not modifying the current bunyan codebase to cater this use-case, however I do suggest modifying the structure in 2.x to resemble a difference as noted in the following:

// from
var TRACE = 10;
var DEBUG = 20;
//...

var levelFromName = {
    'trace': TRACE,
    'debug': DEBUG,
    //...
};


// to
function LogLevel(name_, level_) {
  this.name = name_;
  this.level = level_;
}
var LogLevels = [];
LogLevels.push(new LogLevel('trace', 10));
// etc...

module.exports.LogLevel = LogLevel
module.exports.LogLevels = LogLevels;

Poor naming aside, this would future-proof that portion of the code as well as make it more cohesive. Any helper functions could just be prototyped onto LogLevel by developers wanting to extend its functionality and prevent the need for little utility functions like 'mapLevelToName'.

[marcellodesales]

@trentm Thanks for adding an example... However, it would be great to not only export RotatingFileStream, but also ConsoleRawStream... Our deployments output to console because we deploy using Docker and/or Upstart...

thanks!

[c24w]

I know this is an ancient issue, but for anyone landing here: you can just use bunyan.nameFromLevel to figure out the name from the level number.

[ghostganz]

Still can't actually change the output of the 'level' field to a string though?

Thinking about how to post-process the output, or switching to some other lib.

[c24w]

I don't think so - we ended up doing something like this:

function wrappedRedisTransport() {
  const rt = new RedisTransport({ /* ... */ });

  return {
    write: entry => rt.write(Object.assign(entry, {
      level: bunyan.nameFromLevel[entry.level]
    }))
  };
}

bunyan.createLogger({
  streams: [{
    type: 'raw',
    stream: wrappedRedisTransport()
  }]
});

[emertechie]

For the benefit of anyone else trying to log string levels to file, here's what I came up with:

function levelStringFileStream(filePath) {
    const fileStream = fs.createWriteStream(filePath, {
        flags: 'a',
        encoding: 'utf8'
    });
    return {
        write: log => {

            // Map int log level to string
            const clonedLog = Object.assign({}, log, {
                levelStr: bunyan.nameFromLevel[log.level]
            });

            var logLine = JSON.stringify(clonedLog, bunyan.safeCycles()) + '\n';
            fileStream.write(logLine);
        }
    };
}

Don't forget to set the type to raw when you use it:

bunyan.createLogger({
  streams: [{
    type: 'raw',
    stream: levelStringFileStream('/path/to/file')
  }]
});

[darrelcox]

Here's what I came up with....

function myStdOutWithLevelName() {}
myStdOutWithLevelName.prototype.write = function(data) {
    var logObject = JSON.parse(data)

    // Change log level number to name and write it out
    logObject.level = bunyan.nameFromLevel[logObject.level]
    process.stdout.write(JSON.stringify(logObject) + '\n')
}

bunyan.createLogger({
    name: "console",
    streams: [
        {
            level: "trace",
            stream: new myStdOutWithLevelName()
        }
    ],
    src: false,
    serializers: bunyan.stdSerializers
})

[bfelbo]

Why doesn't this work when just using a serializer?

E.g.

bunyan.createLogger({
  name: "console",
  streams: [{
    stream: process.stdout,
    level: "trace",
    serializers: {
      level: function level(val){
        return bunyan.nameFromLevel[val]
      }
    }
  }]
})

[autocorrectoff]

how can I kick out some fields? 'v', for example

[lukesiler]

Combined a couple of hints here to be

function wrappedStdout() {
    return {
        write: entry => {
            // to simplify StackDriver UX, use severity and timestamp fields
            var logObject = JSON.parse(entry)
            logObject.severity = bunyan.nameFromLevel[logObject.level].toUpperCase();
            logObject.timestamp = logObject.time;
            delete logObject.time;
            process.stdout.write(JSON.stringify(logObject) + '\n');
        }
    };
}

const logger = bunyan.createLogger({
        name: loggerName,
        streams: [
            {
                level: logLevel,
                stream: wrappedStdout()
            }
        ]
    });