Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v6.0.2 Logs are accepted, but actually not stored in Loki #80

Closed
braxnu opened this issue Oct 12, 2021 · 3 comments
Closed

v6.0.2 Logs are accepted, but actually not stored in Loki #80

braxnu opened this issue Oct 12, 2021 · 3 comments

Comments

@braxnu
Copy link

braxnu commented Oct 12, 2021

My setup:

docker run -d -p 3100:3100 grafana/loki
import winston from 'winston'
import Loki from 'winston-loki'

const logger = winston.createLogger({
  transports: [
    new Loki({
      host: 'http://127.0.0.1:3100',
    }),
  ],
})

logger.info("test")

The JSON payload sent to Loki in v6.0.2 contains "[Object object]" instead of the actual labels payload. This was introduced by exactly this change ec82e79

The payload sent to Loki is:

{
  "streams": [
    {
      "stream": "[object Object]",
      "values": [
        [
          "1634077254005000000",
          "test"
        ]
      ]
    }
  ]
}

There is no http error response, but these are impossible to find in grafana.

@SilSalADRESYS
Copy link

We ran into an issue too on one application which was using v6.0.2; basically, the issue ran as follows:

  • The batcher.js, if you log the response from its post request (and NOT the error) in winston-loki/src/batcher.js line 195, you will actually get back an error from the Go Parser used by Loki, in that there is a malformed body.
  • We found that we kept getting error 400 responses in the access logs for our nginx setup in front of Loki (but no error logs), so basically - the errors weren't being returned as errors from Loki, so the winston-loki package thought it was okay. Hence why it never showed up in the Node application logs.

The specific error it got back from the Loki parser is "Value looks like object, but can't find closing '}' symbol", source code here: https://github.com/buger/jsonparser/blob/master/parser.go
It seemed to always point at byte #10 at the end of the log message, which was in the context/metadata object. Fiddling with the Winston formatter settings didn't seem to have an impact.

Reverting to winston-loki v6.0.1 fixed the problem. Hope this helps!

@jtut1731
Copy link

Same here...long time user/fan. 6.0.1 works fine. 6.0.2 means no logs.

@JaniAnttonen
Copy link
Owner

Fixed in #82 :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants