Feature Request: Allow regular expressions in prospector paths and extract to metadata

[DanielYWoo]

We have 800+ applications and trying to migrate to beat. The problem is that our log files are in a standard pattern, e,g.

/opt/logs/<env>/<application_name>/<log_type>.log

We have so many applications that we don't want to specify a prospector for each of those applications, we want to set up a single prospector, with a regular expression pattern instead of glob-pattern, to extract , and into metadata as fields like @metadata.env, @metadata.application_name and @metadata.log_type.

Another option is to write our own processor to handle this but that's not a general solution, if we change our file pattern we have to compile the processor and deploy again. So it's best to support regular expression in prospector paths.

[ph]

Hello, we are not planning to add regular expression as a processor, but we plan to have dissect which will allow to tokenize this string.

We have a PR at #6925

[DanielYWoo]

@ph #6925 is to extract information from the log message. My problem is to extract information from the file path and name.

[ph]

@DanielYWoo The Dissect processor will allow you to target any fields, so in your case you will target the source field of the event which contains /opt/logs/<env>/<application_name>/<log_type>.log and extract the information with a token similar to this

token: "/opt/logs/%{env}/%{application_name}/%{log_type}.log"
example string: "/opt/logs/production/myapp/web.log"

This will generate the following fields:

  env: "production"
  application_name: "myapp"
  log_type: "web"

[ph]

@DanielYWoo I have added a complete example to my previous comment.

[ph]

I will close this for now, @DanielYWoo feel free to reopen it if you think dissect cannot solve your use case.

[DanielYWoo]

Thanks, got it