[pkg/stanza] Add monitoring metrics for open and harvested files in fileconsumer

[ChrsMark]

Blocked on #31618

Description:

This PR adds support for filelog receiver to emit observable metrics about its current state: how many files are opened, and harvested.

Link to tracking Issue: #31256

Testing:

How to test this manually

1. Use the following collector config:

receivers:
  filelog:
    start_at: end
    include:
    - /var/log/busybox/monitoring/*.log
exporters:
  debug:
    verbosity: detailed
service:
  telemetry:
    metrics:
      level: detailed
      address: ":8888"
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [debug]
      processors: []

2. Build and run the collector: make otelcontribcol && ./bin/otelcontribcol_linux_amd64 --config ~/otelcol/monitoring_telemetry/config.yaml
3. Produce some logs:

echo 'some line' >> /var/log/busybox/monitoring/1.log
while true; do echo -e "This is a log line" >> /var/log/busybox/monitoring/2.log; done

4. Verify that metrics are produced:

curl 0.0.0.0:8888/metrics | grep _files
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4002    0  4002    0     0  1954k      0 --:--:-- --:--:-- --:--:-- 1954k
# HELP otelcol_fileconsumer_open_files Number of open files
# TYPE otelcol_fileconsumer_open_files gauge
otelcol_fileconsumer_open_files{service_instance_id="72b4899d-6ce3-41de-a25b-8f0370e22ec1",service_name="otelcontribcol",service_version="0.99.0-dev"} 2
# HELP otelcol_fileconsumer_reading_files Number of open files that are being read
# TYPE otelcol_fileconsumer_reading_files gauge
otelcol_fileconsumer_reading_files{service_instance_id="72b4899d-6ce3-41de-a25b-8f0370e22ec1",service_name="otelcontribcol",service_version="0.99.0-dev"} 1

Documentation:
Added a respective section in Filelog receiver's docs.

[djaglowski]

Do you mind splitting this into two PRs - one to pass in component.TelemetrySettings, the other to add the metrics?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[ChrsMark]

Still WiP.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[ChrsMark]

Still WiP

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[djaglowski]

Was this broken up because there was an intention to use constants?

[ChrsMark]

Not really, but good point. I will make those constants!

[djaglowski]

Would it be possible to increment whenever we open a file and decrement whenever we close one?

[ChrsMark]

I guess that would doable inside the Tracker. Specifically inside the Add(reader *reader.Reader) and ClosePreviousFiles() methods. Is that what you had in mind?

In that case, the metric should be called otelcol_tracker_open_files right?

[djaglowski]

I think the name can stay the same. IMO users don't need to be exposed to the internal package structure. Sticking with fileconsumer seems specific enough.

[ChrsMark]

Would it be possible to increment whenever we open a file and decrement whenever we close one?

So, files are opened by the Manager but are closed by the Manager and the Tracker as well from what I can see.
Also inside the Manager's operations there are multiple places where files are re-opened and then their readers are closed again like at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.100.0/pkg/stanza/fileconsumer/file.go#L203 or https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.100.0/pkg/stanza/fileconsumer/file.go#L222.

Based on this I thought that it makes sense to simplify and only increment the openFiles counter only when we create a fresh reader at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.100.0/pkg/stanza/fileconsumer/file.go#L225-L232.

Incrementing in previous steps and then possibly decrementing after a couple of instrunctions didn't make a lot of sense to me but let me know what you think :).

[newly12]

Hi @ChrsMark, one quick question, are the new metrics shown at detailed metrics level or normal level?

[ChrsMark]

Hi @ChrsMark, one quick question, are the new metrics shown at detailed metrics level or normal level?

They are shown on the normal level as well.

[djaglowski]

This is really the only opening or closing of files we do in the tracker, so it does feel quite awkward to have the tracker manage the meter. Instead, what if we just return the number of files closed from this method? We're closing them one after another, so decrementing after each one or all at once seems equivalent in this case.

[ChrsMark]

In that case we would need to either have the EndConsume() to return the number of files closed as well. or to move the ClosePreviousFiles() outside of it. I'm leaning towards the latter.

It seems that the calling order of the ClosePreviousFiles() is different on Windows so we cannot decouple it from the EndConsume()

[djaglowski]

The latter won't work because the order of operations is different depending on OS.

I think we could get rid of OS dependent behavior for closing files if we implement #32037.

Otherwise, I'm fine with returning the count from EndConsume.

[ChrsMark]

I had missed that EndConsume of noStateTracker closes files directly so we need to have it returning the number of files closed anyway.

[djaglowski]

Are you willing to look into what it takes to pass the context through?

[ChrsMark]

If I'm not mistaken we could pass down the ctx from the Shutdown of the stanza receiver when we stop the internal pipeline at

opentelemetry-collector-contrib/pkg/stanza/adapter/receiver.go

Line 142 in d78d7bb

pipelineErr := r.pipe.Stop()

.

Then if we pass the ctx on every single operator's Stop() we can make it available to the Stop() of the Manager/fileconsumer at

opentelemetry-collector-contrib/pkg/stanza/operator/input/file/input.go

Line 36 in d78d7bb

return i.fileConsumer.Stop()

when we stop the file_input operator.

This would be doable I think but would be a breaking change for the Pipeline interface and the Operator interface (plus the Manager).
Hence, I'm not sure if that's worth doing it unless we find value in doing so anyways and not just for the meter's need?