Add NamedPipe input operator #28841
Merged
Add NamedPipe input operator #28841
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
djaglowski
reviewed
Nov 1, 2023
This is a namedpipe input operator, which will read from a named pipe and send the data to the pipeline. It pretty closely mimicks the file input operator, but with a few differences. In particular, named pipes have an interesting property that they receive EOFs when a writer closes the pipe, but that _doesn't_ mean that the pipe is closed. To solve this issue, we crib from existing `tail -f` implementations and use an inotify watcher to detect whenever the pipe receives new data, and then read it using the standard `bufio.Scanner` reader. Signed-off-by: sinkingpoint <colin@quirl.co.nz>
This allows the user to configure the maximum size of a log line, rather than always using the default. Signed-off-by: sinkingpoint <colin@quirl.co.nz>
sinkingpoint
force-pushed
the
namedpipe_operator
branch
from
5913350 to
5daa897
Compare
|
Rebased and added licenses |
Here we add a couple of tests to cover more cases of named pipe creation. In particular: - A test to check that if a file already exists (i.e. not a named pipe), fail the operator creation. - A test to check that if a _pipe_ already exists, pass the operator construction because we can just use the existing pipe. Signed-off-by: sinkingpoint <colin@quirl.co.nz>
sinkingpoint
force-pushed
the
namedpipe_operator
branch
from
5daa897 to
e1d11bc
Compare
Signed-off-by: sinkingpoint <colin@quirl.co.nz>
sinkingpoint
force-pushed
the
namedpipe_operator
branch
from
e1d11bc to
a546114
Compare
|
So many checks 😅 |
|
@djaglowski is there a |
|
Unfortuantely there's not a catchall because so much is only encoded in github actions. I should mention, if your changes are pretty well isolated to one or two modules, you can run |
djaglowski
approved these changes
Nov 14, 2023
djaglowski
added
the
ready to merge
Merged
RoryCrispin
pushed a commit
to ClickHouse/opentelemetry-collector-contrib
that referenced
this pull request
Nov 24, 2023
**Description:** This is a namedpipe input operator, which will read from a named pipe and send the data to the pipeline. It pretty closely mimics the file input operator, but with a few differences. In particular, named pipes have an interesting property that they receive EOFs when a writer closes the pipe, but that _doesn't_ mean that the pipe is closed. To solve this issue, we crib from existing `tail -f` implementations and use an inotify watcher to detect whenever the pipe receives new data, and then read it using the standard `bufio.Scanner` reader. **Link to tracking Issue:** open-telemetry#27234 **Testing:** We add a couple of tests for the new operator. The first tests simply the creation of the named pipe - checking that it's created as a pipe, with the right permissions. The second goes further by inserting logs over several different `Open`s into the pipe, testing that the logs are read, and that the operator is able to handle the named pipe behavior of skipping over EOFs. **Documentation:** None, at the moment /cc @djaglowski --------- Signed-off-by: sinkingpoint <colin@quirl.co.nz>
djaglowski
pushed a commit
that referenced
this pull request
Dec 13, 2023
**Description:** This adds a new receiver that reads from a named pipe, using the previously created namedpipeinput stanza operator (#28841). Because that operator does the majority of the work, this is mostly boilerplate + a few tests to confirm that it works e2e. Link to tracking Issue: #27234 **Testing:** Added a few tests, loading the config, and actually creating a receiver to verify that it can read logs from the pipe. **Documentation:** None, yet. /cc @djaglowski - where should I add docs for this if any?
cparkins
pushed a commit
to AmadeusITGroup/opentelemetry-collector-contrib
that referenced
this pull request
Jan 10, 2024
**Description:** This adds a new receiver that reads from a named pipe, using the previously created namedpipeinput stanza operator (open-telemetry#28841). Because that operator does the majority of the work, this is mostly boilerplate + a few tests to confirm that it works e2e. Link to tracking Issue: open-telemetry#27234 **Testing:** Added a few tests, loading the config, and actually creating a receiver to verify that it can read logs from the pipe. **Documentation:** None, yet. /cc @djaglowski - where should I add docs for this if any?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This is a namedpipe input operator, which will read from a named pipe and send the data to the pipeline. It pretty closely mimics the file input operator, but with a few differences.
In particular, named pipes have an interesting property that they receive EOFs when a writer closes the pipe, but that doesn't mean that the pipe is closed. To solve this issue, we crib from existing
tail -fimplementations and use an inotify watcher to detect whenever the pipe receives new data, and then read it using the standardbufio.Scannerreader.Link to tracking Issue: #27234
Testing:
We add a couple of tests for the new operator. The first tests simply the creation of the named pipe - checking that it's created as a pipe, with the right permissions. The second goes further by inserting logs over several different
Opens into the pipe, testing that the logs are read, and that the operator is able to handle the named pipe behavior of skipping over EOFs.Documentation:
None, at the moment
/cc @djaglowski