fix: use proper key formating as per stanza persister #3879
Conversation
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
|
I have read the CLA Document and I hereby sign the CLA |
|
Behavior after fixing. You can pull 
|
|
Following is the current behavior (buggy migration). As you can see, each event is read twice. Event though fluentd left at 
|
| Offset int64 | ||
| Fingerprint *Fingerprint | ||
| Offset int64 | ||
| FileAttributes map[string]any |
There was a problem hiding this comment.
We're trying to convert fluentd's format to the following struct as per OTeL's format. We get Offset and Fingerprint by reading the fluentd's pos_file. But the field FileAttributes is left nil.
This causes a panic when the filelogreceiver tries to access this field while decoding it (causes nil reference). Setting it to an empty map resolves this.
| @@ -110,7 +110,7 @@ func (m *Migrator) MigrateCustomPos(matches []string) { | |||
| if err != nil { | |||
| log.Printf("error creating a new DB client for host file checkpoints: %v", err) | |||
| } | |||
| err = client.Set("$.file_input.knownFiles", buf.Bytes()) | |||
| err = client.Set("file_input.knownFiles", buf.Bytes()) | |||
There was a problem hiding this comment.
Have we always had this issue? Is there a breaking change in stanza that occurred?
There was a problem hiding this comment.
Well, not sure tbh. Need to dig in through commit history.
There was a problem hiding this comment.
I am pretty sure we have functional tests on the chart that test this. If not, we must create some or we will have more surprises.
There was a problem hiding this comment.
We don't have any tests for migration from SCK. I'll create a PR in the chart repo covering this part.
If I understand correctly, we're moving to Golang-based functional tests, right?
There was a problem hiding this comment.
@atoulme here's a PR to add functional test cases covering this. signalfx/splunk-otel-collector-chart#1024
PTAL! Thanks
|
Please fix the CI and we can look at getting this in. |
Done. |
|
Please rebase to get the fix on hadoop tests. |
atoulme
force-pushed
the
fix-migration
branch
from
0d061dd to
0c15a6d
Compare
|
I rebased for you. |
Description: Migrating offsets from SCK to SCK-Otel doesn't work. This is because of incorrect keys we use to populate the boltdb cache.
The proper keys should be
file_input.knownFiles,journald_input.lastReadCursorBut we prepend
$.to them and it causes log replication.Refer scoped persister for the current format.
Note: I'm currently in the process of testing this on my Mac. I'd appreciate you're review on this by thenTesting: Manually tested it on SCK and SCK-OTeL with locally built image