Log agent based on container runtime as buffer instead of own filesystem buffer #479
Closed
2 of 5 tasks
Labels
Milestone
Description
As part of the epic kyma-project/kyma#11236 the log agent was designed to support multiple pipelines were every pipeline can run in isolation. Whenever a pipeline runs into conncetivity/backpressure issues and the logs cannot be shipped fast enough, that situation should never impact another pipeline. That could only be achieved by introducing a filesystem based buffering, so that the central tail input reads and dispatches in the persistent buffers. Whenever a buffer uns full, data gets evicted from the buffer but the tail plugin continues to read and dispatch. That
The problems
The goal
Tear down the assumption that users want to route into different backends. Support multiple pipelines but let them be coupled, so if one has backpressure, the other will suffer as well. We see scenarios were multiple pipelines are defined for the same backend only. That will bring the big advantage of passing the back pressure back to the source of the logs, the container runtime, without introducing any buffer layer in between.
The challenges
Actions
Implementation plan first chunk
Leftovers after first chunk
Criterias
The text was updated successfully, but these errors were encountered: