Strimzi
-
|
I am running Kafka Connect Cluster with the file sink connector. I am able to run it fine and it generates files to local pod ephemeral storage. Is there an example config for KafkaConnect to mount a volume (PVC), so that i can write files to persistent storage. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 5 replies
-
|
You cannot mount volumes into Kafka Connect. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the reply, so that would mean file sink or file source connectors won't be supported with Strimzi Kafka Connect? Since in production environment we want files being read or being written to be persisted in case of pod restart etc. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, correct. There is some enhancement issue for it somewhere. There are several issues to how the PVs/PVCs work on Kube and whether you have or don't have read-write many storage etc. which make this not completely straight forward. |
Beta Was this translation helpful? Give feedback.
-
|
Looks like confluent Kafka Connect helm chart supports volume mounts |
Beta Was this translation helpful? Give feedback.
-
|
That is possible. Bu confluent Helm Charts have nothing to do with Strimzi so I do not know what they do or don't do. |
Beta Was this translation helpful? Give feedback.
-
|
If I want to go ahead try doing this anyway i.e add support for mounting volumes, would just changing the KafkaConnect CRD be sufficient or do I need to change the Operator Java code too? |
Beta Was this translation helpful? Give feedback.
-
|
That would require changes to the operator code. And probably a discussion first what and how should be done. |
Beta Was this translation helpful? Give feedback.
-
|
We need to implement a pipeline (ci/cd) that allows the developers to update (whenever they need to) the plugins (and whatnot, but the plugins dir, here, is enough for this example we r discussing) connectors, but also the plugins. Mounting a volume allows for an easier, more manageable way to do so. Other than having to build a new img with the new plugins. This is a relevant feat request for Strimzi. (I feel) |
Beta Was this translation helpful? Give feedback.
-
|
That is not how it should work. Your CI/CD pipelines should add them to the Docker file and build a new container image which you then use. Mounting binaries from some volumes is not exactly containerisation bets practice. |
Beta Was this translation helpful? Give feedback.
-
|
Hi scholj, |
Beta Was this translation helpful? Give feedback.
You cannot mount volumes into Kafka Connect.