[RFE] Provide a way to notify client applications of topology changes

[jsanda]

Client applications refer to pods running within the same Kubernetes cluster. Those pods could be managed by a deployment or some other operator for example.

Topology changes could refer to:

• cluster created
• data center added
• data center removed
• rack added
• rack removed
• (probably more)

For now I want to focus on creating a cluster. I will talk more about adding data centers in a separate ticket. I think that a very common use will be to deploy a C* cluster and then deploy some client application. Let me provide a concrete use case.

I am developing an operator for tlp-stress which is an improved version of the cassandra-stress tool. It would be really nice to have the following happen:

• Deploy a CassandraCluster
• When the cluster is up, CassKop sends a notification that triggers creation of a TLPStress (which is managed by the tlp-stress-operator)
• tlp-stress will then run against the cluster

I intentionally left out details on how that notification process might work because I do not have a good answer yet :)

A easy approach would be too hard code support for specific client applications into CassKop. For example, suppose we add fields in the CasssandraCluster spec for specifying client application deployments. CassKop could then automatically create those deployments when the cluster is ready.

A more generic solution would be preferable. Some sort of messaging/eventing mechanism would be more flexible, at the cost of more moving parts. For example, CassKop could simply fire an event indicating that the cluster is ready.

[cscetbon]

I guess a simple feature as adding a bus for the different events happening at the operator level would be enough. The operator would accept some keys to know which bus and filters to use/apply and a secret for the authentication and would start sending all the relevant events to that endpoint. This way we're not tied to any sort of component.

We would need to add whenever an operation is triggered or done a call to a function to send that event onto the bus.

[jsanda]

As I mentioned in #86,

I think it would be good to consider using open standards, like cloud events. I think a great starting point would be having the ability to specify an http endpoint as part of a CassandraCluster. The operator would then send notifications to that endpoint for interesting events. This would minimize external dependencies for the operator while still providing plenty of flexibility.

I don't want to get too far ahead of myself, but I would interested in prototyping a knative event source that makes use of the http endpoint functionality I described.

It sounds like are more or less on the same page. With respect to authentication, could that be handled by Istio?

[cscetbon]

I would need to look more into knative event which I really know nothing about. I was looking for a generic way to send events in order to not limit the type of the receiver. I thought about Kafka cause using kafka-connect it can actually sink out to a lot of targets. It seems knative can do the same. I'm definitely open to hear about and find the best choice.

It sounds like are more or less on the same page

Yes

With respect to authentication, could that be handled by Istio?

I think so. Istio handles encryption between services, as long as the communication takes place between internal services it can handle it. If it goes out of the cluster it's different, but we should right away decide to keep it internal only