[Proposal] Ingestion Spec controller

[AdheipSingh]

• In ref: Proposal: Ingestion Spec controller Proposal: Ingestion Spec controller  druid-io/druid-operator#313

[styk-tv]

@AdheipSingh this would be a massive step forward and probably one of the most fundamental improvements in the operator. Configuration of kafka sources is fundamental to control of data plane using helm charts. I'd be happy to help with testing.

[AdheipSingh]

@styk-tv I will be sending a draft PR. Your reviews and testing would be really helpful.
Thank you for collaboration.

[styk-tv]

@AdheipSingh this is great news, thank you for your fast reply. waiting for the PR whenever you're ready, please ping here

[itamar-marom]

Regarding the field: supervisorSpec

Instead of spiling an entire JSON, a more common way is to do a reference to a configmap. or secret For example:

apiVersion: "druid.apache.org/v1alpha1"
kind: "DruidIngestion"
metadata:
  name: sample-druid-spec
  namespace: mydruid
spec:
  clusterRef: mydruid
  suspend: false
  supervisorSpecRef:
     kind: ConfigMap
     name: mysupervisor
     namespace: druid

and hold an object like this:

type CrossNamespaceSourceReference struct {
	// API version of the referent.
	// +optional
	APIVersion string `json:"apiVersion,omitempty"`

	// Kind of the referent.
	// +required
	Kind string `json:"kind"`

	// Name of the referent.
	// +required
	Name string `json:"name"`

	// Namespace of the referent, defaults to the namespace of the Kubernetes resource object that contains the reference.
	// +optional
	Namespace string `json:"namespace,omitempty"`
}

It makes the YAMLs much cleaner.

[AdheipSingh]

Instead of spiling an entire JSON, a more common way is to do a reference to a configmap. or secret For example:

• In that case, user needs to do a manual lookup to a configmap, and the operator will need to reconcile cm changes as well as CR ( which seems like not so friendly pattern ) , also it will require to apply a configmap and then apply the CR.
• CR should hold the truth of what an ingestion spec represents. I know its a long spec, but thats what the current CR also does for all the configs be it log4j, dimensions.json etc

[itamar-marom]

Your points are valid.

I know that Prometheus for example has an interesting configuration reload mechanism.

There is a sidecar container that watches the configmaps and when one of them changes, it sends a request to Prometheus API (route /-/reload) and it reloads the configuration from the configmap.

I'm still not familiar enough with Druid but I wonder if there is a better way to do it.

Regarding the users, Adding another configmap.yaml next to their Druid.yaml - I think they will prefer that instead of this big Spec. Maybe worth asking them.

[itamar-marom]

@AdheipSingh We also need a secure solution for providing authentication information.
For example, in hadoop_indexing we can provide s3 authentication:

"tuningConfig" : {
      "type": "hadoop",
      "jobProperties": {
        "fs.s3a.impl" : "org.apache.hadoop.fs.s3a.S3AFileSystem",
        "fs.AbstractFileSystem.s3a.impl" : "org.apache.hadoop.fs.s3a.S3A",
        "mapreduce.job.classloader": "true",
        "fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider",
        "fs.s3a.access.key": "${AWS_ACCESS_KEY}",
        "fs.s3a.secret.key": "${AWS_SECRET_KEY}",
        "fs.s3a.session.token": "${AWS_SESSION_TOKEN}"
      }
    }