Ingestion Controller

[AdheipSingh]

Fixes #3.

Current State of PR

• Supports NativeBatchIndexParallel
• Supports Basic-Auth

CRD Definition

group: druid.apache.org
version: v1alpha1
kind: DruidIngestion

Scope

• CRD is scoped to namespace for k8s API
• DruidIngestion CR is scoped to the same namespace as Druid CR.
• Relation b/w DruidIngestion CR and Druid CR. is one to one.

Installation

• Helm
• Kubectl

Ingestion Method Support

• Kafka
• Kinesis
• NativeBatchIndexParallel
• QueryControllerSQL
• HadoopIndexHadoop

Authentication/TLS With Druid

• Basic Auth
• TLS

Ingestion Controller

Intro

• Ingestion Controller is based on event driven + polling which shall reconciles k8s events with druidingestion custom resource.
• Ingestion controller is hooked in the manager ( main.go ) and runs its own reconcile loop.
• Introducing ingestion controller is fully backward compatible to the druid CR.

DruidIngestion Custom Resource

suspend: false
druidCluster: tiny-cluster
auth:
    type: basic-auth
    secretRef:
        name: ingestion-secret
        namespace: druid
ingestion:
    type: native-batch
    spec: |-
    ...
    ...
 status:
   currentIngestionSpec.json: |-
   ...
   ...
   lastUpdateTime: "2023-05-06T21:34:29Z"
   message: DruidIngestionControllerCreateSuccess
   reason: '{"task":"index_parallel_wikipedia-5_ogmefkld_2023-05-06T21:34:29.497Z"}'
   status: "True"
   taskId: index_parallel_wikipedia-5_ogmefkld_2023-05-06T21:34:29.497Z
   type: DruidIngestionControllerCreateSuccess

State Handling

• Controllers are stateless, and custom resource status is used to store and reflect state changes. For each reconcile loop, the state is constructed based on observation.

• Flow for creation and updation of druidIngestion CR

  1. The controller checks the status of the CR to see if a taskId is associated with the incoming event.
  2. If there is no taskId associated, the controller creates a task by calling the underlying druid API for the specific ingestion method type defined.
  3. Once the request is submitted to the druid API, the response is patched on the druid ingestion CR. If the response code is 200, it is considered a success, and the taskId is populated. This way, on the next subsequent reconcile, the controller is aware of the existence of the task. If the response code is not 200, the status is patched with "failed" and the appropriate response from the druid API.
  4. If a taskId is already associated with the incoming event, the controller checks if anything has changed in the ingestion spec defined using reflect.DeepEqual(). If the current state has changed, the spec is updated by calling the druid API, and the responses are patched to the status.

• Flow for deletion of druidIngestion CR

  1. The DruidIngestion controller uses finalizers for executing logic before the deletion of the CR.
  2. The controller adds a finalizer in the druid ingestion CR if the deletion timestamp is not set.
  3. On deletion of the druid ingestion CR, the deletion timestamp is set, and the controller calls the druid API to shut down the task.
  4. If the response is successful, the finalizer is removed, and the k8s API removes the underlying CR.

• The Druid service is constructed by getting the router Kubernetes service, and the controller expects the druidCluster name to be present on the CR.

• Basic Auth is supported, and users can create a secret and refer to it in the druidIngestion CR. Controllers look up OperatorUserName and OperatorPassword.

[AdheipSingh]

@itamar-marom @styk-tv @cyril-corbon @cintoSunny @harinirajendran

All your initial reviews are welcome !