control-service: asynchronous deployment deletion

[mivanov1988]

Why:
As part of VEP-2272, we need to introduce a process for synchronizing data jobs from the database to Kubernetes. Currently, the process lacks the ability to delete deployments asynchronously.

What:
We have introduced an asynchronous method for deployment deletion. It is based on the logic that if the desired deployment is null and the actual deployment is not null, the process performs deployment deletion.

Testing done:
Integration tests.

Signed-off-by: Miroslav Ivanov miroslavi@vmware.com

[dakodakov]

LGTM

[antoniivanov]

What happens if data job is deleted?

[mivanov1988]

What happens if data job is deleted?

That is a very good point! When you delete a data job, the database will automatically remove the associated deployments from the desired and actual tables. Additionally, the old DeploymentService.deleteDeployment() will be called, and the K8s artifacts will be deleted as well. However, because the synchronization process caches the data jobs and deployments, if a user deletes a data job during synchronization, the Kubernetes artifacts will be left in the cluster, potentially leading to an inconsistent state.

In my opinion, we should remove the references from the desired and actual deployment tables to the data job table. In this scenario, we can delete the data job record from the 'data_job' table and the deployment record from the 'desired_data_job_deployment' table. Subsequently, we will have an existing deployment record in the 'actual_data_job_deployment' table, which will be deleted by the synchronization process during the next synchronization, along with all K8S-related artifacts.

@antoniivanov what do you think?

[antoniivanov]

What happens if data job is deleted?

That is a very good point! When you delete a data job, the database will automatically remove the associated deployments from the desired and actual tables. Additionally, the old DeploymentService.deleteDeployment() will be called, and the K8s artifacts will be deleted as well. However, because the synchronization process caches the data jobs and deployments, if a user deletes a data job during synchronization, the Kubernetes artifacts will be left in the cluster, potentially leading to an inconsistent state.

In my opinion, we should remove the references from the desired and actual deployment tables to the data job table. In this scenario, we can delete the data job record from the 'data_job' table and the deployment record from the 'desired_data_job_deployment' table. Subsequently, we will have an existing deployment record in the 'actual_data_job_deployment' table, which will be deleted by the synchronization process during the next synchronization, along with all K8S-related artifacts.

@antoniivanov what do you think?

Seems a good idea. It make sense that user should not be able to directly touch actual_deployment , this should be done only by the automation. Users (aka API endpoints) interact only with desired state

We need to make sure to test some corner cases:

• synchronization process is delayed or fails (pod restarts) for some reason after the data job is deleted
• job is created almost immediately with the same name after starting deletion
• What would the GraphQL endpoint return (aka the UI show) Probably the job as deleted and hence nothing

[mivanov1988]

What happens if data job is deleted?

That is a very good point! When you delete a data job, the database will automatically remove the associated deployments from the desired and actual tables. Additionally, the old DeploymentService.deleteDeployment() will be called, and the K8s artifacts will be deleted as well. However, because the synchronization process caches the data jobs and deployments, if a user deletes a data job during synchronization, the Kubernetes artifacts will be left in the cluster, potentially leading to an inconsistent state.
In my opinion, we should remove the references from the desired and actual deployment tables to the data job table. In this scenario, we can delete the data job record from the 'data_job' table and the deployment record from the 'desired_data_job_deployment' table. Subsequently, we will have an existing deployment record in the 'actual_data_job_deployment' table, which will be deleted by the synchronization process during the next synchronization, along with all K8S-related artifacts.
@antoniivanov what do you think?

Seems a good idea. It make sense that user should not be able to directly touch actual_deployment , this should be done only by the automation. Users (aka API endpoints) interact only with desired state

We need to make sure to test some corner cases:

• synchronization process is delayed or fails (pod restarts) for some reason after the data job is deleted

It will delete the cron job on the next synchronization.

• job is created almost immediately with the same name after starting deletion

It is okay. The cron job will be deleted, and the job will no longer have an associated one. The UI might display the existing deployment for a short period of time.

• What would the GraphQL endpoint return (aka the UI show) Probably the job as deleted and hence nothing

Nothing happens because the job is a first-class citizen.