feat: clean eps work of the provison cluster when pull clusters get unhealthy

[jwcesign]

What type of PR is this?
/kind feature

What this PR does / why we need it:
Before, the eps work's deletion of the provision clusters is handled by the agent(for pull clusters), when the cluster is disconnected, the karmada-agent can't delete the work, causing resource residue.

This PR moves the deletion to mcs_controller.go(always run on Karmada control plane), to fix the problem

Which issue(s) this PR fixes:
Fixes #none

Special notes for your reviewer:
none

Does this PR introduce a user-facing change?:

`karmada-controller-manager`: The control plane is responsible for deleting expired EPS of MultiClusterService.

[jwcesign]

/cc @Rains6 @XiShanYongYe-Chang @chaosi-zju
Please take a look

[karmada-bot]

@jwcesign: GitHub didn't allow me to request PR reviews from the following users: Rains6.

Note that only karmada-io members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @Rains6 @XiShanYongYe-Chang @chaosi-zju
Please take a look

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (58b1790) 51.87% compared to head (090c1f0) 51.87%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4383   +/-   ##
=======================================
  Coverage   51.87%   51.87%           
=======================================
  Files         243      243           
  Lines       24114    24114           
=======================================
+ Hits        12509    12510    +1     
+ Misses      10923    10922    -1     
  Partials      682      682           

Flag	Coverage Δ
unittests	51.87% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Vacant2333]

should we add/update the e2e for it

[jwcesign]

should we add/update the e2e for it

I will do it in another PR.

Are u willing to add e2e for this PR?

[jwcesign]

Test step:
1.Run hack/local-up-karmada.sh
2.Deploy following manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: hello-app
        image: gcr.io/google-samples/hello-app:1.0
        ports:
        - containerPort: 8080
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 81
    targetPort: 8080
  selector:
    app: nginx
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: mci-workload
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - member3
---
apiVersion: networking.karmada.io/v1alpha1
kind: MultiClusterService
metadata:
   name: nginx
spec:
  types:
    - CrossCluster
  serviceConsumptionClusters:
    - member2
  serviceProvisionClusters:
    - member3

3.check work CR

4.Edit the karmada-agent replicas as 0
5.check clusters' state and work CR

The mcs work/original eps work/dispatch eps work are deleted as expected

[RainbowMango]

/lgtm
/approve

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/controllers/OWNERS [RainbowMango]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Vacant2333]

should we add/update the e2e for it

I will do it in another PR.

Are u willing to add e2e for this PR?

yes

[Vacant2333]

Test step: 1.Run hack/local-up-karmada.sh 2.Deploy following manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: hello-app
        image: gcr.io/google-samples/hello-app:1.0
        ports:
        - containerPort: 8080
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 81
    targetPort: 8080
  selector:
    app: nginx
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: mci-workload
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - member3
---
apiVersion: networking.karmada.io/v1alpha1
kind: MultiClusterService
metadata:
   name: nginx
spec:
  types:
    - CrossCluster
  serviceConsumptionClusters:
    - member2
  serviceProvisionClusters:
    - member3

3.check work CR 4.Edit the karmada-agent replicas as 0 5.check clusters' state and work CR The mcs work/original eps work/dispatch eps work are deleted as expected

hello, im going to add the testcase for this, and on my understandinng, i need add the case to mcs_test.go right?

[jwcesign]

hello, im going to add the testcase for this, and on my understandinng, i need add the case to mcs_test.go right?

Yes, add to test/e2e/mcs_test.go. Thanks for helping