stop proxy cache asynchronously

[ikaven1024]

Signed-off-by: yingjinhui yingjinhui@didiglobal.com

What type of PR is this?
/kind bug

What this PR does / why we need it:
When a member cluster is down, stopping the proxy cache for this cluster takes a few seconds(as below log, stop the pods cache takes 9s)

I0106 17:25:07.764975   62644 resource_cache.go:35] Stop store for yjh-1 /v1, Resource=pods
I0106 17:25:16.609796   62644 resource_cache.go:35] Stop store for yjh-1 /v1, Resource=nodes

During stopping, it hold the lock of MultiClusterCache. This results in client requests will be blocked. Because request handlers also need this lock:

karmada/pkg/search/proxy/store/multi_cluster_cache.go

Lines 162 to 168 in 7b4c541

	func (c *MultiClusterCache) List(ctx context.Context, gvr schema.GroupVersionResource, options *metainternalversion.ListOptions) (runtime.Object, error) {
	var resultObject runtime.Object
	items := make([]runtime.Object, 0, int(math.Min(float64(options.Limit), 1024)))
	requestResourceVersion := newMultiClusterResourceVersionFromString(options.ResourceVersion)
	requestContinue := newMultiClusterContinueFromString(options.Continue)
	clusters := c.getClusterNames()

karmada/pkg/search/proxy/store/multi_cluster_cache.go

Lines 319 to 321 in 7b4c541

	func (c *MultiClusterCache) getClusterNames() []string {
	c.lock.RLock()
	defer c.lock.RUnlock()

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

`karmada-search`: avoid proxy request block when member cluster down.

[jwcesign]

How long it takes now?

[XiShanYongYe-Chang]

Hi @ikaven1024, does this change affect the access of other client requests to the cache resources of the cluster?

I have another question, why not do it here:

https://github.com/karmada-io/karmada/blob/master/pkg/search/proxy/store/multi_cluster_cache.go#L71

[ikaven1024]

How long it takes now?

I add logs to measure the spending time:

I0109 12:12:00.792441   87527 multi_cluster_cache.go:64] MultiClusterCache update cache takes 162.416µs

[ikaven1024]

I have another question, why not do it here:

https://github.com/karmada-io/karmada/blob/master/pkg/search/proxy/store/multi_cluster_cache.go#L71

You are right. But when users remove resource from ResourceRegistry also will stop the cache via clusterCache.updateCache. So I move the go to the place Cacher really stopping:

func (c *resourceCache) stop() {
	klog.Infof("Stop store for %s %s", c.clusterName, c.resource)
	go c.Store.DestroyFunc()
}

[jwcesign]

How long it takes now?

I add logs to measure the spending time:

I0109 12:12:00.792441   87527 multi_cluster_cache.go:64] MultiClusterCache update cache takes 162.416µs

Great!
/lgtm

[XiShanYongYe-Chang]

/approve
Ask for @jwcesign to have a review again.

Hi @ikaven1024, do we need to cherry-pick this path to the previous release?

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XiShanYongYe-Chang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/search/OWNERS [XiShanYongYe-Chang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jwcesign]

/lgtm