fix the error when cert-mgr-mode set to kubelet #359

qclc · 2021-06-18T06:54:06Z

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

/kind bug

What this PR does / why we need it:

Fix the bug reported by yurthub can not running successfully when cert-mgr-mode set to "kubelet" #348
Decouple the GetRestConfig function from certManager

Which issue(s) this PR fixes:

Fixes #348

Special notes for your reviewer:

Does this PR introduce a user-facing change?

None

other Note

openyurt-bot · 2021-06-18T06:54:10Z

@qclc: GitHub didn't allow me to assign the following users: your_reviewer.

Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

/kind bug

What this PR does / why we need it:

Fix the bug reported by yurthub can not running successfully when cert-mgr-mode set to "kubelet" #348

Decouple the GetRestConfig function from certManager

Which issue(s) this PR fixes:

Fixes #348

Special notes for your reviewer:

Does this PR introduce a user-facing change?
None
other Note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

luckymrwang · 2021-06-18T13:28:50Z

pkg/yurthub/restconfig/restconfig.go

+
+const (
+	YurthubCertificateManagerName = "hubself"
+	KubeletCertificateManagerName = "kubelet"


const variable "hubself" and "kubelet" are defined again. subself/cert_mgr.go and kubelet/cert_mgr.go have been defined as certificateManagerName, and util.go/IsSupportedCertMode, NewYurtHubOptions.CertMgrMode have used the same name too. Is it better to unify them in one place and use them in many place?

I think it's a good idea. I have unified the globally constants in the yurthub module into the util.go file.

rambohe-ch · 2021-06-21T03:49:13Z

pkg/yurthub/certificate/hubself/cert_mgr.go

-	DefaultClusterName      = "kubernetes"
-	ClusterInfoName         = "cluster-info"
-	KubeconfigName          = "kubeconfig"
+	yurtHubName             = "yurthub"


how about keep the yurtHubName as hubName? So it looks like variables names are consistent

This variable's name has been changed back to hubname.

rambohe-ch · 2021-06-21T03:55:46Z

pkg/yurthub/restconfig/restconfig.go

+limitations under the License.
+*/
+
+package restconfig


how about put restconfig.go file under pkg/kubernetes/rest dir? and rename file name as config.go

The latest code has been commited. restconfig.go file has been migrated.

rambohe-ch · 2021-06-21T03:58:30Z

pkg/yurthub/restconfig/restconfig.go

+}
+
+// NewRestConfigManager creates a *RestConfigManager object
+func NewRestConfigManager(cfg *config.YurtHubConfiguration, certMgr interfaces.YurtCertificateManager, healthChecker healthchecker.HealthChecker) (*RestConfigManager, error) {


i think we need to add some unit test cases for RestConfigManager?

The latest code has been commited. Unit tests has been added.

rambohe-ch · 2021-06-22T08:41:06Z

pkg/yurthub/kubernetes/rest/config_test.go

+
+			var rc *rest.Config
+			if tt.mode == "hubself" {
+				rc = rcm.getHubselfRestConfigSkipCurrent()


I think we need invoke GetRestConfig() here and not getHubselfRestConfigSkipCurrent(), and then verify the return value.

The unit test has been corrected, and the GetRestConfig() function will be called here.

rambohe-ch · 2021-07-01T04:02:42Z

@qclc Would you upload the detail logs of yurthub startup that check the pr has worked correctly.

qclc · 2021-07-05T05:10:41Z

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:56:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:47:43Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

OS (e.g: cat /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Startup log:

This is the startup log when yurthub uses hubself mode:

I0705 12:54:01.805059       1 config.go:128] yurthub would connect remote servers: https://apiserver.demo:6443
I0705 12:54:01.806778       1 start.go:68] yurthub cfg: &config.YurtHubConfiguration{LBMode:"rr", RemoteServers:[]*url.URL{(*url.URL)(0xc0004ebf80)}, YurtHubServerAddr:"127.0.0.1:10267", YurtHubProxyServerAddr:"127.0.0.1:10261", YurtHubProxyServerDummyAddr:"169.254.2.1:10261", GCFrequency:120, CertMgrMode:"hubself", KubeletRootCAFilePath:"/etc/kubernetes/pki/ca.crt", KubeletPairFilePath:"/var/lib/kubelet/pki/kubelet-client-current.pem", NodeName:"n116", HeartbeatFailedRetry:3, HeartbeatHealthyThreshold:2, HeartbeatTimeoutSeconds:2, MaxRequestInFlight:250, JoinToken:"p8i95d.xkce3p8zum2h7bvh", RootDir:"/var/lib/yurthub", EnableProfiling:true, EnableDummyIf:true, EnableIptables:true, HubAgentDummyIfName:"yurthub-dummy0", StorageWrapper:(*cachemanager.storageWrapper)(0xc000220380), SerializerManager:(*serializer.SerializerManager)(0xc0002203c0)}
I0705 12:54:01.806899       1 start.go:83] 1. register cert managers
I0705 12:54:01.806944       1 certificate.go:60] Registered certificate manager kubelet
I0705 12:54:01.806967       1 certificate.go:60] Registered certificate manager hubself
I0705 12:54:01.806976       1 start.go:89] 2. create cert manager with hubself mode
I0705 12:54:01.807093       1 cert_mgr.go:214] /var/lib/yurthub/pki/ca.crt file already exists, so skip to create ca file
I0705 12:54:01.807154       1 cert_mgr.go:127] use /var/lib/yurthub/pki/ca.crt ca file to bootstrap yurthub
I0705 12:54:01.807383       1 cert_mgr.go:289] yurthub bootstrap conf file already exists, skip init bootstrap
I0705 12:54:01.807503       1 certificate_store.go:130] Loading cert/key pair from "/var/lib/yurthub/pki/yurthub-current.pem".
I0705 12:54:01.835765       1 certificate_manager.go:282] Certificate rotation is enabled.
I0705 12:54:01.835928       1 cert_mgr.go:412] yurthub config file already exists, skip init config file
I0705 12:54:01.835956       1 start.go:97] 3. new transport manager
I0705 12:54:01.836010       1 transport.go:57] use /var/lib/yurthub/pki/ca.crt ca cert file to access remote server
I0705 12:54:01.836237       1 certificate_manager.go:553] Certificate expiration is 2022-07-03 14:34:57 +0000 UTC, rotation deadline is 2022-05-11 05:51:45.924920369 +0000 UTC
I0705 12:54:01.836312       1 certificate_manager.go:288] Waiting 7432h57m44.08861986s for next certificate rotation
I0705 12:54:01.836320       1 start.go:105] 4. create health checker for remote servers 
I0705 12:54:01.839374       1 connrotation.go:145] create a connection from 10.10.103.116:43000 to apiserver.demo:6443, total 1 connections in transport manager dialer
I0705 12:54:01.867103       1 start.go:114] 5. new restConfig manager for hubself mode
I0705 12:54:01.867198       1 start.go:122] 6. new cache manager with storage wrapper and serializer manager
I0705 12:54:01.867412       1 cache_agent.go:68] reset cache agents to [kubelet kube-proxy flanneld coredns yurttunnel-agent]
I0705 12:54:01.868544       1 start.go:130] 7. new gc manager for node n116, and gc frequency is a random time between 120 min and 360 min
I0705 12:54:01.868830       1 gc.go:97] list pod keys from storage, total: 4
I0705 12:54:01.873664       1 config.go:107] re-fix hub rest config host successfully with server https://apiserver.demo:6443
I0705 12:54:01.922436       1 gc.go:125] list all of pod that on the node: total: 4
I0705 12:54:01.922710       1 gc.go:143] gc pod kubelet/pods/kube-system/yurtctl-servant-revert-n116-ssd4x successfully
I0705 12:54:01.922759       1 start.go:139] 8. new reverse proxy handler for remote servers
I0705 12:54:01.922821       1 start.go:148] 9. create dummy network interface yurthub-dummy0 and init iptables manager
I0705 12:54:01.922912       1 gc.go:74] start gc events after waiting 156.23µs from previous gc
I0705 12:54:01.924346       1 config.go:107] re-fix hub rest config host successfully with server https://apiserver.demo:6443
I0705 12:54:01.927500       1 gc.go:163] list kubelet event keys from storage, total: 11
I0705 12:54:01.968879       1 start.go:156] 10. new yurthub server and begin to serve, dummy proxy server: 169.254.2.1:10261
I0705 12:54:01.968915       1 start.go:159] 10. new yurthub server and begin to serve, proxy server: 127.0.0.1:10261, hub server: 127.0.0.1:10267
I0705 12:54:02.132305       1 gc.go:186] gc events kubelet/events/default/n116.168ee1f536cda333 successfully
I0705 12:54:02.132463       1 gc.go:186] gc events kubelet/events/default/n116.168ee1f543020049 successfully
I0705 12:54:02.132580       1 gc.go:186] gc events kubelet/events/default/n116.168ee1f543022fd3 successfully
I0705 12:54:02.132677       1 gc.go:186] gc events kubelet/events/default/n116.168ee1f543024d3d successfully
I0705 12:54:02.132779       1 gc.go:186] gc events kubelet/events/default/n116.168ee1f5495bb60a successfully
I0705 12:54:02.132869       1 gc.go:186] gc events kubelet/events/default/n116.168ee1f54adf9214 successfully
I0705 12:54:02.132987       1 gc.go:186] gc events kubelet/events/default/n116.168ee1f55379cdd2 successfully
I0705 12:54:02.133160       1 gc.go:186] gc events kubelet/events/kube-system/yurtctl-servant-revert-n116-ssd4x.168ee21ee2a28ceb successfully
I0705 12:54:02.133282       1 gc.go:186] gc events kubelet/events/kube-system/yurtctl-servant-revert-n116-ssd4x.168ee21ef6c6d7cc successfully
I0705 12:54:02.133380       1 gc.go:186] gc events kubelet/events/kube-system/yurtctl-servant-revert-n116-ssd4x.168ee21efa8f349b successfully
I0705 12:54:02.133518       1 gc.go:186] gc events kubelet/events/kube-system/yurtctl-servant-revert-n116-ssd4x.168ee21f0746ab87 successfully
I0705 12:54:02.133544       1 gc.go:160] no kube-proxy events in local storage, skip kube-proxy events gc
I0705 12:54:11.647864       1 util.go:232] start proxying: get /api/v1/pods?fieldSelector=spec.nodeName%3Dn116&limit=500&resourceVersion=0, in flight requests: 1
I0705 12:54:11.651170       1 util.go:232] start proxying: get /api/v1/services?limit=500&resourceVersion=0, in flight requests: 2

This is the startup log when yurthub uses kubelet mode:

I0705 12:57:12.614968       1 config.go:128] yurthub would connect remote servers: https://apiserver.demo:6443
I0705 12:57:12.616192       1 start.go:68] yurthub cfg: &config.YurtHubConfiguration{LBMode:"rr", RemoteServers:[]*url.URL{(*url.URL)(0xc00035eb00)}, YurtHubServerAddr:"127.0.0.1:10267", YurtHubProxyServerAddr:"127.0.0.1:10261", YurtHubProxyServerDummyAddr:"169.254.2.1:10261", GCFrequency:120, CertMgrMode:"kubelet", KubeletRootCAFilePath:"/etc/kubernetes/pki/ca.crt", KubeletPairFilePath:"/var/lib/kubelet/pki/kubelet-client-current.pem", NodeName:"n116", HeartbeatFailedRetry:3, HeartbeatHealthyThreshold:2, HeartbeatTimeoutSeconds:2, MaxRequestInFlight:250, JoinToken:"p8i95d.xkce3p8zum2h7bvh", RootDir:"/var/lib/yurthub", EnableProfiling:true, EnableDummyIf:true, EnableIptables:true, HubAgentDummyIfName:"yurthub-dummy0", StorageWrapper:(*cachemanager.storageWrapper)(0xc000690380), SerializerManager:(*serializer.SerializerManager)(0xc0006903c0)}
I0705 12:57:12.616288       1 start.go:83] 1. register cert managers
I0705 12:57:12.616321       1 certificate.go:60] Registered certificate manager kubelet
I0705 12:57:12.616335       1 certificate.go:60] Registered certificate manager hubself
I0705 12:57:12.616344       1 start.go:89] 2. create cert manager with kubelet mode
I0705 12:57:12.617195       1 cert_mgr.go:73] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
I0705 12:57:12.642719       1 start.go:97] 3. new transport manager
I0705 12:57:12.642765       1 transport.go:57] use /etc/kubernetes/pki/ca.crt ca cert file to access remote server
I0705 12:57:12.643677       1 start.go:105] 4. create health checker for remote servers 
I0705 12:57:12.646951       1 connrotation.go:145] create a connection from 10.10.103.116:43104 to apiserver.demo:6443, total 1 connections in transport manager dialer
I0705 12:57:12.673190       1 start.go:114] 5. new restConfig manager for kubelet mode
I0705 12:57:12.673219       1 start.go:122] 6. new cache manager with storage wrapper and serializer manager
I0705 12:57:12.673345       1 cache_agent.go:68] reset cache agents to [kubelet kube-proxy flanneld coredns yurttunnel-agent]
I0705 12:57:12.675219       1 start.go:130] 7. new gc manager for node n116, and gc frequency is a random time between 120 min and 360 min
I0705 12:57:12.675393       1 gc.go:97] list pod keys from storage, total: 3
I0705 12:57:12.715593       1 gc.go:125] list all of pod that on the node: total: 3
I0705 12:57:12.715669       1 start.go:139] 8. new reverse proxy handler for remote servers
I0705 12:57:12.715731       1 start.go:148] 9. create dummy network interface yurthub-dummy0 and init iptables manager
I0705 12:57:12.716027       1 gc.go:74] start gc events after waiting 359.45µs from previous gc
I0705 12:57:12.720930       1 gc.go:163] list kubelet event keys from storage, total: 8
I0705 12:57:12.751626       1 gc.go:160] no kube-proxy events in local storage, skip kube-proxy events gc
I0705 12:57:12.760800       1 start.go:156] 10. new yurthub server and begin to serve, dummy proxy server: 169.254.2.1:10261
I0705 12:57:12.760837       1 start.go:159] 10. new yurthub server and begin to serve, proxy server: 127.0.0.1:10261, hub server: 127.0.0.1:10267
I0705 12:57:13.482092       1 util.go:232] start proxying: get /api/v1/nodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dn116&resourceVersion=10529909&timeoutSeconds=480&watch=true, in flight requests: 1
I0705 12:57:13.483044       1 util.go:232] start proxying: get /apis/storage.k8s.io/v1/csidrivers?allowWatchBookmarks=true&resourceVersion=1&timeout=9m7s&timeoutSeconds=547&watch=true, in flight requests: 2
I0705 12:57:13.485057       1 util.go:232] start proxying: get /api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dkube-flannel-cfg&resourceVersion=10529801&timeout=6m25s&timeoutSeconds=385&watch=true, in flight requests: 3

qclc · 2021-07-05T05:12:14Z

@qclc Would you upload the detail logs of yurthub startup that check the pr has worked correctly.

@rambohe-ch Hi, the detailed log of yurthub startup has been uploaded to the comments.

rambohe-ch · 2021-07-13T07:45:01Z

/lgtm

rambohe-ch · 2021-07-13T07:49:05Z

/approve

rambohe-ch · 2021-07-13T08:04:22Z

/approve

openyurt-bot · 2021-07-13T08:07:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qclc, rambohe-ch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rambohe-ch]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openyurt-bot added the kind/bug kind/bug label Jun 18, 2021

openyurt-bot requested review from charleszheng44 and huangyuqi June 18, 2021 06:54

openyurt-bot added the size/L size/L: 100-499 label Jun 18, 2021

luckymrwang reviewed Jun 18, 2021

View reviewed changes

qclc force-pushed the fix-cert-mgr-mode-bug branch 3 times, most recently from 0c8f3b4 to 289b805 Compare June 20, 2021 11:20

rambohe-ch reviewed Jun 21, 2021

View reviewed changes

openyurt-bot added size/XL size/XL: 500-999 and removed size/L size/L: 100-499 labels Jun 21, 2021

qclc force-pushed the fix-cert-mgr-mode-bug branch from fb8e1cb to 738f81d Compare June 22, 2021 05:51

rambohe-ch reviewed Jun 22, 2021

View reviewed changes

qclc force-pushed the fix-cert-mgr-mode-bug branch 2 times, most recently from 53a16ad to d6df75a Compare June 28, 2021 12:21

qclc force-pushed the fix-cert-mgr-mode-bug branch from d6df75a to 150c84e Compare July 3, 2021 05:32

fix the error when cert-mgr-mode set to kubelet

31f47fc

qclc force-pushed the fix-cert-mgr-mode-bug branch from 150c84e to 31f47fc Compare July 3, 2021 11:01

openyurt-bot assigned rambohe-ch Jul 13, 2021

openyurt-bot added the lgtm lgtm label Jul 13, 2021

openyurt-bot added the approved approved label Jul 13, 2021

openyurt-bot merged commit 0d01f00 into openyurtio:master Jul 13, 2021

qclc mentioned this pull request Sep 14, 2021

Add qclc as a reviewer of openyurt #469

Merged

6 tasks

qclc mentioned this pull request Nov 12, 2021

Add qclc as maintainer of OpenYurt #580

Merged

6 tasks

MrGirl pushed a commit to MrGirl/openyurt that referenced this pull request Mar 29, 2022

fix the error when cert-mgr-mode set to kubelet (openyurtio#359)

65b7c37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix the error when cert-mgr-mode set to kubelet #359

fix the error when cert-mgr-mode set to kubelet #359

qclc commented Jun 18, 2021

openyurt-bot commented Jun 18, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

luckymrwang Jun 18, 2021

qclc Jun 18, 2021

rambohe-ch Jun 21, 2021 •

edited

Loading

qclc Jun 22, 2021

rambohe-ch Jun 21, 2021

qclc Jun 22, 2021

rambohe-ch Jun 21, 2021

qclc Jun 22, 2021

rambohe-ch Jun 22, 2021 •

edited

Loading

qclc Jun 28, 2021

rambohe-ch commented Jul 1, 2021

qclc commented Jul 5, 2021

qclc commented Jul 5, 2021 •

edited

Loading

rambohe-ch commented Jul 13, 2021

rambohe-ch commented Jul 13, 2021

rambohe-ch commented Jul 13, 2021

openyurt-bot commented Jul 13, 2021

fix the error when cert-mgr-mode set to kubelet #359

fix the error when cert-mgr-mode set to kubelet #359

Conversation

qclc commented Jun 18, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

openyurt-bot commented Jun 18, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rambohe-ch Jun 21, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rambohe-ch Jun 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rambohe-ch commented Jul 1, 2021

qclc commented Jul 5, 2021

qclc commented Jul 5, 2021 • edited Loading

rambohe-ch commented Jul 13, 2021

rambohe-ch commented Jul 13, 2021

rambohe-ch commented Jul 13, 2021

openyurt-bot commented Jul 13, 2021

rambohe-ch Jun 21, 2021 •

edited

Loading

rambohe-ch Jun 22, 2021 •

edited

Loading

qclc commented Jul 5, 2021 •

edited

Loading