Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix: kubelet on edge node keeps restarting issue #370

Merged
merged 12 commits into from
Aug 12, 2021

Conversation

Windrow
Copy link
Contributor

@Windrow Windrow commented Jun 25, 2021

Bugfix: kubelet on edge node keeps restarting after yurtctl convert executed issue.

With current configuration, /var/lib/yurthub is used but not cleared on reset, and files in it would not be updated on next setup. Therefore, if we reuse the reset edge node again when credential of the cluster changed, kubehub would use wrong credential after conversion.

yurthub.yaml in yaml-template uses /etc/kubernetes/pki instead of /var/lib/yurthub, which could be a replacement.

See detailed description and investigation process of the issue at #367.

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #367

Special notes for your reviewer:

/assign @rambohe-ch @yixingjia

Does this PR introduce a user-facing change?

NONE

other Note

…xecuted issue. See detailed description of the issue at openyurtio#367.
@openyurt-bot
Copy link
Collaborator

@Windrow: GitHub didn't allow me to assign the following users: your_reviewer.

Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

Bugfix: kubelet on edge node keeps restarting after yurtctl convert executed issue.

With current configuration, /var/lib/yurthub is used but not cleared on reset, and files in it would not be updated on next setup. Therefore, if we reuse the reset edge node again when credential of the cluster changed, kubehub would use wrong credential after conversion.

yurthub.yaml in yaml-template uses /etc/kubernetes/pki instead of /var/lib/yurthub, which could be a replacement.

See detailed description and investigation process of the issue at #367.

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #367

Special notes for your reviewer:

/assign @rambohe-ch @yixingjia

Does this PR introduce a user-facing change?

NONE

other Note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openyurt-bot openyurt-bot added the size/S size/S 10-29 label Jun 25, 2021
@openyurt-bot
Copy link
Collaborator

Welcome @Windrow! It looks like this is your first PR to openyurtio/openyurt 🎉

command:
- yurthub
- --v=2
- --server-addr=__kubernetes_service_addr__
- --node-name=$(NODE_NAME)
- --join-token=__join_token__
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why delete parameter --join-token which generated when converted by yurtctl.

Copy link
Contributor Author

@Windrow Windrow Jun 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

join-token is not specified in openyurt/config/yaml-templateyurthub.yaml, as well. I think it is not a must if we do not use hubself mode.

Precisely to say, this is the trick to make yurthub not use hubself mode but kubelet mode.

I0628 15:53:18.792549       1 cert_mgr.go:372] no join token, so use kubelet config to bootstrap hub

I have to admit, it's a workaround for the issue I encountered, but not a proper cure to the root cause.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Windrow join-token can be generated by yurtctl, and we suggest that use hubself mode with join-token to generate yurthub client certificate that used to connect kube-apiserver. so we need to keep --join-token parameter for yurthub.

Copy link
Member

@qclc qclc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current solution seems to be the ca and key of the kubelet used by default, so it seems that there is no difference between hubself and kubelet mode by default.

@Windrow
Copy link
Contributor Author

Windrow commented Jun 28, 2021

The current solution seems to be the ca and key of the kubelet used by default, so it seems that there is no difference between hubself and kubelet mode by default.

openyurt/cmd/yurthub/app/options/options.go line 67. Default CertMgrMode is hubself there.

#370 (comment)

But if we do not provide join-token, yurthub will use kubelet method instead. They do have different implementations in fact.

@Windrow
Copy link
Contributor Author

Windrow commented Jun 28, 2021

Working on a solution to the root cause.

/wip

path: /var/lib/yurthub
type: DirectoryOrCreate
path: /etc/kubernetes/pki
type: Directory
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/var/lib/yurthub should be kept for yurthub certificates storage. and /etc/kubernetes/pki is not need to added because /etc/kubernetes hostpath has already mounted.

@openyurt-bot openyurt-bot added size/XS size/XS: 0-9 approved approved and removed size/S size/S 10-29 labels Jul 14, 2021
@openyurt-bot openyurt-bot added size/S size/S 10-29 and removed approved approved size/XS size/XS: 0-9 labels Jul 14, 2021
@rambohe-ch
Copy link
Member

@Windrow Thank you for making up pull request.
please fix the following ci errors.

@rambohe-ch
Copy link
Member

@Windrow please fix the following ci errors.

@openyurt-bot openyurt-bot added size/M size/M: 30-99 and removed size/S size/S 10-29 labels Aug 12, 2021
@openyurt-bot openyurt-bot added size/S size/S 10-29 and removed size/M size/M: 30-99 labels Aug 12, 2021
@rambohe-ch
Copy link
Member

@Windrow please fix the following ci errors.

@rambohe-ch
Copy link
Member

/lgtm
/approve

@openyurt-bot openyurt-bot added the lgtm lgtm label Aug 12, 2021
@openyurt-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rambohe-ch, Windrow

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openyurt-bot openyurt-bot added the approved approved label Aug 12, 2021
@openyurt-bot openyurt-bot merged commit 5cc09ee into openyurtio:master Aug 12, 2021
MrGirl pushed a commit to MrGirl/openyurt that referenced this pull request Mar 29, 2022
* Bugfix: kubelet on edge node keeps restarting after yurtctl convert executed issue. See detailed description of the issue at openyurtio#367.

* Revert ec98fef

* Remove yurt-hub config directory and certificates in it when revert edgenode.

* Fix compilation issue.

* Fix compilation issue.

* Fix compilation issue.

* Fix compilation issue.

* Fix compilation issue.

* Fix gofmt warning.

Co-authored-by: Yinzhe.Wu <Yinzhe.Wu@sony.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved approved kind/bug kind/bug lgtm lgtm size/S size/S 10-29
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Edgenode Turns to NotReady, kubelet on It Restart Rapidly
7 participants