Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: add restore mon quorum in go #88

Merged
merged 1 commit into from
Apr 27, 2023

Conversation

subhamkrai
Copy link
Collaborator

@subhamkrai subhamkrai commented Mar 15, 2023

core: add restore mon quorum in go

this commits add a go command for
restore the mon quorum from single
good mon. In the background we,
scale down the operator first, and run some
ceph commands like remove bad mans from
mon map, extracting/injecting monmap.
And, in the end, we scale up the operator which
bring back the mon quorum.

Signed-off-by: subhamkrai srai@redhat.com

pkg/exec/exec.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
@subhamkrai subhamkrai force-pushed the add-go-restore-mon branch 2 times, most recently from 03751d3 to 93555a4 Compare March 20, 2023 07:41
@subhamkrai subhamkrai force-pushed the add-go-restore-mon branch 15 times, most recently from 3b6c319 to b53e6fe Compare March 23, 2023 13:30
@subhamkrai subhamkrai force-pushed the add-go-restore-mon branch 8 times, most recently from beda974 to 17c2b74 Compare March 24, 2023 16:17
@subhamkrai
Copy link
Collaborator Author

found one issue working on the fix.

@subhamkrai subhamkrai force-pushed the add-go-restore-mon branch 4 times, most recently from 8a4389b to 156be14 Compare April 3, 2023 09:49
Copy link
Member

@travisn travisn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing locally, I saw that the restored mon pod has the sleep infinity command instead of having the original ceph-mon command restored. I wouldn't expect the command to be modified on the mon. It should have just been modified on the mon debug pod. If you fix this command when starting back up the mon, hopefully the quorum will start again.

@subhamkrai
Copy link
Collaborator Author

Testing locally, I saw that the restored mon pod has the sleep infinity command instead of having the original ceph-mon command restored. I wouldn't expect the command to be modified on the mon. It should have just been modified on the mon debug pod. If you fix this command when starting back up the mon, hopefully the quorum will start again.

thanks @travisn I,ll check and update.

@subhamkrai subhamkrai marked this pull request as ready for review April 24, 2023 17:14
pkg/debug/start_debug.go Outdated Show resolved Hide resolved
@subhamkrai subhamkrai requested a review from travisn April 24, 2023 17:15
cmd/commands/mons.go Outdated Show resolved Hide resolved
pkg/debug/start_debug.go Outdated Show resolved Hide resolved
pkg/debug/start_debug.go Outdated Show resolved Hide resolved
pkg/debug/start_debug.go Outdated Show resolved Hide resolved
pkg/debug/start_debug.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Show resolved Hide resolved
@subhamkrai subhamkrai force-pushed the add-go-restore-mon branch 3 times, most recently from d9f02f0 to cebe29d Compare April 25, 2023 10:36
.github/workflows/go-test.yaml Show resolved Hide resolved
pkg/debug/start_debug.go Outdated Show resolved Hide resolved
pkg/exec/exec.go Show resolved Hide resolved
pkg/exec/exec.go Show resolved Hide resolved
pkg/exec/exec.go Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
Copy link
Member

@travisn travisn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more small changes...

pkg/exec/exec.go Outdated Show resolved Hide resolved
kubectl-rook-ceph.sh Outdated Show resolved Hide resolved
pkg/exec/exec.go Outdated Show resolved Hide resolved
pkg/exec/exec.go Outdated Show resolved Hide resolved
pkg/mons/restore_quorum.go Show resolved Hide resolved
@subhamkrai subhamkrai requested a review from travisn April 26, 2023 06:14
Copy link
Member

@travisn travisn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commits look like they aren't strictly needed. How about squashing them? Then a separate commit for the log collector likely makes sense.

kubectl-rook-ceph.sh Show resolved Hide resolved
pkg/mons/restore_quorum.go Outdated Show resolved Hide resolved
@subhamkrai
Copy link
Collaborator Author

The commits look like they aren't strictly needed. How about squashing them? Then a separate commit for the log collector likely makes sense.

which commits @travisn, could enter the title not able to get the commit from the comment.

@subhamkrai subhamkrai requested a review from travisn April 26, 2023 15:54
Copy link
Member

@travisn travisn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Can you just update the commit description? It should discuss the command to restore mon quorum, instead of just talking about other small changes.

@subhamkrai subhamkrai requested a review from travisn April 27, 2023 03:14
this commits add a go command for
restore the mon quorum from single
good mon. In the background we,
scale down the operator first, and run some
ceph commands like remove bad mans from
mon map, extracting/injecting monmap.
And, in the end, we scale up the operator which
bring back the mon quorum.

Signed-off-by: subhamkrai <srai@redhat.com>
@subhamkrai subhamkrai merged commit 0c42fa3 into rook:master Apr 27, 2023
@subhamkrai subhamkrai deleted the add-go-restore-mon branch April 27, 2023 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants