Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MySQL can cause cacti to become stalled due to locking issues #4880

Closed
TheWitness opened this issue Aug 4, 2022 · 0 comments
Closed

MySQL can cause cacti to become stalled due to locking issues #4880

TheWitness opened this issue Aug 4, 2022 · 0 comments
Labels
boost Boost related issue bug Undesired behaviour confirmed Bug is confirm by dev team data collection Issues related to data collection resolved A fixed issue
Milestone
v1.2.22

Comments

@TheWitness
Copy link
Member

Describe the bug

In the last two days, I have found issues with boost that will have a ripple effect. The problem is that the set_lock() MySQL function can essentially block graphing and polling if a data source has been locked, and it is unable to be unlocked.

This will not happen on all occasions as this is a timing issue. As such Cacti needs to be a bit more intelligent about it's use of set_lock() in MySQL.

To Reproduce

Steps to reproduce the behavior:

  1. Start boost

  2. Abruptly cancel boost child process through a kill -9 which does not allow the signal handler to clear the lock.

  3. Watch as boost basically gets hung up which can then lead to missed boost processes.

Expected behavior

Less issues that cause Cacti processes to essentially hang.
@TheWitness TheWitness added bug Undesired behaviour confirmed Bug is confirm by dev team labels Aug 4, 2022
@TheWitness TheWitness added this to the v1.2.22 milestone Aug 4, 2022
@TheWitness TheWitness added boost Boost related issue data collection Issues related to data collection labels Aug 4, 2022
@TheWitness TheWitness changed the title The MySQL set_lock() function can lead to stalled boost and dsstats processes The MySQL get_lock() function can lead to stalled boost and dsstats processes Aug 7, 2022
TheWitness added a commit that referenced this issue Aug 7, 2022
@TheWitness
Fixing #4882, #4881, #4880, #4533 - Boost and Structured Paths
6c07b04 
This change adds two features, and corrects some minor bugs:

* issue#4880: The MySQL get_lock() function can lead to stalled boost and dsstats processes
* issue#4882: Under rare conditions the boost process can get hung until the poller times out
* feature#4533: Restructuring RRDfile paths takes too long on large systems and will stop polling for potentially several hours
* feature#4881: When boost is running, it's not clear which processes are running and how long they have to complete
TheWitness added a commit that referenced this issue Aug 7, 2022
@TheWitness
Fixing #4882, #4881, #4880, #4533 - Boost and Structured Paths
0d0b272 
Missed one minor change entry for this commit
@TheWitness TheWitness added the resolved A fixed issue label Aug 7, 2022
@netniV netniV changed the title The MySQL get_lock() function can lead to stalled boost and dsstats processes MySQL can cause cacti to become stalled due to locking issues Aug 14, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Nov 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
boost Boost related issue bug Undesired behaviour confirmed Bug is confirm by dev team data collection Issues related to data collection resolved A fixed issue
Projects
None yet
Milestone
v1.2.22
Development

No branches or pull requests
1 participant
@TheWitness