No alerts for when capacity utilization threshold breached

[julienlim]

I redeployed tendrl using tendrl-ansible from master just now, and I created a near full condition on one of my bricks, and no alerts to say that threshold was breach on node 3:/bricks/brick1 were generated.

I also noticed that in the Tendrl UI, the chart shown the brick utilization shows blue (which may confuse user who sees it differently in the Grafana dashboards since it is not matching the thresholds colors used in the Host dashboards.

@cloudbehl @gnehapk @a2batic @Tendrl/tendrl-qe @mcarrano

[rishubhjain]

@cloudbehl Please check the issue.

[cloudbehl]

@GowthamShanmugam Can you verify why the bricks alert is generated when the threshold is breached.

@gnehapk Is the color coding added in the progress bar if the utilization exceeds 80% it should be yellow and when it exceeds 90% it should be Red.

[GowthamShanmugam]

@cloudbehl verified, it works fine, need to merge this small PR to receive volume alerts from grafana #217

[rishubhjain]

@julienlim Please verify whether the issue is still reproducible or not.

[GowthamShanmugam]

@rishubhjain need this patch to be merged for this it will solve all alert count and alert filter and grafana volume alert catch

#219
Tendrl/gluster-integration#453
Tendrl/commons#766
Tendrl/node-agent#657

[GowthamShanmugam]

@julienlim This issue is fixed, please verify this

[julienlim]

@GowthamShanmugam @rishubhjain @cloudbehl @gnehapk Verified that when I caused one of the bricks threshold to breach that an alert was generated.

Observing that the alerts counter on hosts is not correctly synchronizing with the alerts (as node2 should have 2 vs. 1 alert).

I checked the vol1 (which node2:/bricks/brick1 is on that breached), the vol1 alert counter was incremented to 1.

My environment:

• Tendrl 1.5.4 deployed
• Tendrl server on node0
• Tendrl node-agents on node1, node2, node3 -- node1-3 are in the same Gluster cluster

I filled up node2:/bricks/brick1 to over 90% to trigger a threshold breach. Threshold alert did occur. Other alerts for node1, node2, node3 also triggered based on memory utilization above 80%.

@Tendrl/tendrl-qe

[GowthamShanmugam]

@julienlim memory utilization node2 is very near to max threshold breach of 80.4 so it might be replaced with the clear event before you check alert count, because it may come less than 80 before you check, please past the notification UI screen also. then we can verify easily.

Because i experienced this lot of time, please share notifier screen, then we can identify easily

[julienlim]

@GowthamShanmugam

Here's the alerts & notification screenshot along with host list:

[GowthamShanmugam]

@Tendrl/tendrl-core is brick utilization threshold breach also should come under node alert count? I have added only brick down in node alert counts.

i am sure brick utilization threshold breach should not come under volume alert count, is it related necessary to add in node_alert count ?

[shtripat]

@nthomas-redhat @brainfunked @r0h4n plz comment.
I feel its good to have node level alert count if brick utilization breached threshold.

[r0h4n]

I would suggest we track the new change requests in a separate issue.

[julienlim]

@r0h4n Can you please list what the GitHub issues are so I can track them?