syz-manager: never save crashes locally if dashboard is specified #5425
Conversation
There's 0 chance we will ever look at those. Currently, they only litter the web dashboards of syzbot instances.
| } | ||
| // Don't store the crash locally even if we failed to upload it. | ||
| // There is 0 chance that one will ever look in the crashes/ folder of those instances. | ||
| return mgr.cfg.Reproduce && resp.NeedRepro |
There was a problem hiding this comment.
That helped to have at least some visibility into such errors. If this happens more often than episodically, it's a real problem (we lost bugs/repros).
How can we keep some visibility into these errors? Say, if the rate suddenly increases?
Is using log.Errorf above a good idea? Or will that produce too many alerts?
There was a problem hiding this comment.
Currently we can also estimate the rate retroactively. E.g. check now how may was lost in the past year.
There was a problem hiding this comment.
If we want to estimate the rate, it's more reliable to look at the Cloud logs. I've searched failed to report crash to dashboard across all syzbot instances and it showed 35 times over the last 7 days (i.e. 5 times a day on average).
If we turn them to log.Errorf, that'd be quite a few. Most of them are operating on too many entity groups in a single transaction, so maybe we could address that and the rate will be okay...
There was a problem hiding this comment.
There's 0 chance we will ever look at those.
Currently, they only litter the web dashboards of syzbot instances.