[batch] collection of issues arising from Grafana alerts in January

[danking]

What happened?

Still a lot of deadlock errors. Largely from MJC https://cloudlogging.app.goo.gl/N8hoXPWYYWLiDPPi9

Looks like workers are leaving tasks running when they shutdown https://cloudlogging.app.goo.gl/JFYoACF9qcDvCaqk8

Looks like we need to set the severity correctly in the worker logs. I'm also seeing a lot of this

WARNING: Published ports are discarded when using host network mode
Also looks like we incorrectly log a ContainerTimeoutError as an error log even though that's a user error: https://cloudlogging.app.goo.gl/TUGWNxnFiBiEdsDo9 Moved to #14262

And we log ImageCannotBePulled as an error even though that's a user error: https://cloudlogging.app.goo.gl/TchqwUKNCrd6qqmh7

Also a few like this: Unknown child process pid 12331, will report returncode 255

Version

0.2.127

Relevant log output

No response

[daniel-goldstein]

Looks like we need to set the severity correctly in the worker logs. I'm also seeing a lot of this
WARNING: Published ports are discarded when using host network mode
Also looks like we incorrectly log a ContainerTimeoutError as an error log even though that's a user error: https://cloudlogging.app.goo.gl/TUGWNxnFiBiEdsDo9

For what it's worth, this was showing up as an info log because this is a docker log message not from our code, so it's not going through our logging filters. The reason this showed up in the Google Logging query was because the query included this line

severity=ERROR OR WARNING

which means "logs whose severity is ERROR or whose log entry contains "WARNING"", it is not equivalent to severity=ERROR OR severity=WARNING which does not show that log entry. Either way, #14252 gets rid of that log message entirely.