Add HealthChecker Callback

[hanlint]

What does this PR do?

Adds a HealthChecker callback to monitor node health. Will log a warning if GPUs on a node appear to be in poor health. Provides an optional interface to send a message via Slack webhook.

Note: this adds the pynvml and slack_sdk to the dependencies. Unsure if this should be in the base install or a separate tag.

TODO:

• Run some LLM 1B training runs to verify minimal throughput impact

Before submitting

• Have you read the contributor guidelines?
• Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
• Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
• Did you update any related docs and document your change?
• Did you update any related tests and add any new tests related to your change? (see testing)
• Did you run the tests locally to make sure they pass?
• Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

[mvpatel2000]

Can you please update callbacks.rst to include documentation?

[dakinggg]

Looks like its on your todo, but can you show a screenshot of a slack alert getting triggered?

[mvpatel2000]

Looks like its on your todo, but can you show a screenshot of a slack alert getting triggered?

[nik-mosaic]

No meaningful performance impact on 1B models:

[dakinggg]

LGTM, thanks Hanlin!

[mvpatel2000]

LGTM mod 2 minor nits. Great v1! Excited to test it

[mvpatel2000]

I would prefer a separate target since it doesn't really fit under dev -- the rest are pretty clearly more dev things and this is not. But no strong opinions