building docs instability

[Borda]

🐛 Bug

building docs is randomly failing

To Reproduce

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 552, in _run_code
looking for now-outdated files... none found
    exec(code, ns)
  File "<string>", line 7, in <module>
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/classification/group_fairness.py", line 325, in plot
    return self._plot(val, ax)
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/metric.py", line 681, in _plot
    fig, ax = plot_single_or_multi_val(
  File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/utilities/plot.py", line 117, in plot_single_or_multi_val
    val = {k: torch.stack([val[i][k] for i in range(n_steps)]) for k in val[0]}  # type: ignore
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/utilities/plot.py", line 117, in <dictcomp>
    val = {k: torch.stack([val[i][k] for i in range(n_steps)]) for k in val[0]}  # type: ignore
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/utilities/plot.py", line 117, in <listcomp>
    val = {k: torch.stack([val[i][k] for i in range(n_steps)]) for k in val[0]}  # type: ignore
KeyError: 'EO_1_0'
/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/clustering/dunn_index.py:docstring of torchmetrics.clustering.dunn_index.DunnIndex.plot:19: WARNING: Exception occurred in plotting dunn_index-2
 from /home/runner/work/torchmetrics/torchmetrics/docs/source/clustering/dunn_index.rst:
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 552, in _run_code
    exec(code, ns)
  File "<string>", line 6, in <module>
  File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/metric.py", line 310, in forward
    self._forward_cache = self._forward_full_state_update(*args, **kwargs)
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/metric.py", line 342, in _forward_full_state_update
    batch_val = self.compute()
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/metric.py", line 633, in wrapped_func
    value = _squeeze_if_scalar(compute(*args, **kwargs))
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/clustering/dunn_index.py", line 86, in compute
    return dunn_index(dim_zero_cat(self.data), dim_zero_cat(self.labels), self.p)
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/functional/clustering/dunn_index.py", line 82, in dunn_index
    pairwise_distance, max_distance = _dunn_index_update(data, labels, p)
  File "/home/runner/work/torchmetrics/torchmetrics/src/torchmetrics/functional/clustering/dunn_index.py", line 39, in _dunn_index_update
    torch.stack([a - b for a, b in combinations(centroids, 2)], dim=0), ord=p, dim=1

Expected behavior

Environment

• https://github.com/Lightning-AI/torchmetrics/actions/runs/11269271231/attempts/1

Additional context

[SkafteNicki]

@Borda is it one or two failures?

[Borda]

@Borda is it one or two failures?

seem to be two different:

• plot with group_fairness
• dunn_index

[SkafteNicki]

@Borda fix for the group_fairness is now included in PR #2775, still need to figure out why the dunn index is failing

[Borda]

still need to figure out why the dunn index is failing

I guess it is due to randomness and the cardinality of the example?

[SkafteNicki]

still need to figure out why the dunn index is failing

I guess it is due to randomness and the cardinality of the example?

But the problem is that there is no randomness in that test, both preds and target are deterministic tensors. So it is very weird to me that it fails randomly