track how much time the flush loop is stalled waiting for uploads

[Bodobolero]

Problem

We don't know how much time PS is losing during ingest when waiting for remote storage uploads in the flush frozen layer loop.
Also we don't know how many remote storage requests get an permit without waiting (not throttled by remote_storage concurrency_limit).

Summary of changes

• Add a metric that accumulates the time waited per shard/PS
• in remote storage semaphore wait seconds add a first bucket with 100 microseconds to count requests that do not need to wait on semaphore
• add to remote storage dashboard

Update: created a new version that uses a Gauge (one increasing value per PS/shard) instead of histogram as suggested by review

[jcsp]

LGTM, but let's think about whether we want to take the cost of the histogram long term

[github-actions]

6941 tests run: 6633 passed, 0 failed, 308 skipped (full report)

---

Code coverage* (full report)

• functions: 30.9% (7977 of 25851 functions)
• lines: 48.6% (63315 of 130182 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
b9793ab at 2024-11-26T11:41:15.146Z :recycle:}