feat: add http rsr #18

NikolasHaimerl · 2025-01-12T18:22:04Z

This PR proposes the following changes:

Add Http RSR to number of retrievals over time
Add Http RSR to total number of retrievals in histograms (total and non zero)
Add Http RSR to Non zero SPs over time

cloudflare-workers-and-pages · 2025-01-12T18:32:09Z

Deploying spark-dashboard with Cloudflare Pages

Latest commit:	`41f8c15`
Status:	✅ Deploy successful!
Preview URL:	https://0a1df9a0.spark-dashboard.pages.dev
Branch Preview URL:	https://nhaimerl-add-http-rsr.spark-dashboard.pages.dev

View logs

NikolasHaimerl · 2025-01-13T09:58:11Z

src/index.md

@@ -60,7 +60,7 @@ const end = view(Inputs.date({label: "End", value: getDateXDaysAgo(1) }));
    resize((width) => Histogram(SparkMinerRates, { width, title: "Retrieval Success Rate Buckets", thresholds: 10 }))
  }</div>
  <div class="card">${
-    resize((width) => Histogram(nonZeroSparkMinerRates, { width, title: "Non-zero Miners: Retrieval Success Rate Buckets", thresholds: 10 }))
+    resize((width) => Histogram(nonZeroSparkMinerRates.map((record) => ({success_rate: record.success_rate, success_rate_http: record.success_rate_http? record.success_rate_http: null})), { width, title: "Non-zero Miners: Retrieval Success Rate Buckets", thresholds: 10 }))


We do not want to count success_rate_http of 0 or null for this histogram which is why we set all 0, undefined and null values of success_rate_http to null. The histogram function then filters out all null values as those are not valid for plotting.

We're also filtering for values inside Histogram. Do we need to filter values here too? Also in this case if value is 0 it will be assigned null.

Yes, because the filter inside of Histogram only filters for null values. For this specific chart we also do not want all 0 values, which is why we need to do the filtering before accesing Histogram.

NikolasHaimerl · 2025-01-13T09:59:05Z

src/index.md

@@ -86,7 +88,7 @@ const percentiles = Object.entries(SparkMinerRsrSummaries)
  ].map(above => ({
    day: new Date(day),
    label: `> ${above * 100}%`,
-    count: countAbove(miners.map(m => m.success_rate), above)
+    count_succes_rate: countAbove(miners.map(m => m.success_rate), above),


For naming consistency I also renamed the success rate here.

NikolasHaimerl · 2025-01-13T10:07:38Z

src/components/line-graph.js

+    })),
+    ...filteredEvents.map((event) => ({
+      day: event.day,
+      success_rate_http: Number(event.success_rate_http),


success_rate_http may be null and for consistency with past data as well as the graph Filecoin SPs with a non-zero Spark Retrieval Success Rate' I set past null values to 0. This way the graphs are consistent in how they handle past data.

bajtos · 2025-01-13T13:11:53Z

Great start!

(1)
Is it possible to hide the HTTP line in the chart for days we don't have data for?

(2)
In the miner list at the bottom, please format success_rate_http as percents for consistency with the formatting of success_rate.

(3)
We need to find better names for "Successful" and "HTTP". Most people looking at these charts don't understand the nuances, the names should convey the distinction.

The charts include "Retrieval Success Rate" in the title, I think it's not necessary to repeat that in the legend.

Proposed legend:

Successful -> "HTTP or Graphsync"
HTTP -> "HTTP only"

NikolasHaimerl · 2025-01-13T15:20:21Z

Great start!

(1) Is it possible to hide the HTTP line in the chart for days we don't have data for?

(2) In the miner list at the bottom, please format success_rate_http as percents for consistency with the formatting of success_rate.

(3) We need to find better names for "Successful" and "HTTP". Most people looking at these charts don't understand the nuances, the names should convey the distinction.

The charts include "Retrieval Success Rate" in the title, I think it's not necessary to repeat that in the legend.

Proposed legend:
* Successful -> "HTTP or Graphsync"

* HTTP -> "HTTP only"

Thanks, I worked in your requested changes.

bajtos · 2025-01-13T15:43:05Z

Thanks, I worked in your requested changes.

Lovely! 😍

The miner table still does not show the HTTP success rate in percent, PTAL again.

On the per-miner page, it looks like we are showing only one line that's initially orange and later switches to blue, can you PTAL?

See e.g. here: https://nhaimerl-add-http-rsr.spark-dashboard.pages.dev/provider/f03028412

NikolasHaimerl · 2025-01-13T16:09:48Z

Thanks, I worked in your requested changes.

Lovely! 😍

The miner table still does not show the HTTP success rate in percent, PTAL again.

On the per-miner page, it looks like we are showing only one line that's initially orange and later switches to blue, can you PTAL?

See e.g. here: https://nhaimerl-add-http-rsr.spark-dashboard.pages.dev/provider/f03028412

I think this is because, the success rates are the same and thus overlay. See: https://30a5bd11.spark-dashboard.pages.dev/provider/f03201941

bajtos · 2025-01-13T16:32:07Z

I think this is because, the success rates are the same and thus overlay.

Makes sense 👍🏻

NikolasHaimerl · 2025-01-14T08:59:45Z

Thanks, I worked in your requested changes.

Lovely! 😍
The miner table still does not show the HTTP success rate in percent, PTAL again.

On the per-miner page, it looks like we are showing only one line that's initially orange and later switches to blue, can you PTAL?
See e.g. here: https://nhaimerl-add-http-rsr.spark-dashboard.pages.dev/provider/f03028412

I think this is because, the success rates are the same and thus overlay. See: https://30a5bd11.spark-dashboard.pages.dev/provider/f03201941

I also worked in the changs for the percentage value of the successful http rate.

bajtos

LGTM 👏🏻

bajtos · 2025-01-14T09:17:55Z

@juliangruber @pyropy: You may want to review this pull request, too; you are more familiar with this codebase than I am.

juliangruber

the observable parts lgtm, I trust @bajtos's review covered the spark parts

Nikolas Haimerl added 5 commits January 11, 2025 14:38

add histogram

d16bde9

add: http success data

239299f

filter non zero miners

b7545e1

filter out non zero http rsr

f72ce58

non zero Retrieval rate for miners

8ea8103

set past undefined http success to 0

7ac18d9

NikolasHaimerl commented Jan 13, 2025

View reviewed changes

Nikolas Haimerl added 2 commits January 13, 2025 10:59

comments

cca6130

comments

ee18342

NikolasHaimerl commented Jan 13, 2025

View reviewed changes

NikolasHaimerl marked this pull request as ready for review January 13, 2025 10:20

NikolasHaimerl requested review from bajtos and pyropy and removed request for pyropy January 13, 2025 10:20

Nikolas Haimerl added 2 commits January 13, 2025 16:05

block past null values

dc660ba

label naming

a8b597f

fix rate percent display

41f8c15

bajtos approved these changes Jan 14, 2025

View reviewed changes

NikolasHaimerl merged commit c7a40a1 into main Jan 14, 2025
1 check passed

NikolasHaimerl deleted the nhaimerl-add-http-rsr branch January 14, 2025 09:17

juliangruber self-requested a review January 19, 2025 09:51

juliangruber reviewed Jan 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add http rsr #18

feat: add http rsr #18

NikolasHaimerl commented Jan 12, 2025

cloudflare-workers-and-pages bot commented Jan 12, 2025 •

edited

Loading

NikolasHaimerl Jan 13, 2025

pyropy Jan 14, 2025 •

edited

Loading

NikolasHaimerl Jan 14, 2025

NikolasHaimerl Jan 13, 2025

NikolasHaimerl Jan 13, 2025

bajtos commented Jan 13, 2025

NikolasHaimerl commented Jan 13, 2025

bajtos commented Jan 13, 2025

NikolasHaimerl commented Jan 13, 2025

bajtos commented Jan 13, 2025

NikolasHaimerl commented Jan 14, 2025

bajtos left a comment

bajtos commented Jan 14, 2025

juliangruber left a comment

feat: add http rsr #18

feat: add http rsr #18

Conversation

NikolasHaimerl commented Jan 12, 2025

cloudflare-workers-and-pages bot commented Jan 12, 2025 • edited Loading

Deploying spark-dashboard with Cloudflare Pages

NikolasHaimerl Jan 13, 2025

Choose a reason for hiding this comment

pyropy Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

NikolasHaimerl Jan 14, 2025

Choose a reason for hiding this comment

NikolasHaimerl Jan 13, 2025

Choose a reason for hiding this comment

NikolasHaimerl Jan 13, 2025

Choose a reason for hiding this comment

bajtos commented Jan 13, 2025

NikolasHaimerl commented Jan 13, 2025

bajtos commented Jan 13, 2025

NikolasHaimerl commented Jan 13, 2025

bajtos commented Jan 13, 2025

NikolasHaimerl commented Jan 14, 2025

bajtos left a comment

Choose a reason for hiding this comment

bajtos commented Jan 14, 2025

juliangruber left a comment

Choose a reason for hiding this comment

cloudflare-workers-and-pages bot commented Jan 12, 2025 •

edited

Loading

pyropy Jan 14, 2025 •

edited

Loading