Add optional scatterplot to benchcomp output

[tautschnig]

Scatterplots should make it easier to immediately spot performance trends (and indeed any differences) rather than having to process a (large) number of table rows.

Uses mermaid-js to produce markdown-embedded plots that will display on the job summary page. Scatterplots are not directly supported by mermaid-js at this point (xycharts only do line or bar charts), so quadrant plots are employed with various diagram items drawn in white to make them disappear.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

[tautschnig]

See https://github.com/model-checking/kani/actions/runs/8296849261?pr=3077 for what those diagrams now look like.

[jaisnan]

From the images shown, can you leave a comment here or in documentation somewhere on how to interpret those values? What do the values on the top-right of the plot mean, what do the values on the bottom-left mean etc?

[tautschnig]

From the images shown, can you leave a comment here or in documentation somewhere on how to interpret those values? What do the values on the top-right of the plot mean, what do the values on the bottom-left mean etc?

I am now adding text output to spell out the axis ranges, thank you for calling this out!

[karkhaz]

Thank you, just a few comments.

To be honest the latest version is still a bit hard to read. It's less the axis labels and more that I'm not confident that I would be able to spot a 20% increase or decrease from the y=x line, for example. Also if a particular point has moved far away from the y=x line, it's not that easy to determine which proof or benchmark that actually corresponds to.

As a totally different approach to quickly identifying outlying benchmarks---would it be easier if I actually implemented the benchcomp filter command, which allows you to filter benchmarks based on certain criteria before they get emitted to the Markdown table?

That way, we could print a "filtered" set of Markdown tables (containing only regressions) followed by the complete results. That might be superior to the current approach where regressions are highlighted in bold in the complete table.

So the action would look something like this

      - name: Perf Regression Table
        run: |
          echo "# Regressions" >> "$GITHUB_STEP_SUMMARY"
          new/tools/benchcomp/bin/benchcomp \
            --config new/tools/benchcomp/configs/perf-regression.yaml \
            visualize --only dump_markdown_results_table >> "$GITHUB_STEP_SUMMARY"

      - name: Perf All Results Table
        run: |
          echo "# All Results" >> "$GITHUB_STEP_SUMMARY"
          new/tools/benchcomp/bin/benchcomp \
            --config new/tools/benchcomp/configs/perf-all-results.yaml \
            visualize --only dump_markdown_results_table >> "$GITHUB_STEP_SUMMARY"

      - name: Run other visualizations
        run: |
          new/tools/benchcomp/bin/benchcomp \
            --config new/tools/benchcomp/configs/perf-regression.yaml \
            visualize --except dump_markdown_results_table

[tautschnig]

To be honest the latest version is still a bit hard to read. It's less the axis labels and more that I'm not confident that I would be able to spot a 20% increase or decrease from the y=x line, for example. Also if a particular point has moved far away from the y=x line, it's not that easy to determine which proof or benchmark that actually corresponds to.

Yes, I'd indeed love to be able to draw that diagonal line into the diagram, but I haven't been able to figure out a way to do this in Mermaid. Perhaps it's just about time to create bunch of feature requests against Mermaid.

As a totally different approach to quickly identifying outlying benchmarks---would it be easier if I actually implemented the benchcomp filter command, which allows you to filter benchmarks based on certain criteria before they get emitted to the Markdown table?

Perhaps we should have all of these tools available and then see what helps us the most?

[karkhaz]

Thank you, looking forward to seeing this in CBMC also!