Benchmark not found (Criterion)

[WizardOfMenlo]

Hi there, I have experienced an issue similar to that of #5 but in Criterion rather than in Go. In particular, in my repo, I have a criterion bench that runs with the following output (using --output-format bencher).

Gnuplot not found, using plotters backend
test inv_prod/default/8 ... bench:         139 ns/iter (+/- 6)
test inv_prod/prod_of_inv/8 ... bench:          87 ns/iter (+/- 4)
test inv_prod/default/16 ... bench:         223 ns/iter (+/- 15)
test inv_prod/prod_of_inv/16 ... bench:         110 ns/iter (+/- 7)
test inv_prod/default/32 ... bench:         301 ns/iter (+/- 20)
test inv_prod/prod_of_inv/32 ... bench:         163 ns/iter (+/- 8)
test inv_prod/default/64 ... bench:         454 ns/iter (+/- 25)
test inv_prod/prod_of_inv/64 ... bench:         257 ns/iter (+/- 9)
test inv_prod/default/128 ... bench:         744 ns/iter (+/- 44)
test inv_prod/prod_of_inv/128 ... bench:         441 ns/iter (+/- 19)
test inv_prod/default/256 ... bench:        1489 ns/iter (+/- 83)
test inv_prod/prod_of_inv/256 ... bench:         908 ns/iter (+/- 41)
test inv_prod/default/512 ... bench:        2613 ns/iter (+/- 153)
test inv_prod/prod_of_inv/512 ... bench:        1841 ns/iter (+/- 114)

I have the following workflow:

name: Benchmark
on: ["push", "pull_request"]

jobs:
  benchmark:
    name: Performance regression check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      # Install Rust (stable)
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: stable
          override: true

      # Run benchmark, storing in output.txt
      - name: Run benchmark
        run: cargo bench -- --output-format bencher | tee output.txt
      # Get old benches (if any)
      - name: Download previous benchmark data
        uses: actions/cache@v1
        with:
          path: ./cache
          key: ${{ runner.os }}-benchmark
      - name: Store benchmark result
        uses: rhysd/github-action-benchmark@v1.8.0
        with:
          tool: 'cargo'
          output-file-path: output.txt
          external-data-json-path: ./cache/benchmark-data.json
          fail-on-alert: true

And it fails with No benchmark result was found in /home/runner/work/stabchain/stabchain/output.txt. Benchmark output was 'Gnuplot not found, using plotters backend ...

I have confirmed before that the output is correctly written and that the action is able to read it, so the issue is probably with the parsing.

From looking at the source, it seems like this error can be thrown only in this line. In particular it should only be thrown when there are no benches in the file.

Now, what is really confusing me is that I ran the code in extractCargoResult on the bench output here, and it seems to be parsing it correctly.

Any ideas on what is going wrong? At the moment my best guess is some sort of version error but I might be horribly wrong.

[WizardOfMenlo]

Hi there, I just noticed that this was already fixed in #26 but there has not been a release since then. This explains the weird behavior I was observing. Is there any news on when a new release will happen?

[rhysd]

I'm sorry that I forgot releasing. Now v1.8.1 should be available. Closing.