Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Undo doing less bench iterations under cfg(test) #6021

Merged
merged 1 commit into from
Jul 24, 2024

Conversation

Wumpf
Copy link
Member

@Wumpf Wumpf commented Jul 23, 2024

Messed this up recently, was under the impression that this limits the iteration count only when running tests. Need to find another way to do that.

Still don't quite understand why that happens, I was fairly convinced that cfg(test) is not set for benchmarks in benches directory.

@Wumpf Wumpf requested a review from a team as a code owner July 23, 2024 08:00
@nical nical merged commit 9b680e6 into gfx-rs:trunk Jul 24, 2024
25 checks passed
@teoxoy
Copy link
Member

teoxoy commented Jul 24, 2024

I think we should lower the numbers as the mac CI is now intermittently timing out.

It was close to timing out in this PR, timed out on trunk and #5714 causes it to time out 100% of the time due to the overhead of validation.

        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Computepass Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Submit Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Computepass Time)
        PASS [  78.520s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Computepass Time)
        PASS [  82.692s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Computepass Time)
        PASS [  83.268s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Submit Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Submit Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/4 computepasses x 2500 dispatches (Computepass Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/4 computepasses x 2500 dispatches (Submit Time)
        PASS [  77.072s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Submit Time)
        PASS [  75.059s] wgpu-benchmark::bench/root Computepass: Single Threaded/4 computepasses x 2500 dispatches (Computepass Time)
        PASS [  75.075s] wgpu-benchmark::bench/root Computepass: Single Threaded/4 computepasses x 2500 dispatches (Submit Time)

In this PR (https://github.com/gfx-rs/wgpu/actions/runs/10055037477/job/27791035441)

        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Computepass Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Submit Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Computepass Time)
        PASS [  70.293s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Computepass Time)
 TERMINATING [> 90.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Computepass Time)
 TERMINATING [> 90.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Submit Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Submit Time)
 TERMINATING [> 90.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Submit Time)

On trunk (https://github.com/gfx-rs/wgpu/actions/runs/10071973334/job/27843338838)

        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Computepass Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Submit Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Computepass Time)
        PASS [  77.019s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Computepass Time)
 TERMINATING [> 90.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Computepass Time)
 TERMINATING [> 90.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/1 computepasses x 10000 dispatches (Submit Time)
        SLOW [> 45.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Submit Time)
 TERMINATING [> 90.000s] wgpu-benchmark::bench/root Computepass: Single Threaded/2 computepasses x 5000 dispatches (Submit Time)

In #5714 (https://github.com/gfx-rs/wgpu/actions/runs/10071973334/job/27843338838)

@teoxoy
Copy link
Member

teoxoy commented Jul 24, 2024

I'll play around with the numbers and lower it in #5714.

@Wumpf Wumpf deleted the undo-limit-iterations-for-bench branch July 24, 2024 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants