Add more optional threading

[ErikQQY]

This PR continues the work by @ranocha in PR #1508 and adds the optional threading to all of the explicit RK methods.

Fix: #1511

I still have some questions:

1. As for the algorithm Vern7 and Vern9, in the second perform step function concerning Vern7Cache and Vern9Cache:

perform_step!(integrator, cache::Vern7Cache{<:Array}, repeat_step = false)

perform_step!(integrator, cache::Vern9Cache{<:Array}, repeat_step = false)

it uses @simd rather than @.., do we need to add optional threading here too?

2. As for the algorithms Feagin, I didn't see any @.. used in these algorithms, do we need to add optional threading for these algorithms?

[ErikQQY]

I think maybe the buildkite test failure is not caused by this PR?

[ChrisRackauckas]

That's a stochastic failure. Don't worry about that one.

[ChrisRackauckas]

it uses @simd rather than @.., do we need to add optional threading here too?

Yes. Those branches are for compile time reduction, though I'm not sure that they are necessary anymore since when you do broadcast=false on @.. the compile time should be sane (without it, @.. does have compilation over head). Could you take the time to profile whether those dispatches are still needed? If not, the ones for Verner and Tsit5 methods can just be removed.

[ChrisRackauckas]

That or Polyester. Will you be following up with that? Otherwise we should make sure that we document that threading choices don't apply to those methods when Float64, but I think a simple deletion is all that's necessary (and testing first solve times)

[ErikQQY]

Sure, I will follow up with that. I just benchmarked the Vern7 scheme:

Before I removed that @muladd function perform_step!(integrator, cache::Vern7Cache{<:Array}, repeat_step = false) dispatch:

However, after I removed that dispatch, the performance got a little declined🤔:

[ChrisRackauckas]

That's because it wasn't actually using multithreading. If you solve with Array{Float64}, it's to hit the Array{Float64} branches. My guess is that this case is too small for multithreading to help.

But @benchmark removes compile times. We need to also bench compile times like SciML/DifferentialEquations.jl#786 (comment)

[ErikQQY]

I used SnoopCompile.jl to test the compile time:

Before I removed that branch:

julia> tinf = @snoopi_deep solve(prob,Vern7(OrdinaryDiffEq.True()))
InferenceTimingNode: 2.508216/3.608172 on Core.Compiler.Timings.ROOT() with 3 direct children

And the corresponding flamegraph:

After I removed that branch:

julia> tinf = @snoopi_deep solve(prob,Vern7(OrdinaryDiffEq.True()))
InferenceTimingNode: 2.427201/3.539494 on Core.Compiler.Timings.ROOT() with 3 direct children

The corresponding flamegraph:

Is this the correct way to test the compile time of the Vern7 scheme? Given the result is approximately equal, I think we can remove that branch now?

[ChrisRackauckas]

Yeah, looks like it can be safely removed.

[ErikQQY]

OK, I will send another PR to remove these branches.