Provide Benchmark.quick_compare to quickly compare methods on an object

[evanphx]

Closes #133

Any feedback welcome, especially from @zenspider

[zenspider]

This is not the quick_compare I was hoping for... but not blocking.

[zenspider]

Please note that in #133 I said nothing about "quickly comparing methods on an object". Please don't make this take an obj argument that you send on. benchmark-ips is MUCH more than running microbenchmarks on Array to know that reduce is slower.

Some code needs more complex setup.
Some code is comparing that setup.
Or anything else.

This argument limits things and would require me to make a class with my benchmark methods to get around it. THE POINT of this method is to require less structure to get benchmarks up and running and get an easy comparison of anything.

ary =  [Object.new] * 969
ary += [Assertion.new] * 3
ary += [UnexpectedError.new] * 1
ary += [Skip.new] * 27

RESULTS = ary

def report_orig
  aggregate = RESULTS.group_by { |r| r.class }
  aggregate.default = [] # dumb. group_by should provide this

  [aggregate[Assertion].size, aggregate[UnexpectedError].size, aggregate[Skip].size]
end

def report_orig2
  aggregate = RESULTS.group_by { |r| r.class }.transform_values(&:size)
  aggregate.default = 0

  [aggregate[Assertion], aggregate[UnexpectedError], aggregate[Skip]]
end

def report_tally
  aggregate = RESULTS.map { |r| r.class }
  aggregate = if RUBY_VERSION > "3.1" then
                aggregate.tally(Hash.new 0)
              else
                aggregate.tally.tap { |r| r.default = 0 }
              end

  [aggregate[Assertion], aggregate[UnexpectedError], aggregate[Skip]]
end

def report_tally2
  aggregate = RESULTS.map { |r| r.class }
  aggregate = aggregate.tally.tap { |r| r.default = 0 }

  [aggregate[Assertion], aggregate[UnexpectedError], aggregate[Skip]]
end

def report_tally3
  aggregate = RESULTS.map { |r| r.class }.tally
  aggregate.default = 0

  [aggregate[Assertion], aggregate[UnexpectedError], aggregate[Skip]]
end

[evanphx]

Correct, you didn't say anything about testing on an object. But without a receiver, who do you want the methods called on? just toplevel methods?

[evanphx]

I moved quick_compare to Kernel so it's more flexible, more along the lines of your original version.

[eregon]

What's the problem with Benchmark::IPS.quick_compare(self, :report_orig, :report_orig2, ...)?
Having the object is useful if you have more state you need to pass in.
Then for instance this state could live in an object instance.
Or the methods would be defined as singleton methods of a module, to have the necessary constant scope, etc.

I am against polluting Kernel.
I think a benchmarking harness should not monkey-patch anything.

Also for 10+ years users of this gem managed just fine without this helper method, so maybe we should not add it.
At least I think we should get multiple opinions to not add something which only works for one person.
The existing API is also the same as stdlib Benchmark, it's pretty simple.

[nateberkopec]

I'm back from the dead.

1. I too don't like polluting Kernel by default. I think it's fine if we don't require 'benchmark/ips/quick by default. But, then you're kind of missing the point of this whole feature ofc because now you have to add another 15 characters at the top of your benchmark. In my head, the interface that makes the most sense is actually Benchmark.ips_quick_compare but I'm neutral and would be fine with Benchmark::IPS.quick_compare.
2. I agree that benchmark/ips is about more than microbenchmarking, but microbenchmarking is cool too so we should minimize overhead where it's easy to do so.
3. Supporting 2.3 is rad and doesn't block this feature

If no one objects I'll pick this up and run w/it

[amomchilov]

Can we use regular kwargs here?

Suggested change

	def quick_compare(*methods)
	def quick_compare(*methods, warmup: nil, time: nil)

[amomchilov]

How about using one of the faster forms, like the explicit while loop, or string form?

[nateberkopec]

I've changed it to my take.

1. I tried @eregon's suggestion to send it directly to reduce overhead BUT that means you can't just define a method on main/Kernel and then quick compare it, because that method you just defined is private. The more important part of this feature is convenience rather than suitability for microbenching, so I decided to keep __send__
2. It's been moved to Benchmark.ips_quick.
3. Expanded docs.

LMK about objections, otherwise I'll add a test and merge

[MatheusRich]

I love this!

[MatheusRich]

I wonder if having quick in the name would make people think that it runs faster than the normal version

[nateberkopec]

Thanks for everyone's feedback. This should be the final form now, incl. a basic test. If no objections, will merge in a few days.

[nateberkopec]

Just saw the version bump here. Changing to 2.14.