Why ugrep-v3.1.7-release is 1.95 times faster than rg-v12.1.1-release?

[saifali-ch]

After reading some loooong loooong benchmarks and a lot of grep-like tools posts from different sites. I thought to test the speed of ugrep and rg myself.

So

• I used hyperfine 1.11.0 with -r 10 runs as you can see in the picture below.
• I did a simple string literal search against a 69 lines text file with both tools.
• Surprisingly, ugrep is 13.5ms faster than rg. And it just a single file with only 69 lines.
• I don't know why. Could anyone please explain what I'm missing here or doing wrong?

Laptop Specs

• Windows 10
• Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
• 8GB DDR3 RAM
• ADATA SU630 480GB SSD
• Dell Lattitude e7450
• And approx ~36 chrome tabs 😜

Screenshot

[BurntSushi]

13ms is nominal. That could be, for example, explained by the number of threads each tool spins up. Your benchmark does not control for that. It could also be explained by different searching techniques, e.g., mmap vs standard read calls. Or maybe ripgrep is reading a config file that you have. I'm on mobile and haven't done any analysis, so I can't say for sure. My point is that the benchmark is somewhat senseless. Please find a benchmark with more than a few milliseconds of difference. For example, by substantially increasing corpus size.

Surprisingly, ugrep is 13.5ms faster than rg. And it just a single file with only 69 lines.

Why are you surprised?

[saifali-ch]

I like rg by heart and it was just a question (either to correct my approach or to check if it is really an issue).

That could be, for example, explained by the number of threads each tool spins up. Your benchmark does not control for that.

I have read your blogpost and I know my test is silly as compared to your way of benchmarking. But I supposed, hyperfine should be fair in this regard.

Or maybe ripgrep is reading a config file that you have.

I don't use any configuration files.

My point is that the benchmark is somewhat senseless.

I agree, but please you should benchmark at least once, I'm sure you will see the difference above few milliseconds when using a large corpus.

Why are you surprised?

Because, if for this tiny file it is 13ms then for large files it might increase substantially. But after using a large corpus, it turns out to be that it didn't increase that much as I have thought earlier.

Please find a benchmark with more than a few milliseconds of difference.
For example, by substantially increasing corpus size.

Sure! this time I have used three text files and the difference is 200+ milliseconds. Can I improve this? Any thoughts?

• 156 million lines combined (see the attached SS)
• 1.59GB total size

[BurntSushi]

But I supposed, hyperfine should be fair in this regard.

hyperfine has nothing to do with this. hyperfine is just a nice way of timing commands. I did not question your observation. I questioned its relevance.

Because, if for this tiny file it is 13ms then for large files it might increase substantially.

That's an incorrect assumption and is often wrong. For the reasons I stated and more.

I have already run ripgrep vs ugrep benchmarks. Some are in the README. Find me a better benchmark and I would be happy to analyze it when I have time.

Sure! this time I have used three text files and the difference is 200+ milliseconds. Can I improve this? Any thoughts?

Dunno. I need the ability to run it.

You shouldn't be surprised if ugrep is faster in some cases. GNU grep is faster than ripgrep in some cases too. Different algorithms are used by each tool. All such algorithms employ heuristics, which implies there will always be some cases where one tool is superior than the other. This is why benchmarking is hard, because it's easy to introduce bias.

[saifali-ch]

That's an incorrect assumption and is often wrong. For the reasons I stated and more.

As I have already stated above (.....increase substantially. But after using a large corpus...) that my assumption was incorrect in this regard. So, please bear with me.

Find me a better benchmark and I would be happy to analyze it when I have time.

I will when I have some free time and proper environment setup. Also, after using both tools for a long timespan.

You shouldn't be surprised if ugrep is faster in some cases. ... All such algorithms employ heuristics.

I know that each tool has some edge cases. According to my observation ugrep was faster in literal search than ripgrep while ripgrep was faster in regex search. (Yet, I'm not 100% confirm and that's the reason I started the discussion)

This is why benchmarking is hard, because it's easy to introduce bias.

Yes, it is. But I'm not biased against any tool.

As in your 2020 benchmark (linux_literal) ripgrep (0.130ms) was ~180ms faster than ugrep (0.309ms).
But when I did my (not good enough) test, the results were the opposite.
Further, I saw that ugrep detected those (text) files as binary. This might be the issue and I'll do some more investigation about it.