About the SIMD acceleration feature of ripgrep.

[hongyi-zhao]

On Ubuntu 20.04, I compiled the git master version of ripgrep with the following command:

$ proxychains-ng-socks5 cargo build --release --features 'pcre2' 

$ target/release/rg --version
ripgrep 12.1.1 (rev c7730d1f3a)
-SIMD -AVX (compiled)
+SIMD +AVX (runtime)

As you can see, it also shows the SIMD featrue at the runtime level, but I haven't used any of the parameters here, as shown below:

RUSTFLAGS="-C target-cpu=native" cargo build --release --features 'simd-accel'

Any hints will be highly appreciated.

Regards,
HY

[BurntSushi]

In Rust, some uses of SIMD work on Rust stable. Some don't. Specifically, platform dependent use of specific vendor intrinsics on x86_64 works on stable Rust. And in particular, one can compile portable binaries where only specific functions in the code will use, say, 256-bit vectors while the rest of the code won't. (And it would be undefined behavior to call said functions on platforms that don't support 256-bit vectors.)

Most of the SIMD in ripgrep and its dependencies uses this form of SIMD.

But there is one dependency that doesn't: encoding_rs. That crate sticks with the unstable but platform independent higher level SIMD APIs. Enabling those requires using Rust nightly. Thus, when building ripgrep, you must opt into that.

The reasons why encoding_rs sticks with the unstable portable API are beyond the scope of this discussion forum. The TL;DR is that it's an ecosystem concern of becoming too tightly coupled to x86_64.

The only thing simd-accel takes advantage of in the context of ripgrep is faster UTF-16 transcoding to UTF-8. If you aren't searching UTF-16, then simd-accel doesn't matter.

[hongyi-zhao]

Thank you very much for your in-depth explanation.

The only thing simd-accel takes advantage of in the context of ripgrep is faster UTF-16 transcoding to UTF-8. If you aren't searching UTF-16, then simd-accel doesn't matter.

Generally speaking, on modern OSes, the default locale is UTF-8. So the searching UTF-16 encoded file names or contents shouldn't be a routine task.

Regards,
HY

[BurntSushi]

It's not uncommon on Windows.

[hongyi-zhao]

Why do they have to use UTF-16?

[BurntSushi]

I don't understand your question. UTF-16 is prevalent on Windows. It is prevalent because Windows adopted UTF-16 (and UCS-2 before it, IIRC) back in the 90s before UTF-8 became the "obvious" winner.

Other ecosystems, like Java, use UTF-16 for similar reasons.

But I am not an expert on this history. What matters to me is what users need. This was a highly requested feature by Windows users before it existed in ripgrep.

[hongyi-zhao]

Thanks again. You answered my question very well.