possible to ripgrep encrypted files, or virtual filesystem?

[bionicles]

Describe your feature request

To comply with certain regulations around security and privacy, I'd like to enable users to search encryped data.

Is it possible to either,

A) store files encrypted and streaming-decrypt like https://docs.rs/cryptostream/0.3.1/cryptostream/ them right before ripgrep?
or
B) store nothing, and just have a in-memory virtual filesystem like https://docs.rs/vfs/0.5.1/vfs/, and ripgrep over that, then delete the virtual filesystem?
or
C) all of the above, streaming-decrypt an encrypted virtual filesystem and pipe it into ripgrep

This would be pretty rad

[BurntSushi]

For (A), I would expect that you could accomplish this by writing a small program that decrypts stdin to stdout and use the --pre flag. There's more about that in the guide: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#preprocessor

For (B), if it's a normal file system that any tool can interact with as if it were a file system, then I don't see why it wouldn't work? Try it!

[bionicles]

Dear @BurntSushi,

Thanks for your prompt reply and the help. I did figure out how to make a little demo, it's pretty cool and I want to polish it.

If you pipe into Ripgrep, it treats it like one big file, so you don't get filenames, and need to re-decrypt the file every time you want to change the query.

That prompted me to want to try fzf + ripgrep, which is beautiful, but only if we make a filesystem, which could theoretically enable snooping. I made a RAM Disk, and am looking at tmpfs, but here's a question:

• Does RG cache data? If so, where? Ideally, one could specify a cache path, and put it with the data; then we can delete data and cache together and subject both to the same security / sandbox rules. Otherwise, it would suffice to know the location, to delete it at the end of the search

• Would you be open-minded to document direct linkage from another Rust crate for something like Terminal UI? That would enable fast search UX without a virtual filesystem to secure. Could just decrypt the data into Rust, search with RG, render results. I could call CLI from Rust but seems messy.

Thanks for the epic tool! I'm excited!

Bion

[BurntSushi]

Does RG cache data? If so, where?

No, and it never will.

Would you be open-minded to document direct linkage from another Rust crate for something like Terminal UI?

The only interface ripgrep exposes is its CLI interface. It has a --json output mode for exactly this purpose.

You might have better luck with the ripgrep-all project, which specializes a bit more in integrations with other things. I would suggest that you give more concrete examples of what you want to do. Include shell commands if possible, for example.

[bionicles]

Thanks for the tip about --json. Here's some code (tested on a mac):

# download some synthetic medical records
# https://storage.googleapis.com/synthea-public/synthea_sample_data_fhir_r4_sep2019.zip
# unzip to /data/records folder wherever you want to do this. Run commands from within the data folder

# there are some scripts out there to set this more securely
export MYPASSWORD="test"

# make a ramdisk
diskutil erasevolume HFS+ 'RAMDisk' `hdiutil attach -nobrowse -nomount ram://4194304`

rga-fzf() {
	RG_PREFIX="rga --files-with-matches"
	local file
	file="$(
		FZF_DEFAULT_COMMAND="$RG_PREFIX '$1'" \
			fzf --sort --preview="[[ ! -z {} ]] && rga --pretty --context 5 {q} {}" \
				--phony -q "$1" \
				--bind "change:reload:$RG_PREFIX {q}" \
				--preview-window="70%:wrap"
	)"
}

# compress a folder, then encrypt it with password $MYPASSWORD 
tar c - records | lz4 -c | openssl enc -e -aes-256-cbc -pbkdf2 -out records.tar.lz4 -pass pass:$MYPASSWORD

# decrypt, decompress the data to the ramdisk
openssl enc -d -aes-256-cbc -pbkdf2 -pass pass:$MYPASSWORD -in records.tar.lz4 | lz4 -dc | tar -x /Volumes/RAMDisk

# go onto the ramdisk and search!
cd /Volumes/RAMDisk && rga-fzf

# unmount the ramdisk to clear the data
umount /Volumes/RAMDisk

# delete the RGA cache
rm ~/Library/Caches/rga
# C:\Users\username\AppData\Local\rga # windows
# ~/.cache/rga # linux

what i'd rather do:

openssl enc -d -aes-256-cbc -pbkdf2 -pass pass:$MYPASSWORD -in records.tar.lz4 | lz4 -dc | $SEARCH_TOOL

however, this treats the text as one big file, so you don't get filenames

[BurntSushi]

I don't maintain ripgrep-all. Please ask them about it. Please consider producing a minimal example that's easy for others to try.