Why does ripgrep traverse into directories that don't match the provided glob?

[oliversalzburg]

What version of ripgrep are you using?

ripgrep 11.0.1 (rev 973de50c9e)
-SIMD -AVX (compiled)

How did you install ripgrep?

Downloaded the binary from the GitHub Releases page.

What operating system are you using ripgrep on?

Windows 10 x64

Describe your question, feature request, or bug.

Through an issue I was experiencing with an extension for VS Code, I looked at the behavior of ripgrep, given the underlying command that was causing the excessive load.

The relevant parts of the command are:

rg --files --follow --no-ignore -g /test/**/*.js

This will quickly produce the desired file names from the test subfolder in the current working directory and then it will print out huge chains of folder names (resulting from symlinks) in a folder named node_modules in the current working directory.

Now my question is, why does ripgrep even traverse down into the node_modules folder, when it doesn't match the requested glob anyway? Is that by-design or is that unintended behavior?

[BurntSushi]

Now my question is, why does ripgrep even traverse down into the node_modules folder, when it doesn't match the requested glob anyway? Is that by-design or is that unintended behavior?

Neither. It's simply a missed performance optimization. While it's obvious to us humans that looking in node_modules is a waste of time, ripgrep doesn't know that because it doesn't do the sophisticated analysis on globs required to implement the optimization you want here. In particular, ripgrep must descend into directories when given a glob like this unless the glob specifically excludes that directory, otherwise ripgrep might miss files. For example, the path /test does not match the glob /test/**/*.js. So in order to implement this optimization, ripgrep would need to reason about glob prefixes and the like.

Moreover, the command you've presented is fairly idiosyncratic. I don't understand why you're using in a glob for this instead of just using your shell's glob expansion to search the test directory directly. e.g., rg --files --follow --no-ignore /test/**/*.js. If your shell isn't capable of that, then try rg --files --follow --no-ignore /test -g '*.js'.

Invariably, this is a duplicate of #546, so I'm going to close this one as well for the same reason.

[oliversalzburg]

Awesome. Thanks a lot for the detailed explanation. The presented command is not issued in a shell and I have no direct control over it either, as it is third-party code invoking ripgrep. I was merely trying to understand why things are happening as they are. You explained it well. Thanks again.

…

On Mon, Jun 17, 2019, 23:23 Andrew Gallant ***@***.***> wrote: Now my question is, why does ripgrep even traverse down into the node_modules folder, when it doesn't match the requested glob anyway? Is that by-design or is that unintended behavior? Neither. It's simply a missed performance optimization. While it's obvious to us humans that looking in node_modules is a waste of time, ripgrep doesn't know that because it doesn't do the sophisticated analysis on globs required to implement the optimization you want here. In particular, ripgrep *must* descend into directories when given a glob like this unless the glob specifically excludes that directory, otherwise ripgrep might miss files. For example, the path /test does *not* match the glob /test/**/*.js. So in order to implement this optimization, ripgrep would need to reason about glob prefixes and the like. Moreover, the command you've presented is fairly idiosyncratic. I don't understand why you're using in a glob for this instead of just using your shell's glob expansion to search the test directory directly. e.g., rg --files --follow --no-ignore /test/**/*.js. If your shell isn't capable of that, then try rg --files --follow --no-ignore /test -g '*.js'. Invariably, this is a duplicate of #546 <#546>, so I'm going to close this one as well for the same reason. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1304?email_source=notifications&email_token=AAMVARKRP5D4ZZMNGBJVKM3P276GHA5CNFSM4HY2CLO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX4PVFI#issuecomment-502856341>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMVARL5K4AE4OOFSV42EVDP276GHANCNFSM4HY2CLOQ> .