feat: add option to persist rule index

[jbedard]

What type of PR is this?
Feature

What package or component does this PR mostly affect?
cmd/gazelle

What does this PR do? Why is it needed?

Add the ability to persist the RuleIndex across runs to allow updating only the specified directories when indexing is required.

Which issues(s) does this PR fix?

Ref: #1181

[fmeum]

How large does this file get? I'm wondering whether json.NewDecoder could be more appropriate here. Doesn't matter for the first iteration though.

[jbedard]

I assume this could get fairly large and I've been wondering if we'll want something more compact then json. But I figured if this is EXPERIMENTAL and we can break it whenever we want then plain json is a good place to start?

[jbedard]

Suggestions for the naming of the cli arg? I used "indexdb" because this is caching the RuleIndex data 🤷
Maybe we should use a --experimental prefix?

[jbedard]

Just to re-iterate a few points and limitations of this:

This is EXPERIMENTAL with no api guarantees. 99% chance there will be breaking changes. You should never commit this file to source-control.

When this EXPERIMENTAL feature is used:

• the resolver.Imports/Embeds() calls for the entire workspace (the "rule index") is written to disk
• when specifying dirs (default: .) on the command line only those directories are updated in the "rule index"
• only the dirs specified on the command line will be traversed, therefore:
  • only those dirs will have Imports/Embeds() invoked
  • only those dirs will have Fix(), GenerateRules() invoked
  • only those dirs will have BUILDs updated, warnings generated etc.
  • only those dirs will have things like resolve warnings outputted

If resolver.Imports/Embeds() have side-effects then this won't work very well when they are cached and not invoked. Don't do that.

There is no cache invalidation here. It is entirely up to the user to pass the correct list of dirs that require reprocessing.

FUTURE: if those updated dirs changed what rules they generate and the Imports/Embeds() of those rules change it is possible that other rules outside dirs are now referencing something incorrect. I think if Imports/Embeds() of those dirs change we might want to re-run Fix + GenerateRules for all directories.

[jbedard]

@fmeum did you have any opinions on this going forward? Is this something you think should live in gazelle?

For large repos even simply walking the fs can be slow (even after the changes making it concurrent) so any type of incremental BUILD-generation needs to run on a subset of the workspace which is the goal of this PR.

[fmeum]

I could very well see this live in Gazelle. Do you think that any of the two recent proposals #1891 and #1907 could substantially improve the situation for your large repos? If so, I would prefer to explore this route first, mostly since serialization, even if experimental, often ends up constraining future efforts to change the persisted data structure. Not doing unnecessary work in the first place could save us the complexity of having to persist this work.

[jbedard]

Avoiding any type of caching/persisting would always be ideal if we can do it 👍

For persisting/serialization... I would prefer just thinking of it as caching, it should never be required. If we just throw a hash at the start of the serialized data then we can ignore any cached data from an older version of gazelle or different list of languages etc (hash = gazelle version + every language compiled into the gazelle binary maybe), then we should never have to deal with older data formats at least.

[jbedard]

I'll try to leave comments on #1891 and #1907. I worry any major improvements will require breaking changes to truly help though, maybe completely replacing traversal+indexing+resolving with something lazy by default?

I think a lot of scenarios might still need to know the fs tree in order to do things based on convention or lazy indexing though, but if we can then do the actual indexing+resolving in a lazy fashion that would be great.