Core Services: 3.8 Tracking Issue & beyond

[mrnugget]

Planning 3.8 and beyond

Main questions:

• What will break with 80k repositories?
• What's the delta between the current state of Sourcegraph and the state that would close current deals in our pipeline?

3.8

• Tomás: Improve ACL performance
  • #4812 - Global Repository Permissions Caching (@tsenart)
• Thorsten: Improve site-admin experience
  • #5145 - Use streaming in repo-updater to make syncing process more responsive
  • #5146 - Make external service syncing errors visible to site-admins
• Keegan:
  • Assisting and helping out the search team with architectural decisions & design process
  • Improve Zoekt performance
    • Horizontally scaling Zoekt
    • (If time permits: single-node performance, see Beyond)

Tickets & Bugs & misc

• Fix Quinn's issue with EXTSVC_CONFIG_FILE setting: #5134
• Allow configuring external service with orgs (do not require repos or repositoryQuery to be set if orgs is set): #4322
• Customer issue: repo-updater "failed to run repository clone purge" (the <> bug)
• Fix gitserver purging process: introduce "mark & sweep process". #5056 and comment
• Querying repository by name with github.com prefix always returns empty string for name: #5125
• Gitolite EOF: #5119
• Gitolite not cloning: #5121
• GitHub Oauth scope: #5111

Beyond

Zoekt

• Investigate, understand, tweak and improve concurrency model
• Introduce shard compaction to improve read performance (Inspiration: Cassandra + SS Tables)
• Bloom filters in main process to avoid looking into shards that won't yield results
• Improve memory usage
• Experiment with more efficient posting lists in Zoekt index: Use roaring bitmaps for content posting lists zoekt#10
• Improve parsing of newlines in Zoekt

Frontend/Zoekt

• Improve memory usage by getting rid of "list of 80k repos". Send IDs instead

Others

• Improving gitolite performance: large-scale customer reported that it takes gitolite 27 min to list all repositories (that is: without Sourcegraph being involved at all, it's pure gitolite)
• Determine resource allocation "equation" and update docs for admins to provision them accordingly for their Sourcegraph installation (e.g. https://confluence.atlassian.com/bitbucketserver/scaling-bitbucket-server-776640073.html)

Ideas for far future

• Index commit diffs (diff search right now is unindexed)
• Automation

[tsenart]

@mrnugget, @keegancsmith: Is there any context you'd like to capture from this tracking issue before we close it?

[mrnugget]

I think the most important thing to capture from my side is that in the middle of this iteration we decided that I join @tomas in the work on A8N so we can reach the goal of shipping a first MVP (RFC28) in 3.8.

That decision also means that I put my work on the streaming repo-updater in https://github.com/sourcegraph/sourcegraph/issues/5145 on hold for now, since A8N has a higher priority right now.

But I opened https://github.com/sourcegraph/sourcegraph/pull/5526 with what I have so far and that is actually a all-tests-passing version of a streaming repo-updater. As described in the PR itself, the big outstanding TODOs are testing (locally, on k8s.sgdev.org), analyzing the intermediate phases of the sync (e.g.: do we delete & re-insert repos needlessly in a single run?) and then polishing it.

[keegancsmith]

Horizontal scaling is close, but lacking proper testing. So shifting to 3.9.

[tsenart]

So can I close this issue?