Core Services: 3.11 tracking issue

[keegancsmith]

Theme: What will break with 80k repositories across multiple code hosts?

Q4 Vision Statement: Sourcegraph operations are fast (e.g. search), instances scale to 80k repositories, private code is secure and respects ACLs for every authorization provider, and the product settings are easy to set up and understand for our largest customers.

Core Services Roadmap

Housekeeping

• @unknwon 12/16 - 12/17 OOO, 12/18 - 12/23 Part-time (CST +8 timezone)

Required

• RFC 40: Explicit Repository Permissions Model - Support explicit repository permissions model #6086 [@unknwon]
• RFC 45: Extending Bitbucket Server Plugin Permissions Layer - Extending Bitbucket Server Plugin #6091 [@kzh]
• RFC 48: Structural Search Production Ready - Structural search improvements and documentation #6679 [@rvantonder]
• RFC 79: Refactor CI Pipeline - - Refine CI Pipeline tracking issue #6690 [@keegancsmith]
• RFC ??: GitLab Production Ready - GitLab Sourcegraph.com production readiness #6738 [@keegancsmith]
• RFC 70: Expose full repository languages/file types inventory via GraphQL - [@ryanslade]

Polish

These are smaller tasks which anyone can do. They are all stretch tasks, with the main tasks being first priority.

Note: If the task is assigned to someone already, you can do it (please ping the person first).

• Rev not found error for commits that exist on code host Rev not found error for commits that exist on code host #6539
• On any change to external services, restart sync On any change to external services, restart sync #6058
• 3.9.2: panic during gitserver cleanup 3.9.2: panic during gitserver cleanup #6710
• Gitserver should auto-recover from corrupt repository clones gitserver should auto-recover from corrupt repository clones #6676
• Default symbol sidebar to using Zoekt for indexed branches Default symbol sidebar to using Zoekt for indexed branches #6078
• Verify sourcegraph.com falls back to non-indexed search when indexed pod is down Verify sourcegraph.com falls back to non-indexed search when indexed pod is down #6057
• Speed improvements for repohascommitafter Address repohascommitafter search filter performance bottleneck #6745
• Setup 80k repos on k8s.sgdev.org (old issue Setup 80k repos on k8s.sgdev.org #5726, now we have zoekt scaling try again).
• Band aid for poor diff/commit search experience https://github.com/sourcegraph/sourcegraph/issues/5519#issuecomment-555226517
• Backend returns invalid filter chip with space "lang:ignore list" Backend returns invalid filter chip with space "lang:ignore list" #6498

3.10 tracking issue is #6031

[keegancsmith]

@sourcegraph/core-services please update the issue description with what you might want to work on. Defer on the side of adding too much, then we can trim it down. Please do so by EOD Thursday (@kzh @unknwon @rvantonder). @mrnugget @ryanslade @tsenart I assume automation work will be tracked separately.

Each item you add to the tracking issue, please link to another issue with some details of what you want to do for that issue in 3.11. (ie the issue should be closeable in 3.11, even if the broader issue isn't complete).

[tsenart]

@keegancsmith: From what I saw on Slack, @rvantonder is going to be working on diff search scalability as well. Is there going to be an RFC for that too, so that we can link it here?

[keegancsmith]

�There isn't an RFC yet, and I think rijnard was gonna tackle improving a UI issue, not making diff search scale? So that was more of a polish issue (so @rvantonder please add to the polish section, or correct me if I am wrong).

[tsenart]

OK, so in that case, diff search scalability is still missing. I believe it's quite high priority as we discussed in the last Core Sync.

[rvantonder]

From what I saw on Slack, @rvantonder is going to be working on diff search scalability as well. Is there going to be an RFC for that too, so that we can link it here?

I think there is a misunderstanding here. I committed to solving the part about a better UX experience that customers are running into related to this, estimated to be a 1h task (it'll probably take a bit longer than that). That's all. I don't have the bandwidth to address the underlying scalability issue.

I've added that to the polish section.

[keegancsmith]

OK, so in that case, diff search scalability is still missing. I believe it's quite high priority as we discussed in the last Core Sync.

I chatted a bit to @nicksnyder about diff search scalability and the outcome is it is quite a big project that makes more sense to start tackling in the new year. There is some important (Sourcegraph) developer velocity tasks that will be good to address with the time we have left before jingle bells is impossible to escape from the radio. I've added a TODO in my personal tracker to start an RFC about diff search to get the ball rolling though.

[tsenart]

That makes sense to me, specially with the low-effort UX improvement we're doing in the short term. cc @christinaforney

[keegancsmith]

W48 Plan [2019-11-25]

RFC for GitLab + Sourcegraph.com production readiness is a must deliver this week (https://github.com/sourcegraph/sourcegraph/issues/6738). Otherwise I plan on working on the E2E buildkite pipeline (https://github.com/sourcegraph/sourcegraph/issues/6840) and experimenting with buildkite on VMs (https://github.com/sourcegraph/sourcegraph/issues/6887).

[keegancsmith]

W50 Plan [2019-12-09]

must deliver docker scales on our CI. Last week I spent some time working on this, but the two approaches I tried (switch to VMs, use node's docker) had shortcomings/failed to work. Last week I made some great progress on our CI, mostly focussed around performance (e2e pipeline, tweaking steps) which lead to PRs getting a green tick from CI in under 5min.

RFC for GitLab + Sourcegraph.com production readiness is a must deliver this week #6738

This was a must deliver, but I ended up not spending much time on it at all. The only thing I really contributed was improving our sentry reporting. The fun of working on our buildkite pipeline took hold of me. This week though I will ensure we can operationally respond to too much load from GitLab.com.

Meta: Forgot to do a plan for W49, I regret not doing it since the focus from
setting a must deliver is very useful.

[beyang]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.11 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you