Core Services: 3.9 tracking issue

[nicksnyder]

Theme: What will break with 80k repositories across multiple code hosts?

Required

We expect this section to grow after setting up 80k repos task is complete.

• Setup 80k repos on k8s.sgdev.org Setup 80k repos on k8s.sgdev.org #5726
• Finish src-expose src-expose: Snapshot directories into git repos and serve them #5469
• Horizontally scaling indexed-search Horizontally scaling indexed-search #5725

Polish

These are extra tasks mostly unrelated to the theme, so can slip on this milestone. However, they are generally smaller so are useful to do inbetween larger tasks.

• zoekt-indexserver: wait for frontend to start search: replace zoekt-wrapper script sleep hack #3371
• zoekt: Cleanup failed shards zoekt: Cleanup failed shards #5300
• Handle bad repositories with ref HEAD Handle bad repositories with ref HEAD #5291
• k8s.sgdev.org: symbols pod evicted k8s.sgdev.org: symbols pod evicted #5305
• Search query repohascommitafter:"january 1 2018" fails if you have an empty repository Search query repohascommitafter:"january 1 2018" fails if you have an empty repository #5149

Investigate

These tasks are similiar to the polish tasks (can slip), but require some investigation.

• Gitserver GC logic potentially not working Gitserver GC logic potentially not working #5522
• Re-index all repos after search.largeFiles changes Re-index all repos after search.largeFiles changes #4889
• GraphQL request for repository returning null GraphQL request for repository right after adding external service and waiting for repo to be mirrored returns null (even though UI shows repo) #5286

This tracking issue does not include work being done for automation, which is tracked separately in #5678.

3.8 tracking issue is #5147

[keegancsmith]

@kzh @christinaforney @nicksnyder I've done some triaging, and updated the issue description with a list of tasks. What is missing is a "bigger" task for @kzh, but I believe once he has setup 80k repos on k8s.sgdev and seen what breaks, we will find the bigger task. @kzh I just assigned you some work, but it is all WIP, so we must discuss this still before finalising.

[keegancsmith]

Trying something new, posting my weekly plan note I normally just keep in my personal org-mode files. Will keep posting these if others find them useful.

W39 Plan [2019-09-30 Mon]

Weeks left to branch cut: 2.5
Deadlines: Performance review
Need to ship src-expose for customers. Then I can fully focus on merging v0 of horizontal search.
No other high priority short tasks planned or expected this week, so expect to have lots of focus time on code.
Last week had a mix of vacation and feeling ill so didn't have much focus time for code (but did lots of planning/reviews).

[kzh]

Keegan's meeting notes:

• At 58k, nothing has broken yet.
• Biggest problem was editing a config and nothing appearing, but that is fixed on master.
• Next concern was not knowing what Sourcegraph is doing. Probably can do some
  work around better info on what SG is doing in aggregate (which repos are
  cloning, which indexserver queue, etc). Or show notifications directly
  related to admin actions (like indexing/cloning a new repo).

Next steps

• Document frustrations in admin experience.
• Investigate logs to find out what may be going wrong.
• Re-investigate once at 80k. Should include using the product a bit (basic
  code intel, symbol sidebar, etc).
• We probably need to do a load test at 80k if everything is just working.

[keegancsmith]

W40 Plan [2019-10-07 Mon]

Working days left to branch cut: 6
Polish up src-expose after some feedback from Ryan (demoing to customers next week). Must ship v0 of horizontal search. I expect to have the main PR out by Thursday. Decent risk of final version (v2 in RFC) not being ready for branch cut, but will be in a state were customers can start trying it in 3.9.
Spent more time last week on src-expose than I expected (about 3 days vs 1 I expected). Sent out the infrastructure change I was most worried about, migrating indexed-search from a Deployment to StatefulSet. Also did a little bit of polish work every day which was satisfying.

[tsenart]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.9 release is scheduled for tomorrow at 10:00 CEST.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

[keegancsmith]

Closing this issue out, we will create a 3.10 issue this week. Quick retrospective from my perspective:

• Lots of items in our CHANGELOG :)
  • Shipped a nice solution for streaming sync early on in the iteration. So early that it isn't even represented in our plan for 3.9.
  • We fixed a lot of polish issues this milestone, which I am very happy with.
• Didn't spend much time on horizontal search, even though it was a top of line issue. This was an issue with personal prioritisation on my part, where I kept prioritising other work that came up / src-expose over it. Last week I even had v0 as a must ship in my plan, but I spent very little time on it.
• It was really nice to not having last release take up much time. IE the testing process was minimal / and no major fires from customers. It felt mostly like another week, so we could continue our focus on creating/fixing stuff.