Core Services: 3.10 tracking issue

[keegancsmith]

Theme: What will break with 80k repositories across multiple code hosts?

Q4 Vision Statement: Sourcegraph operations are fast (e.g. search), instances scale to 80k repositories, private code is secure and respects ACLs for every authorization provider, and the product settings are easy to set up and understand for our largest customers.

Housekeeping

• @rvantonder joins Core Services 🎉
• @ryanslade joins Core Services Nov 4th 🎉
• @keegancsmith out of office Nov 4th->15th

Required

• RFC 30: Zoekt Horizontal Scaling - Horizontally scaling indexed-search #5725 [@keegancsmith]
• RFC 40: Explicit Repository Permissions Model - Support explicit repository permissions model #6086 [@unknwon]
• RFC 45: Extending Bitbucket Server Plugin Webhooks - Extending Bitbucket Server Plugin #6091 [@kzh]
• RFC 48: Structural Search - Add support for Structural Search #6073 [@rvantonder]
• RFC 70: Expose full repository languages/file types inventory via GraphQL - #2587 [@ryanslade]

@ryanslade we have assigned a task for you. Once you have joined and onboarded a bit we can change the task, this is a good initial task to try out but there may be something more appropriate. It is also under the "required" section. However, you will be onboarding so this can slip. We can also update this document with your real GitHub handle when appropriate :)

Polish

These are smaller tasks which anyone can do. They are all stretch tasks, with the main tasks being first priority.

Note: If the task is assigned to someone already, you can do it (please ping the person first).

• On any change to external services, restart sync On any change to external services, restart sync #6058
• Default symbol sidebar to using Zoekt for indexed branches Default symbol sidebar to using Zoekt for indexed branches #6078
• Single ctags docker image for both unindexed symbol svc and Zoekt Single ctags docker image #6080
• Verify sourcegraph.com falls back to non-indexed search when indexed pod is down Verify sourcegraph.com falls back to non-indexed search when indexed pod is down #6057
• Rename Sourcegraph Go module to succinct name Update module to sourcegraph.com #4982
• Drop recent_searches table/feature Drop recent_searches table/feature #6056
• Bitbucket Cloud repo syncing should allow hyphens in team names Bitbucket Cloud repo syncing should allow hyphens in team names #6154
• Bitbucket Cloud external service doesn't provide "exclude" config option Bitbucket Cloud external service doesn't provide "exclude" config option #6035
• Serious problems with external service auth provider config are never shown Serious problems with external service auth provider config are never shown #6061

3.9 tracking issue is #5710

[tsenart]

@keegancsmith: Are we going to organize a planning meeting soonish?

[keegancsmith]

@keegancsmith: Are we going to organize a planning meeting soonish?

yes.

@sourcegraph/core-services please update the issue description with what you might want to work on. Defer on the side of adding too much, then we can trim it down in our planning meeting. Please do so by EOD Wednesday (@kzh @unknwon). I assume automation work will be tracked separately.

Edit: Each item you add to the tracking issue, please link to another issue with some details of what you want to do for that issue in 3.10. (ie the issue should be closeable in 3.10, even if the broader issue isn't complete).

[tsenart]

I assume automation work will be tracked separately.

Yes.

[tsenart]

Please invite me to the planning meeting.

[unknwon]

Note: Ryan#2 will join core services during this iteration, Oct 30th IIRC. #2587: Expose full repository languages/file types inventory via GraphQL is what we proposed for his first project during our roadmap meeting.

[rvantonder]

I think my proposed work on structural search falls under core services, so added it as well.

[unknwon]

@rvantonder Welcome to the core services 😁

[keegancsmith]

I've updated the description taking into account all the edits done. Please give a 👍 if it looks good to you, otherwise comment / edit then do a thumbs up.

@kzh @unknwon can you create tracking issues for the implementation work on the respective RFCs.

The only thing that is missing that seems like it may make sense to do in 3.10 is RFC 33: Usage-based billing. Thoughts?

Please propose anything we need to discuss that is currently missing from the description. As it stands this seems pretty well shaped, so I don't think we need to do a full meeting on it. We can also continue some discussion async on this issue. If all goes well I am thinking we can discuss it in our next sync (Tuesday). @tsenart I know you asked about a planning meeting, what do you think about doing things more async + a shorter meeting as part of our weekly sync?

[tsenart]

@tsenart I know you asked about a planning meeting, what do you think about doing things more async + a shorter meeting as part of our weekly sync?

Sounds good to me.

[unknwon]

@kzh @unknwon can you create tracking issues for the implementation work on the respective RFCs.

Added.

The only thing that is missing that seems like it may make sense to do in 3.10 is RFC 33: Usage-based billing. Thoughts?

Asked @dadlerj last Friday, and I believe he thinks it is not a high priority thing to do at the moment. But we could still plan it since we're going to need this anyway in the long term.

[unknwon]

Quinn added https://github.com/sourcegraph/sourcegraph/issues/5519#issuecomment-538321132 to project roadmap as Diff search across 25+ repositories - TODO prioritize at next sync with a comment.

[tsenart]

Quinn added #5519 (comment) to project roadmap as Diff search across 25+ repositories - TODO prioritize at next sync with a comment.

Added to the next Core Services sync agenda.

[keegancsmith]

W42 Plan [2019-10-21]

Working days left to vacation: 9
Singular focus this week: Ship horizontal search. Can't work on anything else unless it is unblocking someone else (review/etc). Been really bad with my personal focus on it, and instead focussed on other tasks.

[keegancsmith]

W43 Plan [2019-10-28]

Working days left to vacation: 5

Must ship: Fully tested, documented and ready for release version of horizontal search. Otherwise I plan on writing an RFC this week (undecided on what, got a few ideas).

Managed to hit my plan for last week and horizontal search v0 is in master \o/.

[keegancsmith]

Must ship: Fully tested, documented and ready for release version of horizontal search.

Shipped! Please read the important notes at https://github.com/sourcegraph/sourcegraph/issues/5725#issuecomment-548730923

Otherwise I plan on writing an RFC this week (undecided on what, got a few ideas).

Didn't manage to get around to this.

Holiday time, see you all in two weeks.

[beyang]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.10 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

[rvantonder]

Structural Search

Structural Search MVP is merged. I:

• tested that it works on the built server Docker image (I wanted to make sure the comby binary was being picked up properly).
• wanted to test on k8s.sgdev, but we are currently not tracking master, and I imagine we will update sgdev for 3.10 soon, so that I can play around with it there.
• have not entered structural search queries on sourcegraph.com, and do not want to do this yet--I prefer to do this on k8s.sgdev first :)

Structural search is activated by typing patterntype:structural in the search query input, followed by comby syntax for search patterns.

Some parts of integrating with search took longer than expected and I didn't get to RFC 48 part 2 (strictly a performance increase, no change in functionality), and triage it for next release.