Make Knative eventing more serverless and scalable

[aslom]

Make Knative eventing more serverless and scalable

Problem
A short explanation of the problem, including relevant restrictions.

Knative eventing (sources, channels, brokers, ...) should scale to zero when not used and scale up in proportion to number of events

Persona:
Which persona is this feature for?

System Integrator, Event consumer (developer)

Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.

When no events coming into Knative eventing then it should scale down to zero (knative eventing services should take minimal or no memory or cpu, channels should scale down as well). When events are received over HTTP it should scale up.

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

weeks

Additional context (optional)
Add any other context about the feature request here.

Let start discussion about what would be the lowest hanging fruits?

[aslom]

Improvements should help with related issue about Broker internal design improvements #1555

[aslom]

There may be other related issue such as about making control-plane scalable as well?

[aslom]

I would like to find who is interested in this topic: @nachocano @slinkydeveloper @Harwayne @matzew @lionelvillard ?

[aslom]

What would be the best way to capture possible options short term (MVP, before 1.0?) and long term ideal solution?

I am thinking to start with design docs for each sub-topic such as sources and channels?
Using design doc using template?
https://github.com/knative/community/blob/master/CONTRIBUTING.md#design-documents
https://docs.google.com/document/d/1QRREfL8gSVSURHMkDLjDtMh1BBsNiwfmaBiGnJUDQ2g/edit

[mikehelmick]

/cc @grantr

[nachocano]

Related to this: #2060 #2061 #2083

…

On Wed, Nov 6, 2019 at 8:36 AM Mike Helmick ***@***.***> wrote: /cc @grantr <https://github.com/grantr> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2152?email_source=notifications&email_token=ABD65DF33MYNWNTLZJEUXBLQSLW7HA5CNFSM4JJWYSR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDHFGAY#issuecomment-550392579>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABD65DGKUC33DZ4BOIMAXRTQSLW7HANCNFSM4JJWYSRQ> .

[grantr]

/area performance

[slinkydeveloper]

I think it's quite interesting the idea to scale up, but I'm not sure it would be simple to go to 0.
Are we sure that we can consider every knative component stateless, so it can be scaled down to 0 without data loss?
+1 for the document

[aslom]

@slinkydeveloper hopefully we can scale control-plane to zero - only spin it up when CR is created/modified/deleted? For data plan may be more complex as some sources requires pull or persistent polling connection so something needs ot keep running even if there is no events recevied?

[slinkydeveloper]

There is any way atm to let k8s spin up control-plane only when CR changes happens?

[antoineco]

@slinkydeveloper that would require one component to always be up and running to watch for changes to relevant resources and then start the corresponding controller. Kubernetes does not actively notify controllers when changes occur (push), the controllers maintain a long connection to receive events about resources they are interested in (pull).

[slinkydeveloper]

@antoineco If that's the case, I'm kinda worried this kind of solution only makes the architecture more complex without giving relevant benefits. If we consider only eventing, it's quite easy, but eventing itself is somewhat "extensible" adding eventing-contrib components, so how do we manage all components with same "listener & autoscaler" stuff? If we end up with multiple "listener & autoscaler", it doesn't make any sense 😄
BTW for data-plane I really agree we should autoscale up and down (not necessarely to 0)

[antoineco]

Fully agree with everything you said, I really don't see any benefit scaling the control plane.

[aslom]

@antoineco @slinkydeveloper that starts to matter if you have very small (and want scaling to zero to conserve resources) and very large knative cluster (lot of activity in control plane) and may ot of controllers installed for different source and channel types?

Also that may be bigger issue for multi-tenant cluster as you would want to run all supported types of sources and channels so tenants can easily create them?

I am going ot open separate issue to discuss the control plane - even if we decide there is no need to improve it now we will have record of it?

[antoineco]

Moved my comment to #2161 (comment)

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[aslom]

/reopen

[slinkydeveloper]

Is this still relevant? Since I see you worked on this in more specific issues, can we close this more general issue?

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.