New KEP for SCTP support feature #2276

janosi · 2018-06-16T18:29:46Z

KEP document for SCTP support.
Relevant PR: kubernetes/kubernetes#64973

janosi · 2018-06-16T18:46:21Z

thockin · 2018-06-18T21:02:31Z

keps/sig-network/0015-20180614-SCTP-support.md

+  - "@janosi"
+owning-sig: sig-network
+participating-sigs:
+  - sig-cloud-provider


at least ALSO sig-network, not sure cloud-provider will matter here?

OK, so it is not enough then that sig-network is the owning-sig.

thockin · 2018-06-18T21:02:42Z

keps/sig-network/0015-20180614-SCTP-support.md

+reviewers:
+  - "@thockin"
+approvers:
+  - TBD


thockin · 2018-06-18T21:02:56Z

keps/sig-network/0015-20180614-SCTP-support.md

+  - TBD
+editor: TBD
+creation-date: 2018-06-14
+last-updated: yyyy-mm-dd


start with creation date

Do you mean "creation-date" should be the date when I pushed the doc to github first time?

thockin · 2018-06-18T21:03:07Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+# SCTP support
+<!---
+This is the title of the KEP.


please delete the template

thockin · 2018-06-18T21:08:41Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+### Non-Goals
+
+It is not a goal here to add SCTP support to load balancers that are provided by cloud providers. I.e. the Kubernetes user can define Services with type=LoadBalancer and Protocol=SCTP, but if the actual load balancer implementation does not support SCTP then the creation of the Service/load balancer fails.


Do we know of any that work? We should probably default to a validation error if this combination is specified.

How about NodePort ? If LB is disabled, maybe NodePort should be too

As I know, OpenStack supports SCTP as LB protocol. Though, of course we can start with the approach, that for type=LoadBalancer Protocol=SCTP is not allowed, and whenever there is a need a new PR can enable that for the relevant cloud provider.
UPDATE: I checked the OpenStack LBaaS API (again), and SCTP is not supported on the API. So, I remove my OpenStack related modification and I make sure that SCTP becomes a restricted protocol for the OpenStack case, too. Also it means, that for the time being we can restrict the usage of SCTP with type=LoadBalancer, as you proposed.

I will check the NodePort use case.

thockin · 2018-06-18T21:16:15Z

keps/sig-network/0015-20180614-SCTP-support.md

+The Kubernetes API modification for Services is obvious.
+The selected port shall be reserved on the node, just like for TCP and UDP now. Unfortunately, golang does not have native SCTP support in the "net" package, so in order to access the kernel's SCTP API we have to introduce a new 3rd party package as a new vendor package.
+For Services with type=LoadBalancer we have to check how the cloud provider implementations handle new protocols, and we have to make sure that if SCTP is not supported then the request for a new load balancer, firewall rule, etc. with protocol=SCTP is rejected gracefully.
+


You also need to talk about HostPorts and demonstrate that we can make it work

thockin · 2018-06-18T21:16:50Z

keps/sig-network/0015-20180614-SCTP-support.md

+#### SCTP in NetworkPolicy
+The Kubernetes API modification for the NetworkPolicy is obvious.
+In order to utilize the new protocol value the network controller must support it. 
+


You need to talk about DNS and list the changes needed there (at least SRV logic)

@johnbelamaric @bowei

Yes, we would at least need to use _sctp for SRV. You should probably explicitly state the expectations around the multi-homing functionality. I assume that will not be supported? If it were supported, I think we would want something special around the service discovery to differentiate between IPs that are part of the same multi-homed endpoint vs. part of a separate sctp peer. My knowledge of sctp is pretty limited but reading RFC 4960 section 5.1.2 it looks like an individual name should resolve to a set of addresses that can be used for multi-homing. This is really different from what we do with SRV (an multiple A records) today, which resolve to separate endpoints.

To implement SCTP multi-homing in K8s, first of all NAT should support SCTP. There is an IETF draft about it: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-natsupp/

thockin · 2018-06-18T21:21:26Z

keps/sig-network/0015-20180614-SCTP-support.md

+#### Interworking with applications that use a user space SCTP stack
+A userspace SCTP stack implementation cannot work together with the SCTP kernel module (lksctp) on the same node. That is, the loading of the SCTP kernel module must be avoided on nodes where such applications that use userspace SCTP stack are planned to be run. The problem comes with the introduction of the SCTP protocol option for Services with Virtual IP: once such a service is created the relevant iptables/ipvs management logic kicks-in on every node, and as a consequence it loads the SCTP kernel module. There are some ideas how to solve this interworking problem:
+
+1. "-p sctp" is not used in the iptables rules, the processing of requests to the Virtual IP is executed purely based on the destination IP address. In case of ipvs the protocol is a mandatory parameter, so ipvs with SCTP rules cannot be used on the node where userspace SCTP applications should run.


Not working with IPVS seems like a pretty dire problem. With the "port ranges" proposal, one idea was to forward whole IPs, identity mapped. Does that help here?

There are the following idea alternatives, that may work with both iptables and ipvs:

Configure simple IP forwarding for the ClusterIP on host level: ClusterIP -> localhost. The userspace proxy would bind on localhost + the service specific port.

Every ClusterIP is configured to the loopback interface of the host. The userspace proxy binds on the ClusterIP+the service specific port.

thockin · 2018-06-18T21:22:17Z

keps/sig-network/0015-20180614-SCTP-support.md

+A userspace SCTP stack implementation cannot work together with the SCTP kernel module (lksctp) on the same node. That is, the loading of the SCTP kernel module must be avoided on nodes where such applications that use userspace SCTP stack are planned to be run. The problem comes with the introduction of the SCTP protocol option for Services with Virtual IP: once such a service is created the relevant iptables/ipvs management logic kicks-in on every node, and as a consequence it loads the SCTP kernel module. There are some ideas how to solve this interworking problem:
+
+1. "-p sctp" is not used in the iptables rules, the processing of requests to the Virtual IP is executed purely based on the destination IP address. In case of ipvs the protocol is a mandatory parameter, so ipvs with SCTP rules cannot be used on the node where userspace SCTP applications should run.
+2. Fall back to the user space proxy on those specific nodes. The user space proxy shall also use a user space SCTP stack, of course. Also the iptables rules that direct the client traffic to the userspace proxy must be created without the "-p sctp" option.


userspace is a dead-end. It's not likely to get extended.

"shall also use a user space SCTP stack, of course"? Ah... is it not possible to implement a generic SCTP proxy via the sockets API?

If the userspace proxy needs to use the userspace SCTP stack then that means the "tainting" runs in both directions: if you want to run the userspace proxy on a node for whatever reason, then you can't run containers that expect to use kernel SCTP on that node.

And of course if you don't use kernel SCTP support then that kind of kills my earlier claim that NetworkPolicy would be easy to implement because the kernel does all the hard stuff. Plugins with iptables-, OVS-, or eBPF-based NetworkPolicy presumably would not be able to do SCTP at all on nodes without kernel SCTP.

@thockin Indeed, a user space proxy does not sound good, but on a node where a userspace SCTP app runs we cannot use the SCTP kernel module anyway. So, if we want to have a proxy on those nodes, then we have to implement the SCTP stack for that proxy in the userspace.
@danwinship Exactly, you see it right. On a node where a userspace SCTP application runs we cannot run such applications that would require the SCTP kernel module. And it is true on the other way around, too.

Thinking about this part again: it will not work. In order to route a request to e.g. a userspace proxy the port on which the userspace proxy listens must be configured in iptables/ipvs. And it means that the protocol must be configured, that loads the kernel module.

~~Currently I think, there is no nice way to solve the interworking of userspace SCTP stacks and kernel module based services in the same k8s cluster. ~~

UPDATE: we have an idea, let's see what do you think. I update the document accordingly.

thockin · 2018-06-18T21:23:48Z

keps/sig-network/0015-20180614-SCTP-support.md

+In any case we shall be able to dedicate these nodes for those userspace SCTP applications, or at least, we must achieve that "regular" SCTP user applications are not deployed on these nodes. The solution proposal for this node separation:
+
+- there shall be a new kube-proxy parameter. If the parameter is set, the kube-proxy switches to this new mode of operation (described above) for SCTP services
+- if the new kube-proxy parameter is set the node must be tainted with a new taint, so the scheduler places only such SCTP applications on this node that use userspace SCTP stack. We must avoid the deployment of "regular" SCTP users on this node.


This is a really niche requirement (1 request in 4 years). I am not in favor of adding complexity like this. Can we make this the cluster-admin's problem? If you need SCTP, arrange taints or node labels..

Yes, but what we have to achieve is, that the proxy obeys those taints/labels, so the proxy does not execute any workflow which would load the SCTP kernel module on the node.
Another option is, that the whole SCTP support (as it is now in the code) is put behind a cluster level parameter - if SCTP support is allowed, the cluster uses the kernel modules everywhere. If not allowed, the cluster works as without this enhancement, and userspace SCTP stacks can work as they were before.
Or, back to the drawing table, and we really allow SCTP only for headless services ;) -> however, NetworkPolicy engines can still make things problematic for the userspace stacks.

m1093782566 · 2018-06-19T01:47:48Z

keps/sig-network/0015-20180614-SCTP-support.md

+<!---
+This is where we get down to the nitty gritty of what the proposal actually is.
+--->
+### User Stories [optional]


I am curious to learn the user stories of having SCTP support?

Do you mean, why SCTP support is requested at all?

danwinship · 2018-06-20T15:57:41Z

@janosi So do you (or anyone else here) actually want to do SCTP over the "normal" k8s network? (What the multi-network group is calling the "Kubernetes Cluster-Wide Default Network"). Or are you only interested in SCTP over physical NICs / SR-IOV / DPDK / etc? From what I've been able to gather, the telecom use cases for SCTP also have bandwidth/latency constraints such that you wouldn't want the traffic going over any sort of SDN. Is that accurate? (Or if that's not accurate in general, is it at least accurate for the use cases that require userspace SCTP?)

janosi · 2018-06-20T19:06:11Z

@danwinship Our original purpose was to have SCTP based endpoints/pods in the kube-dns. That was the reason why we initiated the relevant k8s issue.
From the protocol's usage perspective: we already have SCTP usage in our k8s clusters, in production. We use SCTP on top of "normal" k8s networks, though we have the capability to provide multiple networks per pod. We do not utilise the accelerator techniques you listed in every product. Some products may need such accelerators, some others not. Also it is up to the deployment environment, capacity requirements, maintaining the network stack, etc.
Though with the current capabilities of k8s we have to maintain our own service discovery methods for those endpoints/pods, and we would like to simplify this situation.
I am not sure, that high bandwidth and SDN would be mutually exclusive.
Please note, the "userspace SCTP" mentioned here, and the DPDK (and alike) based user space network stack are not the same. The "userspace SCTP" mentioned here is a raw socket or UDP-encapsulation based solution (usually, as I understand), but the underlying network stack is the regular kernel network stack. Or of course it could be anything (DPDK...), as the "userspace SCTP" stack just sits on top of any IP stack that provides the usual Linux/BSD API (and syscalls) to manage the underlying protocol layers. The "userspace SCTP" requirement came from @MaximProshin in the PR, i.e. it is not our "native" use case in the original issue, but because k8s is a community project, I wanted to have that requirement listed here, so it is not ignored or something (I do not want to polish myself, just explaining the situation, so everyone understands the source of the requirements).

MaximProshin · 2018-06-21T11:22:56Z

@janosi I appreciate that you consider my comment about user-land SCTP as a requirement!

danwinship · 2018-06-21T20:24:11Z

Please note, the "userspace SCTP" mentioned here, and the DPDK (and alike) based user space network stack are not the same.

Yeah, I know. My questions were trying to figure out if the userspace SCTP use case needs to involve "kubernetes networking", or if it just involves "kubernetes" and "networking" but not "kubernetes networking" (and DPDK was an example of something that isn't "kubernetes networking").

I think the userspace SCTP case creates a lot of problems, and if you don't personally care about it, you might want to just drop that stuff from this proposal, and only worry about (a) not implementing things in a way that would make it harder to add userspace SCTP support later, and (b) not implementing things in a way that would break people who were previously successfully using userspace SCTP behind kubernetes's back.

Eg, for the latter, we could make sure that kubernetes itself does not create any IPPROTO_SCTP sockets or -p sctp iptables rules unless it sees a Pod or Service with an SCTP port; and say that plugins should likewise try to avoid causing the kernel sctp module to be loaded if they don't actually need it.

bowei · 2018-06-28T20:54:14Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+#### Headless Service with SCTP
+As a user of Kubernetes I want to define headless Services for my applications that use SCTP as L4 protocol on their interfaces, so client applications can discover my applications in kube-dns, or via any other service discovery method that gets information about endpoints via the Kubernetes API.
+Example:


can you give the example records that would be added?

I added an example now. Please check and comment whether that fits for you.

bowei · 2018-06-28T20:57:18Z

keps/sig-network/0015-20180614-SCTP-support.md

+      port: 7777
+```
+#### User space SCTP stack
+As a user of Kubernetes I want to deploy and run my applications that use a user space SCTP stack. 


It feels like this could be separated out from the KEP which is mostly k8s API focused, unless there is some sort of interaction...

bowei · 2018-06-28T20:58:39Z

keps/sig-network/0015-20180614-SCTP-support.md

+The Kubernetes API modification for the NetworkPolicy is obvious.
+In order to utilize the new protocol value the network controller must support it.
+
+#### Interworking with applications that use a user space SCTP stack


This feels like a separate KEP ? to enable use of user-space protocol stacks + kube-proxy.

janosi · 2018-07-03T11:57:46Z

@bowei @MaximProshin @danwinship I added alternative proposals to solve the userspace SCTP stack interworking. Also I added the explanation why the userspace and kernel space SCTP stack cannot work together currently. Please check those.

KomorkinMikhail · 2018-07-03T14:49:53Z

@janosi Thanks for analysing the interworing issue and pointing out the above alternatives.
I suppose that the easiest way for introducing this kind of "userspace proxy" on dedicated nodes is to reuse the same iptables creation logic but without socket creation/listening. This way dedicated nodes will have needed ip rules for user space SCTP service but without LKSCTP.
It may be some kube-proxy startup option that will be switched on on dedicated nodes. And then checked during service creation:

if option is turned off: add iptables/ipvs rules, load up kernel sctp, listen to port (LKSCTP nodes)
if option is turned on: just add iptables/ipvs rules (user space nodes)

Another way is as I mentioned in kubernetes/kubernetes#64973 (comment) introduce an option for service itself alowing user to decide if creation of sctp socket is needed.

janosi · 2018-07-03T15:53:13Z

@KomorkinMikhail Thank yo for checking the document!
Unfortunately, the creation of the iptables/ipvs rules triggers the loading of lksctp, too. So, we must avoid the creation of those. I.e. it is not enough to avoid the creation of an SCTP socket.

idealhack · 2018-07-04T08:07:09Z

/ok-to-test

KomorkinMikhail · 2018-07-04T08:27:18Z

@janosi Actually no, at least for iptables I can guarantee that. We have included SCTP support locally, we did the same code changes as you but when it comes to openLocalPort() invocation we don't do this for SCTP type of services. Then proxy goes straight to rules adding, as a result LKSCTP is not loaded (it loads up exactly at the moment of socket creation/bind/listen procedure). Simply adding lines into iptables chains has nothing to do with loading up kernel modules.
We haven't checked the ipvs scenarios but I'm pretty sure that it should work as well.

Could you please recheck it on your side? Perhaps LKSCTP has been already loaded when you tried this scenario?

janosi · 2018-07-04T12:35:57Z

@KomorkinMikhail Hmmmm, yes, you are right. My shame :( I do not remember why I became this sure about the loading of the lksctp module when iptables rules are defined. :( I was wrong.
Thank you!
I have created some ipvs rules on my node, ipvs rules with "--sctp-service" do not load lksctp either.
So, now we should get an indication from @thockin or @bowei whether it is acceptable to skip the openLocalPort() step for SCTP Services, or, to limit the scope of this change, skip the openLocalPort() step on those nodes where userspace SCTP applications would be deployed. Because, as I understand, this is a safety mechanism to prevent that the same port is re-used by a process on the host. But anyhow, if we want to support userspace SCTP applications we must skip that openLocalPort() on their dedicated nodes at least.

KomorkinMikhail · 2018-07-04T13:55:17Z

@janosi Great!
I've also tried to figure out can we somehow ignore LKSCTP on nodes containing pods with usercpace SCTP, for example drop LKSCTP abort packets with additional iptables rules. But looks like there's no way we can identify whether it's an LKSCTP packet or ours anyway (even the sending IP is the same as the container IP).

janosi · 2018-07-04T17:00:30Z

@KomorkinMikhail Yes, in some cases an ABORT is necessary, and on iptables level we cannot really tell why the ABORT was initiated.
I wonder if the kernel module could check that the packet was also delivered to a raw socket and it would not send ABORT in that case, but it would just ignore ("silently discard") that.
However, I think, if we can use the same logic here which was used previously without k8s (dedication of nodes) then we are fine.

MaximProshin · 2018-07-04T19:22:54Z

@janosi and @KomorkinMikhail This is a good progress! Thanks!
I was always against of that LKSCTP listens all incoming SCTP packets even though some of them might not belong to it. I don't think that the kernel should not send a packet to LKSCTP if it was also sent to a raw socket but instead LKSCTP should "smartly" handle it. However, this is something to be discussed with LKSCTP developers.

thockin · 2018-07-06T20:13:35Z

I dislike it but if that is the only thing preventing it from working, I could probably forego it.

But we should only be opening a port if teh Service is type=NodePort or LoadBalancer, which we already said we won't do, right?

janosi · 2018-07-09T14:07:44Z

@KomorkinMikhail Indeed, and you can find my analysis in the PR. LB providers more or less covered the requirement, that they should filter protocols before the request go to the actual cloud's LB controller. Though, as I understood, @thockin asked for blocking the Service creation with type=LB at validation.

KomorkinMikhail · 2018-07-09T14:47:54Z

@janosi Yes, I get it, but why do we need to block it on k8s level if cloud providers can handle the check on their own? In case cloud provider can handle SCTP protocol k8s should accept the service

janosi · 2018-07-09T19:43:21Z

@KomorkinMikhail I think we have to turn to @thockin for the answer. This was his suggestion: "Do we know of any that work? We should probably default to a validation error if this combination is specified."

KomorkinMikhail · 2018-07-10T08:20:22Z

@janosi @thockin
I beleive there's no need in internal blocking. I've checked the code of supported loadbalancers and they do handle cases of unsupported protocols:

// The service controller verified all the protocols   match on the ports, just check and use the first one
--
if ports[0].Protocol != v1.ProtocolTCP && ports[0].Protocol != v1.ProtocolUDP {
return "", fmt.Errorf("Invalid protocol %s, only TCP and UDP are supported", string(ports[0].Protocol))

K8s can be used in many other Cloud providers in addition to supported ones. And we are also looking into posibility of some loadbalancer options in future.

janosi · 2018-07-10T08:36:01Z

@KomorkinMikhail @thockin I am fine with any of those options.
Though, for example there was no such protocol check for the GCE internal load balancer, so the rejection would came from the GCE API side in that case.
What I need is an agreement, so I can implement the selected way.
Also k8s is an open source SW. I believe, that the current SCTP implementation will be modified in the future. My understanding is, that the blocking based solution brings lower risk for the first introduction of SCTP support. Then it can be enhanced with time if needed.

KomorkinMikhail · 2018-07-10T08:55:55Z

@janosi Good, let's wait for response from @thockin

janosi · 2018-08-17T05:39:49Z

Can anyone please suggest how to move this KEP forward? Will there be a sig-network meeting next week?
@kubernetes/sig-network-pr-reviews @kubernetes/sig-network-proposals

k8s-ci-robot · 2018-08-17T05:39:59Z

@janosi: Reiterating the mentions to trigger a notification:
@kubernetes/sig-network-pr-reviews, @kubernetes/sig-network-proposals

In response to this:

Can anyone please suggest how to move this KEP forward? Will there be a sig-network meeting next week?
@kubernetes/sig-network-pr-reviews @kubernetes/sig-network-proposals

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

danwinship

@janosi yes, there is a sig-network meeting next week, Thursday the 23rd.

danwinship · 2018-08-17T13:46:06Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+It is not a goal to support SCTP as protocol value for the container's HostPort. The reason: [the usage of HostPort is not recommended by Kubernetes][], and to ensure proper interworking of HostPort with userspace SCTP stacks (see below) would require an additional kubelet/kubenet configuration option. In order to keep the complexity and impact of the introduction of SCTP on a lower level we do not plan to support SCTP as new protocol value for HostPort.
+
+[the usage of HostPort is not recommended by Kubernetes]:https://kubernetes.io/docs/concepts/configuration/overview/#services


That doesn't say that use of HostPorts is not recommended. It just points out that you shouldn't gratuitously add HostPorts to pods that don't actually need them, because host ports are a limited resource.

(So I think we should support SCTP HostPorts, unless there was some other argument against it.)

I don't see why not, I guess.

OK. The reason I proposed this way was, that listening on HostPorts (in order to reserve the port) is not handled by the kube-proxy but by the kubelet (kubelet -> Docker Service -> Kubenet plugin -> hostportmanager). That is, in order to have the same "userspace sctp config" based listening on HostPorts it would require another config parameter for another component - i.e. the user shall configure the userspace sctp support in 2 different places. And because I understood the cited k8s documentation so, that the usage of HostPorts is not recommended, I thought that then I can skip the support of SCTP for HostPorts. But of course, if the decision is to have SCTP support for HostPorts, too, then I do it.

In truth, I am not sure how valuable kubelet's logic is here. ISTR there was/is a bug in it that it doesn't re-open ports if Kubelet restarts, and nobody has complained about it.

Maybe we should just remove that logic altogether? We could start by skipping it for SCTP and have a separate discussion about removal... Just make sure it is well commented why we do it for everything except SCTP.

Fine for me.

danwinship · 2018-08-17T13:47:42Z

keps/sig-network/0015-20180614-SCTP-support.md

+      role: myservice
+  policyTypes:
+  - Ingress
+  - Egress


(FWIW, this NetworkPolicy object blocks all egress. You probably want to remove the line "- Egress" here.)

danwinship · 2018-08-17T14:07:14Z

keps/sig-network/0015-20180614-SCTP-support.md

+    - protocol: SCTP
+      port: 7777
+```
+#### Userspace SCTP stack


It's probably worthwhile to explain why this really is a valid use case. (Because it's not possible to write a server that proxies/filters arbitrary SCTP streams using the sockets APIs and kernel SCTP.)

All right. For me the justification came from the fact that there are apps that use userspace SCTP for any reason right now, and most probably they do not want to re-write their network handling logic at the same time when they move to containers+k8s - so they must have userspace SCTP support.

danwinship · 2018-08-17T14:14:05Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+SCTP is a widely used protocol in telecommunications. It would ease the management and execution of telecommunication applications on Kubernetes if SCTP were added as a protocol option to Kubernetes. 
+
+### Goals


Accepting SCTP connections from clients outside the cluster should be listed as either a Goal or a Non-Goal. (And if the former, should be demonstrated by a User Story.)

Also, allowing pods to make outgoing SCTP connections to servers outside the cluster should be either a Goal or a Non-Goal. (In particular, if it's a Goal, then plugins need to set up SCTP NAT or whatever, which would conflict with the "Documentation only" approach to allowing userspace SCTP.)

danwinship · 2018-08-17T14:19:23Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+### Non-Goals
+
+It is not a goal here to add SCTP support to load balancers that are provided by cloud providers. 


I think it's fine to not require cloud providers to add SCTP support, but I don't see any reason to forbid SCTP load-balancing, especially given that some providers apparently do already support it.

I guess I can hold my nose on this. I really dislike things that are quietly non-portable, like this, but it's hardly the first or worst.

Arguably my position on SCTP LoadBalancers here contradicts my position on NetworkPolicy ipBlock ingress on the list... (Although I feel like ipBlock ingress is more broken than SCTP LoadBalancing is so it's not too inconsistent.) Having stronger guidelines for situations like this would be good... I think sig-network outsources more behavior to plugins and third parties than most other SIGs do so this is probably something that affects us more than everyone else.

danwinship · 2018-08-17T14:20:27Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+In case of Servies with ClusterIP or NodePort or externalIP the selected port shall be reserved on the respective nodes, just like for TCP and UDP currently.  Unfortunately, golang does not have native SCTP support in the "net" package, so in order to reserving those ports via the kernel's SCTP API we have to introduce a new 3rd party package as a new vendor package. We plan to use the go sctp library from github.com/ishidawataru/sctp.
+
+For Services with type=LoadBalancer we reject the Service creation request for SCTP services at API validation time.


As before, this seems wrong.

danwinship · 2018-08-17T14:20:54Z

keps/sig-network/0015-20180614-SCTP-support.md

+#### SCTP in NetworkPolicy
+The Kubernetes API modification for the NetworkPolicy is obvious.
+
+In order to utilize the new protocol value the network controller must support it.


network plugin

thockin · 2018-08-17T23:09:49Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+As we can easily see, it is pretty easy to separate application pods that use a userspace SCTP stack from those application pods that use the kernel space SCTP stack: the usual nodeselector label based mechanism,  or taints are there for this very purpose. 
+
+The real challenge here is to ensure that when an SCTP Service is created in a Kubernetes cluster the Kubernetes logic does not create listening SCTP sockets on those nodes that are dedicated for the applications that use  userspace SCTP stack - because such an action would trigger the loading of the kernel module.


It's not clear to me.

Is this proposal saying that Kubelet and kube-proxy will not do the usual "open and hold" operation for SCTP ports?

Is that sufficient? If I have a cluster with an SCTP service, that triggers loading iptables modules for sctp, but does not listen() on the port - will userspace SCTP be OK?

Is this proposal saying that Kubelet and kube-proxy will not do the usual "open and hold" operation for SCTP ports?

Yeah, on nodes that were marked userspace-sctp. Which would mean that HostPort/ExternalIP SCTP services wouldn't work on userspace-sctp nodes, and NodePort SCTP services would not be reachable on userspace-sctp nodes.

But that weirdness would be limited to clusters that used userspace sctp; in clusters with no userspace sctp nodes, SCTP would behave just like everything else.

If I have a cluster with an SCTP service, that triggers loading iptables modules for sctp, but does not listen() on the port - will userspace SCTP be OK?

It looks like the kernel has some code to parse SCTP packets even without the sctp module being loaded, and you can add rules with matches like -p sctp --dport 80 without needing the kernel sctp module. (It will cause xt_sctp to be loaded, but that's just part of iptables, and wouldn't interfere with userspace SCTP processing.)

It is possible that some iptables extension rules (NAT?) might cause sctp to be loaded.

I'm not very excited by formalizing userspace SCTP as a concept in Kubernetes based on labels. In general, we don't want to use annotations or labels for config, when we have real config (even if kube-proxy is behind the curve wrt component config).

We could add a flag/component-config param here, and leave labels to users. I would be OK with that.

Wrt SCTP NAT: according to our observations, iptables based NAT does not cause any harm. As @KomorkinMikhail wrote above in this PR, when they implemented SCTP related patches into k8s they simply skipped those "openlocalport" parts for SCTP, and they could use it with NodePort and ExternalIP as well.

Wrt the separation of nodes: my intention with that description was not to tell, that we can/should solve it with labels. I just wanted to say, that on app level it is easy to direct the app pods to the right nodes with the usual label based mechanism. The current implementation in the code PR introduces a new parameter in kube-proxy config ("SCTPUserSpaceNode") for the purpose of skipping the listening on the SCTP ports.

OK. This wasn't clear. The main points, if I understand:

skip the opening/holding of the port in both kubelet and kube-proxy IFF SCTP

suggest that user segregate their user-space SCTP work and their kernel-space SCTP work by node labels, but the details of that are left to the user

Yepp, my English.... :/
Exactly, that was the plan.

…, we cannot use the port value in iptables rules if the protocol is not defined

…worked based on further investigations.

…the SCTP kernel module. The proposed solution is updated accordingly, and it became a lot simpler. Also the usage of SCTP as protocol value in the Pod/container descriptor is described in the document now.

…rt with SCTP is clarified (not supported).

… HostPort with SCTP shall be supported; type=LoadBalancer with SCTP shall be supported

janosi · 2018-08-24T08:40:39Z

@thockin @danwinship I updated the document according to the comments above and the discussion in the SIG Network meeting yesterday. Please check the document, and if it is OK, please move it forward into a next state.
If the KEP doc is according to the agreements I modify the code accordingly in the PR.

thockin

Thanks A few notes for progress into implementation.

thockin · 2018-08-24T17:14:05Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+### Non-Goals
+
+It is not a goal here to add SCTP support to load balancers that are provided by cloud providers. The Kubernetes side implementation will not restrict the usage of SCTP as the protocol for the Services with type=LoadBalancer, but we do not implement the support of SCTP into the cloud specific load balancer implementations.


@bowei We should queue this up for discussion in GCP

@andrewsykim @jagosan Can we put this on an agenda for sig-cloudprovider to make sure people know it is coming?

Will do! Thanks!

thockin · 2018-08-24T17:15:50Z

keps/sig-network/0015-20180614-SCTP-support.md

+  - 80.11.12.10
+```
+
+#### NetworkPolicy with SCTP


As part of the implementation of this, I'll ask you to reach out to NetworkPolicy implementors and at least let them know it is coming. I might not hold alpha for it, but I will hold beta.

thockin · 2018-08-24T17:16:56Z

keps/sig-network/0015-20180614-SCTP-support.md

+
+#### SCTP in Services
+##### Kubernetes API modification
+The Kubernetes API modification for Services to support SCTP is obvious.


NB we will have to filter/reject it during alpha

thockin · 2018-08-24T17:18:27Z

keps/sig-network/0015-20180614-SCTP-support.md

+For Services with type=LoadBalancer we expect that the cloud provider's load balancer API client in Kubernetes rejects the requests with unsupported protocol.
+
+#### SCTP support in Kube DNS 
+Kube DNS shall support SRV records with "_sctp" as "proto" value. According to our investigations, the DNS controller is very flexible from this perspective, and it can create SRV records with any protocol name. I.e. there is no need for additional implementation to achieve this goal.


Please cross-check CoreDNS support and track it as part of alpha.

@johnbelamaric

thockin · 2018-08-24T17:20:01Z

/lgtm
/approve

OK to move to implementable.

k8s-ci-robot · 2018-08-24T17:20:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-network/OWNERS~~ [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

m1093782566 · 2018-08-24T17:48:13Z

I am interested in joining this effort and can take the service/kube-proxy implementation for SCTP if no one is working on this ...

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 16, 2018

k8s-ci-robot requested a review from dcbw June 16, 2018 18:29

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 16, 2018

k8s-ci-robot requested a review from thockin June 16, 2018 18:29

k8s-ci-robot added sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/network Categorizes an issue or PR as relevant to SIG Network. labels Jun 16, 2018

janosi mentioned this pull request Jun 16, 2018

SCTP support implementation for Kubernetes kubernetes/kubernetes#64973

Merged

k8s-ci-robot assigned thockin Jun 16, 2018

thockin reviewed Jun 18, 2018

View reviewed changes

m1093782566 reviewed Jun 19, 2018

View reviewed changes

bowei reviewed Jun 28, 2018

View reviewed changes

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 4, 2018

jdumars mentioned this pull request Jul 12, 2018

KEP #15(dup): SCTP support kubernetes-retired/architecture-tracking#15

Closed

k8s-ci-robot added the kind/design Categorizes issue or PR as related to design. label Aug 17, 2018

danwinship reviewed Aug 17, 2018

View reviewed changes

thockin reviewed Aug 17, 2018

View reviewed changes

janosi and others added 8 commits August 24, 2018 10:37

New KEP document at keps/sig-network/0015-20180614-SCTP-support.md

20c3b98

protocol is mandatory in order to use port in iptables rules. That is…

106b8e7

…, we cannot use the port value in iptables rules if the protocol is not defined

Modified based on the first comments. Userspace SCTP related part re-…

f368332

…worked based on further investigations.

Added alternatives to handle the userspace SCTP incompatibility

fdf2fc2

Example DNS SRV record added. Userspace SCTP alternatives updated.

d0ad13a

It turned out that iptables and ipvs does not trigger the loading of …

3bb6404

…the SCTP kernel module. The proposed solution is updated accordingly, and it became a lot simpler. Also the usage of SCTP as protocol value in the Pod/container descriptor is described in the document now.

Handling of Services with type=LoadBalaner changed. Support of HostPo…

683f9a0

…rt with SCTP is clarified (not supported).

Updated according to the comments on github. Main functional changes:…

c234ca8

… HostPort with SCTP shall be supported; type=LoadBalancer with SCTP shall be supported

janosi force-pushed the k8s-sctp-kep-doc branch from 303d38b to c234ca8 Compare August 24, 2018 08:38

thockin reviewed Aug 24, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 24, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 24, 2018

k8s-ci-robot merged commit 5bda154 into kubernetes:master Aug 24, 2018

janosi mentioned this pull request Sep 11, 2018

SCTP support for Services, Pod, Endpoint, and NetworkPolicy kubernetes/enhancements#614

Closed


		### Non-Goals

		It is not a goal here to add SCTP support to load balancers that are provided by cloud providers. I.e. the Kubernetes user can define Services with type=LoadBalancer and Protocol=SCTP, but if the actual load balancer implementation does not support SCTP then the creation of the Service/load balancer fails.


		It is not a goal to support SCTP as protocol value for the container's HostPort. The reason: [the usage of HostPort is not recommended by Kubernetes][], and to ensure proper interworking of HostPort with userspace SCTP stacks (see below) would require an additional kubelet/kubenet configuration option. In order to keep the complexity and impact of the introduction of SCTP on a lower level we do not plan to support SCTP as new protocol value for HostPort.

		[the usage of HostPort is not recommended by Kubernetes]:https://kubernetes.io/docs/concepts/configuration/overview/#services


		SCTP is a widely used protocol in telecommunications. It would ease the management and execution of telecommunication applications on Kubernetes if SCTP were added as a protocol option to Kubernetes.

		### Goals


		In case of Servies with ClusterIP or NodePort or externalIP the selected port shall be reserved on the respective nodes, just like for TCP and UDP currently. Unfortunately, golang does not have native SCTP support in the "net" package, so in order to reserving those ports via the kernel's SCTP API we have to introduce a new 3rd party package as a new vendor package. We plan to use the go sctp library from github.com/ishidawataru/sctp.

		For Services with type=LoadBalancer we reject the Service creation request for SCTP services at API validation time.


		As we can easily see, it is pretty easy to separate application pods that use a userspace SCTP stack from those application pods that use the kernel space SCTP stack: the usual nodeselector label based mechanism, or taints are there for this very purpose.

		The real challenge here is to ensure that when an SCTP Service is created in a Kubernetes cluster the Kubernetes logic does not create listening SCTP sockets on those nodes that are dedicated for the applications that use userspace SCTP stack - because such an action would trigger the loading of the kernel module.

New KEP for SCTP support feature #2276

New KEP for SCTP support feature #2276

Conversation

janosi commented Jun 16, 2018 • edited Loading

janosi commented Jun 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janosi Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnbelamaric Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danwinship Jun 18, 2018 • edited Loading

Choose a reason for hiding this comment

danwinship Jun 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janosi Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janosi Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danwinship commented Jun 20, 2018

janosi commented Jun 20, 2018

MaximProshin commented Jun 21, 2018

danwinship commented Jun 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janosi commented Jul 3, 2018 • edited Loading

KomorkinMikhail commented Jul 3, 2018

janosi commented Jul 3, 2018

idealhack commented Jul 4, 2018

KomorkinMikhail commented Jul 4, 2018 • edited Loading

janosi commented Jul 4, 2018

KomorkinMikhail commented Jul 4, 2018

janosi commented Jul 4, 2018

MaximProshin commented Jul 4, 2018

thockin commented Jul 6, 2018

janosi commented Jul 9, 2018

KomorkinMikhail commented Jul 9, 2018

janosi commented Jul 9, 2018

KomorkinMikhail commented Jul 10, 2018 • edited Loading

janosi commented Jul 10, 2018

KomorkinMikhail commented Jul 10, 2018

janosi commented Aug 17, 2018

k8s-ci-robot commented Aug 17, 2018

danwinship left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janosi commented Jun 16, 2018 •

edited

Loading

janosi Jun 19, 2018 •

edited

Loading

johnbelamaric Jun 19, 2018 •

edited

Loading

danwinship Jun 18, 2018 •

edited

Loading

danwinship Jun 18, 2018 •

edited

Loading

janosi Jun 19, 2018 •

edited

Loading

janosi Jun 19, 2018 •

edited

Loading

janosi commented Jul 3, 2018 •

edited

Loading

KomorkinMikhail commented Jul 4, 2018 •

edited

Loading

KomorkinMikhail commented Jul 10, 2018 •

edited

Loading

janosi Aug 22, 2018 •

edited

Loading

janosi commented Aug 24, 2018 •

edited

Loading

m1093782566 commented Aug 24, 2018 •

edited

Loading