Add IPv4/IPv6 dual stack KEP #648

aojea · 2018-12-05T12:02:07Z

Moving kubernetes/community#2254 from k/community to k/enhancements
cc: @leblancd

leblancd · 2018-12-05T14:29:03Z

@aojea - Thanks for moving this. I have some edits in the works based on comments on the previous PR, mostly rearranging some sections. I should be able to get these posted late next week (currently busy with some other issues, as you know :) ).

uablrek · 2018-12-05T15:41:07Z

This remark;

Since IPVS functionality does not yet include IPv6 support...

is not valid. Ipvs has full support for ipv6. The sub-sequent reference to a kube-router issue is just saying that kube-router does not (yet) have ipv6 support.

leblancd · 2018-12-05T18:48:42Z

@uablrek - Thanks for the clarification. That was something that was mentioned earlier, and will be reworded.

m1093782566 · 2018-12-07T03:51:03Z

Yes, IPVS proxier has full support for ipv6.

thockin · 2018-12-17T21:17:33Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+
+One alternative to adding transition mechanisms would be to modify Kubernetes to provide support for IPv4 and IPv6 communications in parallel, for both pods and services, throughout the cluster (a.k.a. "full" dual stack).
+
+A second, simpler alternative, which is a variation to the "full" dual stack model, would be to provide dual stack addresses for pods and nodes, but restrict service IPs to be single-family (i.e. allocated from a single service CIDR). In this case, service IPs in a cluster would be either all IPv4 or all IPv6, as they are now. Compared to a full dual-stack approach, this "dual-stack pods / single-family services" approach saves on implementation complexity, but would introduce some minor feature restrictions. (For more details on these tradeoffs, please refer to the "Variation: Dual-Stack Service CIDRs" section under "Alternatives" below).


In re-reading, the complexity saved is pretty minimal. We have to run dual-stack kube-proxy, so I think it's safe to say that the "full" path is almost certain to come to be, though it is 100% OK to add that as a second-step (on the assumption it is easier).

As an alternative, is it worthwhile to discuss an intermediate delivery where all kube-proxy is single-stack (including NodePorts) ?

In other words: step1 is just pod IPs, hostPorts, and headless services (DNS). step2 adds nodeports, external IPs, ingress, and endpoints. step3 add service IPs and single-family deployments.

Viable?

@thockin I'm all for chunking this task as much as we can. Your list of steps seems reasonable to me.

Added implementation plan as suggested in #808

thockin · 2018-12-17T21:29:38Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+  - Service addresses: 1 service IP address per service
+- Kube-DNS is expected to be End-of-Life soon, so dual-stack testing will be performed using coreDNS.
+- External load balancers that rely on Kubernetes services for load balancing functionality will only work with the IP family that matches the IP family of the cluster's service CIDR.
+- Dual-stack support for Kubernetes orchestration tools other than kubeadm (e.g. miniKube, KubeSpray, etc.) are considered outside of the scope of this proposal.


We'll need to communicate HOW to enable this.

Added note to communicate usage through documentation - #808

thockin · 2018-12-17T21:34:43Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+  - NodePort: Support listening on both IPv4 and IPv6 addresses
+  - ExternalIPs: Can be IPv4 or IPv6
+- Kube-proxy IPVS mode will support dual-stack functionality similar to kube-proxy iptables mode as described above. IPVS kube-router support for dual stack, on the other hand, is considered outside of the scope of this proposal.
+- For health/liveness/readiness probe support, a kubelet configuration will be added to allow a cluster administrator to select a preferred IP family to use for implementing probes on dual-stack pods.


Possible option: check on both? Does this offer any avenue for single-family apps? Maybe not (readiness is per-pod). How would we want health-checks to work in a multi-net model?

Addressed in #808

thockin · 2018-12-17T21:35:47Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+  - ExternalIPs: Can be IPv4 or IPv6
+- Kube-proxy IPVS mode will support dual-stack functionality similar to kube-proxy iptables mode as described above. IPVS kube-router support for dual stack, on the other hand, is considered outside of the scope of this proposal.
+- For health/liveness/readiness probe support, a kubelet configuration will be added to allow a cluster administrator to select a preferred IP family to use for implementing probes on dual-stack pods.
+- The pod status API changes will include a per-IP string map for arbitrary annotations, as a placeholder for future Kubernetes enhancements. This mapping is not required for this dual-stack design, but will allow future annotations, e.g. allowing a CNI network plugin to indicate to which network a given IP address applies.


Are we going to give CNI/CRI hooks to fill these in?

addressed as part of #808

thockin · 2018-12-17T21:43:43Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+```
+
+##### Default Pod IP Selection
+Older servers and clients that were built before the introduction of full dual stack will only be aware of and make use of the original, singular PodIP field above. It is therefore considered to be the default IP address for the pod. When the PodIP and PodIPs fields are populated, the PodIPs[0] field must match the (default) PodIP entry. If a pod has both IPv4 and IPv6 addresses allocated, then the IP address chosen as the default IP address will match the IP family of the cluster's configured service CIDR. For example, if the service CIDR is IPv4, then the IPv4 address will be used as the default address.


How will we want this to work if/when we support dual-stack service IPs?

thockin · 2018-12-17T22:04:14Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+```
+  --cluster-cidr  ipNetSlice   (IP CIDRs, in a comma separated list, Default: [])
+```
+Only the first CIDR for each IP family will be used; all others will be ignored.


logged and ignored

updated in #808

thockin · 2018-12-17T22:05:29Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+
+### IPVS Support and Operation
+
+Since IPVS functionality does not yet include IPv6 support (see [cloudnativelabs/kube-router Issue #307](https://github.com/cloudnativelabs/kube-router/issues/307)), support for IPVS functionality in a dual-stack cluster is considered a "nice-to-have" or stretch goal.


Isn't this out of date? IPVS does support v6 though kube-proxy might not. I don't think IPVS is optional - it is a GA feature.

Confirmed out of date and removed in #808

thockin · 2018-12-17T22:06:33Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+
+Currently, health, liveness, and readiness probes are defined without any concern for IP addresses or families. For the first release of dual-stack support, a cluster administrator will be able to select the preferred IP family to use for probes when a pod has both IPv4 and IPv6 addresses. For this selection, a new "--preferred-probe-ip-family" argument for the for the [kubelet startup configuration](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) will be added:
+```
+  --preferred-probe-ip-family  string   ["ipv4", "ipv6", or "none". Default: "none", meaning use the pod's default IP]


@mtaufen should we be adding flags or JUST component config fields?

resolved in #808. It was removed as we updated the behavior here.

For health/liveness/readiness probe support, the default behavior will not change and an additional optional field would be added to the pod specification and is respected by kubelet. This will allow application developers to select a preferred IP family to use for implementing probes on dual-stack pods.

thockin · 2018-12-17T22:08:16Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+
+- Because service IPs will remain single-family, pods will continue to access the CoreDNS server via a single service IP. In other words, the nameserver entries in a pod's /etc/resolv.conf will typically be a single IPv4 or single IPv6 address, depending upon the IP family of the cluster's service CIDR.
+- Non-headless Kubernetes services: CoreDNS will resolve these services to either an IPv4 entry (A record) or an IPv6 entry (AAAA record), depending upon the IP family of the cluster's service CIDR.
+- Headless Kubernetes services: CoreDNS will resolve these services to either an IPv4 entry (A record), an IPv6 entry (AAAA record), or both, depending on the service's endpointFamily configuration (see [Configuration of Endpoint IP Family in Service Definitions](#configuration-of-endpoint-ip-family-in-service-definitions)).


5 lines up it said "It is not expected that any changes will be needed for CoreDNS" but this is clearly a change. It has to be taught about the new plural field on endpoints

Resolved as part of #808

thockin · 2018-12-17T22:10:37Z

keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md

+        fieldPath: status.podIPs
+```
+
+This definition will cause an environmental variable setting in the pod similar to the following:


What about extra fields of podIPs? The parameters block, for example? Do we want to do anything with that? I am not sure.

Addressed in #808

thockin · 2018-12-17T22:11:58Z

I want to say what a good KEP this is. Thanks.

justaugustus · 2019-01-09T19:05:55Z

Post holiday bump.

justaugustus

Please remove any references to NEXT_KEP_NUMBER and rename the KEP to just be the draft date and KEP title.
KEP numbers will be obsolete once #703 merges.

k8s-ci-robot · 2019-01-22T08:03:17Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aojea
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: thockin

If they are not already assigned, you can assign the PR to them by writing /assign @thockin in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

keps/sig-network/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

KEP numbers will be obsolete once kubernetes#703 merges.

thockin · 2019-01-26T00:13:11Z

@aojea @leblancd

I am reluctant to merge this until we address possible staging of delivery (and who will be working on it).

lachie83 · 2019-01-28T17:41:44Z

cc @khenidak

khenidak · 2019-01-28T22:40:37Z

@aojea Thank you for this. Are you planning to carry this forward with implementation as well? or just the KEP?

thockin · 2019-01-28T23:21:44Z

The KEP is over a month with no word. @khenidak I think if you want to start thinking about impl, that's fair. You could even open a PR that builds on this one (keeping original work intact) if you want to push it ahead.

feiskyer · 2019-01-29T06:47:24Z

I'd like to move this forward if there's still no responses from @aojea or @leblancd.

uablrek · 2019-01-29T07:38:36Z

Hi. Has implementation started? I would like to contribute, foremost in the "ipvs" area since I think the "nice to have" is not the right way, but also in other areas. Where do I sign up?

aojea · 2019-01-29T08:36:06Z

@feiskyer @khenidak We should ask @leblancd , I was just trying to help to keep this alive but I can't commit to carrying on any activity

khenidak · 2019-01-29T17:08:17Z

keps/sig-network/20180612-ipv4-ipv6-dual-stack.md

+  - NodePort: Support listening on both IPv4 and IPv6 addresses
+  - ExternalIPs: Can be IPv4 or IPv6
+- Kube-proxy IPVS mode will support dual-stack functionality similar to kube-proxy iptables mode as described above. IPVS kube-router support for dual stack, on the other hand, is considered outside of the scope of this proposal.
+- For health/liveness/readiness probe support, a kubelet configuration will be added to allow a cluster administrator to select a preferred IP family to use for implementing probes on dual-stack pods.


This should be an auto-detect. Although the cluster will be running in dual stack mode. Some pods will be only using one address family

khenidak · 2019-01-29T17:11:00Z

keps/sig-network/20180612-ipv4-ipv6-dual-stack.md

+    //    Properties: Arbitrary metadata associated with the allocated IP.
+    type PodIPInfo struct {
+        IP string
+        Properties map[string]string


what would be an example of properties?

Is worth to check previous discussion https://github.com/kubernetes/community/pull/2254/files#r227458664

khenidak · 2019-01-29T17:14:45Z

keps/sig-network/20180612-ipv4-ipv6-dual-stack.md

+```
+  --pod-cidr  ipNetSlice   [IP CIDRs, comma separated list of CIDRs, Default: []]
+```
+Only the first address of each IP family will be used; all others will be ignored.


This is going to be difficult for user to understand. I would think an additional flag --pod-cidr-alternative (or so). Forcing the user to think about mail ip address family and alternative ip address family.

previously discussed here https://github.com/kubernetes/community/pull/2254/files#r227469823

khenidak · 2019-01-29T17:15:04Z

keps/sig-network/20180612-ipv4-ipv6-dual-stack.md

+```
+  --cluster-cidr  ipNetSlice   [IP CIDRs, comma separated list of CIDRs, Default: []]
+```
+Only the first address of each IP family will be used; all others will be ignored.


same as pod-cidr

https://github.com/kubernetes/community/pull/2254/files#r227469961

khenidak · 2019-01-29T17:16:34Z

keps/sig-network/20180612-ipv4-ipv6-dual-stack.md

+    // "PodIP" field, and this default IP address must be recorded in the
+    // 0th entry (PodIPs[0]) of the slice. The list is empty if no IPs have
+    // been allocated yet.
+    PodIPs []PodIPInfo `json:"podIPs,omitempty" protobuf:"bytes,6,opt,name=podIPs"`


What would would a situation where a pod has more than two ips? the argument i have here is to have one ip (current) and one additional.

IPv4+IPv6 now. I think it's short sighted to think that the first addition we want to make is also going to be the last.

I can think of things we might want to do in the future like: assigning external addresses directly to pods, IPv6 privacy addresses (which also rotate periodically), addresses from multiple IPv4/IPv6 prefixes during renumbering transitions, or as some future pod-to-pod federation optimisation.
We certainly don't have to design for any of these possibilities now, but once we have an array of addresses, I bet $5 we'll find a reason to use a third address.

IPv6 can have several addresses on the same interface depending on the scope IPv6 Scoped Address Architecture

khenidak · 2019-01-29T18:00:42Z

keps/sig-network/20180612-ipv4-ipv6-dual-stack.md

+#### Type Load Balancer
+
+The cloud provider will provision an external load balancer. If the cloud provider load balancer maps directly to the pod iP's then a dual stack load balancer could be used. Additional information may need to be provided to the cloud provider to configure dual stack.
+


we will have to gate this KEP on input from the cloud providers. Currently aws, azure and gcp use node ports for routing LB to node (even when CNI that hands out IPs from cluster network are used).

Addressed as part of #808

Arvinderpal · 2019-01-29T18:27:04Z

I would also love to contribute to this effort. I've been working on adding IPv6 functionality and e2e testing to the kube-router project and have developed a vagrant based ipv6-only test environment.

lachie83 · 2019-01-30T22:11:28Z

keps/sig-network/20180612-ipv4-ipv6-dual-stack.md

+
+- Pod Connectivity: IPv4-to-IPv4 and IPv6-to-IPv6 access between pods
+- Access to External Servers: IPv4-to-IPv4 and IPv6-to-IPv6 access from pods to external servers
+- NGINX Ingress Controller Access: Access from IPv4 and/or IPv6 external clients to Kubernetes services via the Kubernetes NGINX Ingress Controller.


Could you shed some light on why the nginx ingress controller is part of the goals of this KEP? Does that mean there is a commitment to implement the necessary changes (if any) to the nginx ingress controller? Why would we not leave that up to individual ingress controllers to implement?

Explained in L456

khenidak · 2019-01-31T17:33:02Z

I think we are also missing node cidr modification. I think we should also follow the model where the node has one cidr as is then list of cidrs as a secondary property. That way CNIs that depend on node.cidr can allocate multi family ips as needed

CC @lachie83

lachie83 · 2019-02-21T22:34:24Z

I think we are also missing node cidr modification. I think we should also follow the model where the node has one cidr as is then list of cidrs as a secondary property. That way CNIs that depend on node.cidr can allocate multi family ips as needed

CC @lachie83

Added as part of #808

Awareness of Multiple NodeCIDRs per Node

As with PodIP, corresponding changes will need to be made to NodeCIDR. These changes are essentially the same as the aformentioned PodIP changes which create the pularalization of NodeCIDRs to a slice rather than a singular and making those changes across the internal representation and v1 with associated conversations.

k8s-ci-robot · 2019-02-21T22:34:26Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

aojea · 2019-02-21T23:21:05Z

obsoleted by #808

Add IPv4/IPv6 dual stack KEP

1a03c72

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 5, 2018

k8s-ci-robot requested review from idvoretskyi and justaugustus December 5, 2018 12:02

aojea mentioned this pull request Dec 5, 2018

Add IPv4/IPv6 dual stack KEP kubernetes/community#2254

Closed

uablrek mentioned this pull request Dec 5, 2018

Trace works about enable the ipvs mode for the kube-proxy by default kubernetes/kubernetes#71202

Closed

6 tasks

pmichali mentioned this pull request Dec 7, 2018

DualStack: Add IPv4/IPv6 Dual-Stack support and awareness kubernetes/kubernetes#62822

Closed

thockin self-assigned this Dec 17, 2018

thockin reviewed Dec 17, 2018

View reviewed changes

briantopping mentioned this pull request Dec 28, 2018

Support multiple pod IP addresses kubernetes/kubernetes#27398

Closed

justaugustus requested changes Jan 20, 2019

View reviewed changes

Rename and remove KEP number references

407dc76

KEP numbers will be obsolete once kubernetes#703 merges.

aojea force-pushed the dual-stack branch from 3eb7bb5 to 407dc76 Compare January 22, 2019 08:05

khenidak requested changes Jan 29, 2019

View reviewed changes

lachie83 reviewed Jan 30, 2019

View reviewed changes

lachie83 mentioned this pull request Feb 4, 2019

Add revised IPv4/IPv6 dual stack KEP #808

Merged

Arvinderpal mentioned this pull request Feb 7, 2019

IPv6 Support kubernetes-sigs/kind#280

Closed

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 21, 2019

aojea closed this Feb 21, 2019

justaugustus removed the sig/pm label Apr 19, 2020


		One alternative to adding transition mechanisms would be to modify Kubernetes to provide support for IPv4 and IPv6 communications in parallel, for both pods and services, throughout the cluster (a.k.a. "full" dual stack).

		A second, simpler alternative, which is a variation to the "full" dual stack model, would be to provide dual stack addresses for pods and nodes, but restrict service IPs to be single-family (i.e. allocated from a single service CIDR). In this case, service IPs in a cluster would be either all IPv4 or all IPv6, as they are now. Compared to a full dual-stack approach, this "dual-stack pods / single-family services" approach saves on implementation complexity, but would introduce some minor feature restrictions. (For more details on these tradeoffs, please refer to the "Variation: Dual-Stack Service CIDRs" section under "Alternatives" below).


		### IPVS Support and Operation

		Since IPVS functionality does not yet include IPv6 support (see [cloudnativelabs/kube-router Issue #307](https://github.com/cloudnativelabs/kube-router/issues/307)), support for IPVS functionality in a dual-stack cluster is considered a "nice-to-have" or stretch goal.

		#### Type Load Balancer

		The cloud provider will provision an external load balancer. If the cloud provider load balancer maps directly to the pod iP's then a dual stack load balancer could be used. Additional information may need to be provided to the cloud provider to configure dual stack.

Add IPv4/IPv6 dual stack KEP #648

Add IPv4/IPv6 dual stack KEP #648

Conversation

aojea commented Dec 5, 2018 • edited Loading

leblancd commented Dec 5, 2018

uablrek commented Dec 5, 2018 • edited Loading

leblancd commented Dec 5, 2018

m1093782566 commented Dec 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Dec 17, 2018

justaugustus commented Jan 9, 2019

justaugustus left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 22, 2019

thockin commented Jan 26, 2019

lachie83 commented Jan 28, 2019

khenidak commented Jan 28, 2019

thockin commented Jan 28, 2019

feiskyer commented Jan 29, 2019

uablrek commented Jan 29, 2019

aojea commented Jan 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea Feb 21, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arvinderpal commented Jan 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khenidak commented Jan 31, 2019

lachie83 commented Feb 21, 2019

Awareness of Multiple NodeCIDRs per Node

k8s-ci-robot commented Feb 21, 2019

aojea commented Feb 21, 2019

aojea commented Dec 5, 2018 •

edited

Loading

uablrek commented Dec 5, 2018 •

edited

Loading

aojea Feb 21, 2019 •

edited

Loading