Make Ceph-CSI topology aware.

[humblec]

Describe the feature you'd like to have

CSI provide a mechanism to provision volumes based on topology. its called as topology aware provisioning. This functionality is available in upstream ( Beta @1.14) and this is the feature request for adopting the same in ceph-csi.

What is the value to the end user? (why is it a priority?)

The user will be able to provision volumes based on failover domains, DR..etc .

Next AI: A design PR for the same.

[mmgaggle]

This rook-ceph feature request would be a consumer of this logic:

rook/rook#3639

[mmgaggle]

Another potential consumer:

#559

[humblec]

@poornimag would you like to volunteer for this RFE implementation?

[mmgaggle]

I might, if I had experience writing golang.

That said, I'd be happy to work up a design document for someone that would do a much better job.

[Madhu-1]

I might, if I had experience writing golang.

That said, I'd be happy to work up a design document for someone that would do a much better job.

@mmgaggle sounds good.

[humblec]

@poornimag this is a very important functionality we are looking for at v2.0.0 . Assigning this to you. Please let me know if you need any help in getting this done.

[humblec]

#557

[obnoxxx]

This issue does not really describe what is the desired effect. Please elaborate.

As far as I understand it, topology aware provisioning is not very relevant at all to what we're doing with ceph on kubernetes:

Kubernetes topology aware provisioning is a feature that is essentially resulting in a delayed creation of the PV, it is delayed until a pod using this PVC is scheduled. The motivation is to make sure that the PVC is created "near" the consuming pod so that the pod can actually reach the storage. Typical example would be an EBS PV on AWS. EBS are only available within an AZ. So topology aware provisioning makes sure that the EBS disk is only created in the right AZ once the consuming pod is scheduled. (If the EBS is created in one AZ and then the pod is scheduled in a different one, the pod can't reach the PV...).

Now Ceph is usually not restricted to an AZ. First of all, we are typically creating backend OSD disks in multiple AZs. And even if we aren't, the ceph volumes, being networked storage are even reachable across zone boundaries. So I really don't see where topology aware provisioning would help us in any way.

Please explain.

@humblec @Madhu-1 @nixpanic @mmgaggle

[dillaman]

Now Ceph is usually not restricted to an AZ. First of all, we are typically creating backend OSD disks in multiple AZs. And even if we aren't, the ceph volumes, being networked storage are even reachable across zone boundaries. So I really don't see where topology aware provisioning would help us in any way.

... reachable at an additional $ cost and speed cost.

[mmgaggle]

@obnoxxx take a look at #559, it's the primary use case for this sort of thing.

And as @dillaman mentioned, inter-az traffic is billed at a premium and the latency is higher.

So if CSI was toplogy aware, and we closed #559, then I could have a pool for each AZ, a pool that spans AZs, and users could select from storage classes -

csi-standard-rbd/cephfs
csi-regional-rbd/cephfs

standard favors latency and cost
regional favors durability and availability

[humblec]

@ShyamsundarR Can you please udpate where we are with this PR or design ?

[ShyamsundarR]

@ShyamsundarR Can you please udpate where we are with this PR or design ?

This is not for 2.0.0 release, at it would take time (3-6 weeks IMO). This was updated out of band to the team.

Current design thoughts/doc can be found here: https://gist.github.com/ShyamsundarR/8ff154555f137e71e5c0a80e224c61a2 which I intend to complete this week and post as a proposal PR.

[humblec]

@ShyamsundarR sure and thanks for the update. May be we can post the design proposal here for more reviews .

[Madhu-1]

moving this one out of release-v3.0.0 release

[Madhu-1]

@humblec / @ShyamsundarR can this be done in 3.1.0? or do we need to move it out to next release?

[ShyamsundarR]

@humblec / @ShyamsundarR can this be done in 3.1.0? or do we need to move it out to next release?

This will be post 3.1.0

[humblec]

Topology Aware provisioning has been moved to GA in upstream kuberenetes and requires K8s api server >= 1.17 . Considering this feature has been came to GA state, its good to continue our efforts on this and hopefully move topology feature support in Ceph CSI to Beta/GA. I would like to propose this for next release ( 3.2.0).
@ShyamsundarR can you please share your thoughts/plans on taking this forward.. ?

[ShyamsundarR]

Most of the work for RBD is complete. CephFS was blocked on the need for an info API, which is now present can hence I will work on enabling the same for CephFS as well. Cannot ensure this will be for 3.2.0 release.

As an aside, the use-case for read optimized topology usage, assumed we could takeover the topology domain value to a surviving domain on failure, will not work owing to key/value being a 1:1 rather than 1:n. This would hence require support for weighted topology constraints in kubernetes (and possibly CSI as well).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.