Add new topic for expanding PV size #8124
Conversation
ghost
added
branch/enterprise-3.9
peer-review-needed
openshift-ci-robot
added
the
size/M
| ** If the volume is of a type that does not require a file system resize, then `pvc.status.capacity` is immediately updated to reflect the new size. This would conclude the volume expand operation, and `pvc.Status.Conditions` is updated to `Ready: True`. | ||
| ** If the volume is of a type that requires a file system resize, then a file system resize is performed on kubelet. | ||
| * If the volume plugin is of type that cannot do resizing of attached volumes (such as Cinder), then `ExpandVolumeDevice` can return an error by checking for volume status with its own API (such as by making an Openstack Cinder API call in this case). The controller continues trying to resize the volume until it is successful. | ||
| * To consider cases of missed PVC update events, an additional loop reconciles bound PVCs with PVs. This additional loop loops through all PVCs, matches `pvc.spec.resources.requests` with `pv.spec.capacity`, and adds the PVC in the volume_expand_controller's work queue if `pv.spec.capacity` is less than `pvc.spec.resources.requests`. |
There was a problem hiding this comment.
s/pvc.spec.resources.requests/pvc.spec.resources.requests.storage/
There was a problem hiding this comment.
This is shop talk, i dont think its relevant to the end user.
|
@openshift/team-documentation - PTAL. |
| == Overview | ||
|
|
||
| You can increase the size of the persistent volumes (PVs) that your pods are using. To do this, you can update the persistent volume claim (PVC) to request a new size. The controller then applies the change to the PV, which is bound to the PVC. | ||
|
|
There was a problem hiding this comment.
Then, the controller applies
(also there may be an extra space between "the" and "controller")
There was a problem hiding this comment.
I wouldn't mention any of the 'internal workings' of this, it will confuse the user.. (no controller, no attributes on PVC objects, nothing on how it works.. this should only show the UX..
|
|
||
| A new controller called `volume_expand_controller` listens for PVC size expansion requests and takes action as needed. The new controller does the following: | ||
|
|
||
| * Watches for PVC update requests and adds the PVC to the controller's work queue if an increase in volume size was requested. Once the PVC is added to the controller's work queue, `- pvc.Status.Conditions` is updated to `ResizeStarted: True`. |
There was a problem hiding this comment.
Would boil this down to say that the PVC status will reflect that the resize is happening (the rest is not end user friendly)
| * Watches for PVC update requests and adds the PVC to the controller's work queue if an increase in volume size was requested. Once the PVC is added to the controller's work queue, `- pvc.Status.Conditions` is updated to `ResizeStarted: True`. | ||
| * For unbound or pending PVCs,` - resize ` triggers no action in `volume_expand_controller`. | ||
| * If `pv.Spec.Capacity already` is greater or equal than the requested size, then no action is performed by the controller. | ||
| * A separate goroutine reads the work queue and performs a corresponding volume resize operation. If there is a resize operation in progress for the same volume, then the resize request is pending and retries once the previous resize request is complete. |
There was a problem hiding this comment.
goroutine? Should this be 2 separate words? Not sure what this is.
There was a problem hiding this comment.
this should not include discussion of the internals except where they are visible to the user (no mention of controller, goroutine, etc..)
| * If `pv.Spec.Capacity already` is greater or equal than the requested size, then no action is performed by the controller. | ||
| * A separate goroutine reads the work queue and performs a corresponding volume resize operation. If there is a resize operation in progress for the same volume, then the resize request is pending and retries once the previous resize request is complete. | ||
| * Controller resizing is level-based rather than edge-based. If there is more than one pending resize request for the same PVC, then new resize requests for the same PVC replace older pending requests. | ||
| * Resize is performed via the volume plugin interface, which is executed inside a goroutine spawned by `operation_exectutor`. |
There was a problem hiding this comment.
shouldn't mention 'goroutine" and i'd clarrify level-based / edge-based.. thats too shop-talky for end user.
| * A separate goroutine reads the work queue and performs a corresponding volume resize operation. If there is a resize operation in progress for the same volume, then the resize request is pending and retries once the previous resize request is complete. | ||
| * Controller resizing is level-based rather than edge-based. If there is more than one pending resize request for the same PVC, then new resize requests for the same PVC replace older pending requests. | ||
| * Resize is performed via the volume plugin interface, which is executed inside a goroutine spawned by `operation_exectutor`. | ||
| * A new plugin interface called `volume.Expander` is added to the volume plugin interface. The expander interface defines if the volume requires a file system resize. |
There was a problem hiding this comment.
s/plugin/plug-in (a few instances here and across the page)
| * The controller calls to expand the PVC. | ||
| * Once the volume expansion is successful, the volume is marked as expanded and the new size is reflected or updated in `pv.spec.capacity`. Any errors are reported as events on the PVC object. | ||
| * If the resize operation fails, then `pvc.Status.Conditions` is updated with `ResizeFailed: True`. The corresponding error is also added to the condition field. | ||
| * Depending on the volume type: |
There was a problem hiding this comment.
Should the content under "Depending on the volume type" be taken out of the nested list format? I guess it just depends on how it looks when built.
There was a problem hiding this comment.
Would only mention the user visible parts here, not controller.. would also re-phrase to remove shop talk.
| * Once the volume expansion is successful, the volume is marked as expanded and the new size is reflected or updated in `pv.spec.capacity`. Any errors are reported as events on the PVC object. | ||
| * If the resize operation fails, then `pvc.Status.Conditions` is updated with `ResizeFailed: True`. The corresponding error is also added to the condition field. | ||
| * Depending on the volume type: | ||
| ** If the volume is of a type that does not require a file system resize, then `pvc.status.capacity` is immediately updated to reflect the new size. This would conclude the volume expand operation, and `pvc.Status.Conditions` is updated to `Ready: True`. |
There was a problem hiding this comment.
possible extra space before `pvc.Status.Conditions
There was a problem hiding this comment.
Would simplify to only UX and omit the flow/details.
| * Depending on the volume type: | ||
| ** If the volume is of a type that does not require a file system resize, then `pvc.status.capacity` is immediately updated to reflect the new size. This would conclude the volume expand operation, and `pvc.Status.Conditions` is updated to `Ready: True`. | ||
| ** If the volume is of a type that requires a file system resize, then a file system resize is performed on kubelet. | ||
| * If the volume plugin is of type that cannot do resizing of attached volumes (such as Cinder), then `ExpandVolumeDevice` can return an error by checking for volume status with its own API (such as by making an Openstack Cinder API call in this case). The controller continues trying to resize the volume until it is successful. |
There was a problem hiding this comment.
Would remove all shop talk here too. "Once a pod is schedule, the node (avoid kubelet in OSE docs) performs the file system resize is sufficient.
| ** If the volume is of a type that does not require a file system resize, then `pvc.status.capacity` is immediately updated to reflect the new size. This would conclude the volume expand operation, and `pvc.Status.Conditions` is updated to `Ready: True`. | ||
| ** If the volume is of a type that requires a file system resize, then a file system resize is performed on kubelet. | ||
| * If the volume plugin is of type that cannot do resizing of attached volumes (such as Cinder), then `ExpandVolumeDevice` can return an error by checking for volume status with its own API (such as by making an Openstack Cinder API call in this case). The controller continues trying to resize the volume until it is successful. | ||
| * To consider cases of missed PVC update events, an additional loop reconciles bound PVCs with PVs. This additional loop loops through all PVCs, matches `pvc.spec.resources.requests` with `pv.spec.capacity`, and adds the PVC in the volume_expand_controller's work queue if `pv.spec.capacity` is less than `pvc.spec.resources.requests`. |
There was a problem hiding this comment.
s/loops/cycles (maybe?)
`volume_expand_controller`'s
| ** If the volume is of a type that requires a file system resize, then a file system resize is performed on kubelet. | ||
| * If the volume plugin is of type that cannot do resizing of attached volumes (such as Cinder), then `ExpandVolumeDevice` can return an error by checking for volume status with its own API (such as by making an Openstack Cinder API call in this case). The controller continues trying to resize the volume until it is successful. | ||
| * To consider cases of missed PVC update events, an additional loop reconciles bound PVCs with PVs. This additional loop loops through all PVCs, matches `pvc.spec.resources.requests` with `pv.spec.capacity`, and adds the PVC in the volume_expand_controller's work queue if `pv.spec.capacity` is less than `pvc.spec.resources.requests`. | ||
| * There are additional checks in the controller that expand the PV size. These checks ensure that you do not make volume plugin API calls that can reduce the size of the PV. |
There was a problem hiding this comment.
more shop talk, mention of API or how the controller works should be avoided.
| * glusterfs | ||
| * rbd | ||
|
|
||
| You can allow expanding persistent volume claims by setting the `ExpandPersistentVolumes` feature gate to true. You should also enable the `PersistentVolumeClaimResize` admission plugin to perform additional validations of volumes that can be resized. |
There was a problem hiding this comment.
This is in shop-talk form. It should explain specifically how the feature gate works in OSE (where to add the flag etc..)
| You can allow expanding persistent volume claims by setting the `ExpandPersistentVolumes` feature gate to true. You should also enable the `PersistentVolumeClaimResize` admission plugin to perform additional validations of volumes that can be resized. | ||
|
|
||
| Once the `PersistentVolumeClaimResize` admission plug-in has been turned on, resizing is only allowed for storage classes whose `allowVolumeExpansion` field is set to true, as shown below. | ||
|
|
| allowVolumeExpansion: true | ||
| ---- | ||
|
|
||
| Once both the feature gate and the aforementioned admission plug-in are turned on, users can request a larger volume for their `PersistentVolumeClaim` by editing the claim and requesting a larger size. This triggers expansion of the volume that is backing the underlying `PersistentVolume`. A new `PersistentVolume` is not created to satisfy the claim. Kubernetes instead attempts to resize the existing volume. |
There was a problem hiding this comment.
remove "aforementioned"
s/users can/you can
s/their/your
ahardin-rh
added
peer-review-done
|
This topic needs to be added to the _topic_map.yml file so that it builds as part of the doc set. That will also allow you to run |
| [[expanding-pvcs]] | ||
| == Expanding Persistent Volumes Claims | ||
|
|
||
| Kubernetes 1.8 added Alpha support for expanding persistent volumes. In Kubernetes 1.9, the following volume types support expanding PVCs: |
There was a problem hiding this comment.
I am still not convinced why should Openshift documentation talk about Kubernetes version. We are introducing Volume expansion as a tech. preview in 3.9, that is what we should document. cc @childsb
There was a problem hiding this comment.
+1 with @gnufied .... Everything should be in the context of OSE.
|
|
||
| Once both the feature gate and the aforementioned admission plug-in are turned on, users can request a larger volume for their `PersistentVolumeClaim` by editing the claim and requesting a larger size. This triggers expansion of the volume that is backing the underlying `PersistentVolume`. A new `PersistentVolume` is not created to satisfy the claim. Kubernetes instead attempts to resize the existing volume. | ||
|
|
||
| To expand volumes containing a file system, file system resizing is only performed when a new pod is started using the `PersistentVolumeClaim` in `ReadWrite` mode. If a volume being expanded is used in a pod or deployment, you need to delete and recreate the pod for file system resizing to take place. Also, file system resizing is only supported for XFS, Ext3, and Ext4 file system types. |
There was a problem hiding this comment.
Lets add a new section for Expanding volume types with file system. Such as - EBS, GCE-PD, Cinder and RBD. That section should clearly document 2 step process.
Step 1: Edit your PVC to have new size. User should wait for PVC to have condition PersistentVolumeClaimFileSystemResizePending
Step 2: User can describe the PVC and when the PVC has condition PersistentVolumeClaimFileSystemResizePending then user can (re-)start a pod to finish file system resize of volume on node.
|
|
||
| For volume types that require both file system expansion and a volume plugin-based modification, growing PVs is a two-step process. For volume types that only require a volume plugin-based API call, growing PVs is a one-step process. | ||
|
|
||
| A new controller called `volume_expand_controller` listens for PVC size expansion requests and takes action as needed. The new controller does the following: |
There was a problem hiding this comment.
It appears that we are documenting internals of Volume Expand Controller here. There is no harm in doing that but the top section of this documentation should start with "How to Expand PVCs". Not with how volume expansion works.
There was a problem hiding this comment.
We shouldn't have this, only the UX.. it will confuse users.
|
|
||
| You can increase the size of the persistent volumes (PVs) that your pods are using. To do this, you can update the persistent volume claim (PVC) to request a new size. The controller then applies the change to the PV, which is bound to the PVC. | ||
|
|
||
| For volume types that require both file system expansion and a volume plugin-based modification, growing PVs is a two-step process. For volume types that only require a volume plugin-based API call, growing PVs is a one-step process. |
There was a problem hiding this comment.
should only show the UX (user wont know what volume plugin based api call means)
|
I left lots of line item comments, but I dont think these docs have the correct audience... The upstream Kube docs are aimed at platform vendors and deeply technical tire kickers.. Some of the value-add of OSE is making kube user-friendly and much of that is the human-readable/non-geek docs The end-user shouldn't be exposed to any internals, only the UX they will have.. Docs shouldn't mention of kube, or any implementation details, only how to enable & use the feature, or how errors are reported to user. Components should be framed in the context of OSE and use end-user friendly terms.. |
|
@childsb Should we just use the information in the release notes draft? |
|
@tmorriso-rh Yes that's probably good. Maybe add where the error on failure is visible. @gnufied does this feature need to be enabled in 3.9 to use, or is on by default? Are there any extra deployment/setup steps? |
|
Now, according the content of the PR, seems it should be linked to here: https://github.com/openshift/openshift-docs/tree/master/install_config/persistent_storage |
|
@aheslin Ali, can you review this PR to determine where this content should be placed? It's currently targeted for the developer guide. If you need additional details, please see the comments on this Trello card: https://trello.com/c/GdqaM5R3/531-resize-perform-cloudprovider-resize-in-controller |
openshift-ci-robot
added
size/L
|
looks good to me from technical point of view. Assigning to @ahardin-rh for further approval. |
|
/lgtm |
|
/assign @ahardin-rh |
|
Bump. @aheslin Ali, can you review this PR to determine where this content should be placed? It's currently targeted for the developer guide. If you need additional details, please see the comments on this Trello card: https://trello.com/c/GdqaM5R3/531-resize-perform-cloudprovider-resize-in-controller @ahardin-rh Ashley, can you review the link to the Admission Controllers topic? Should this be for 3.7 or 3.9? |
| Apart from that, {product-title} administrators must enable the | ||
| `ExpandPersistentVolumes` feature flag and turn on the | ||
| `PersistentVolumeClaimResize` admission controller. Refer to | ||
| xref:../architecture/additional_concepts/admission_controllers.adoc[Admission Controllers] |
There was a problem hiding this comment.
@tmorriso-rh This path should be:
xref:../architecture/additional_concepts/admission_controllers.adoc#architecture-additional-concepts-admission-controllers[Admission Controllers]
It is also good to run asciibinder build and test the links in your preview build
|
New changes are detected. LGTM label has been removed. |
ghost
added
peer-review-done
|
@gnufied Before merging, can you please verify if this also applies to 3.10, dedicated, or online? Thanks. |
|
@tmorriso-rh it should apply to 3.10 as well. And since online is mostly managed cluster, I am not sure yet if we are going to enable resizing feature in the cluster. I would for now, abstain from making this part of openshift online docs. |
ghost
merged commit
|
/cherrypick enterprise-3.9 |
|
@tmorriso-rh: new pull request created: #8602 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cherrypick enterprise-3.10 |
|
@tmorriso-rh: new pull request created: #8603 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@gnufied @qinpingli PTAL and update as needed. This content was pulled from kubernetes/community#657 and https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims.
Trello card: https://trello.com/c/e0Awnt9f/771-document-resize-perform-cloudprovider-resize-in-controller