GKE: Resizing Persistent Volumes

In this post I’ll show an example of how to resize a persistent volume (PV) that is in use by a StatefulSet using a VolumeClaimTemplate on a Google Kubernetes Engine (GKE) Autopilot cluster and not lose the data on it.

Cheat Sheet:

# core commands:
# check status of pv:
kubectl get pvc <pvc-name> -o yaml
kubectl get pv | grep 

# update kubernetes pvc
kubectl patch pvc <pvc-name> <edits...>
kubectl edit pvc <pvc-name>

What you’ll need:

• GCP Account with appropriate IAM roles / permissions
• GKE Cluster (I’m using Autopilot)
• Kubernetes deployment that uses a stateful set and volume template
• Access to GCP Portal (Compute Engine >> Disks)
• Helm installation
• Kubectl access (helpful)

References:

Useful information about this topic at kubernetes.io

Demo chart found here.

Useful stackoverflow post.

Create a Helm Chart:

For this example let’s create a helm chart to which we’ll make edits along the way.

helm create demo

Create a StorageClass:

Let’s create a new StorageClass for our demo so that we can specify some important parameters.

Create demo >> templates >> sc.yaml and put the following code inside of it.

############## VB
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: demo-rwo
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-balanced
allowVolumeExpansion: true
############## VB

Notable items above include:

• name: A custom name that we’ll reference in deployment.yaml
• provisioner: We’re using a GKE CSI driver to allocate our storage.
• type: We’re using a balanced disk
• allowVolumeExpansion: If the provisioner accepts expansion, set this to true

Update deployment.yaml:

Now open demo >> templates >> deployment.yaml and make the following edits:

In the above code, reference the # VB comments. Key modifications to the defaul helm chart are:

• kind: Update the kind: to a StatefulSet
• serviceName: StatefulSets require a serviceName
• storageClassName: We’re going to use the StorageClass we defined above
• volumeClaimTemplates: Define a template for persisting the PV even after a helm uninstall

Now, we need to configure the container to use the storage configured above.

In the above code block add the volumeMounts block to the container and specify a where the volume should be mounted inside of the container.

Now, save the changes and open a terminal window within the demo/ folder.

Install the Chart:

Install the helm chart as follows:

helm install demo .

You should see the following output (disregard any autopilot resource sizing warnings):

Verify that the demo service is running, via kubectl get pods.

Let’s now look at the persistent volume (PV) and the persistent volume claim (PVC) that kubernetes created based on our helm chart modifications.

Here we can see that kubernetes allocated a 1Gi PV of type demo-rwo.

Next we’re going to bash into the container and verify that we see the PV mounted where we expect it to be.

Above we can see the 1Gi PV on /mnt. Next we’ll write a file to the mount.

Pro Tip: At this point, we can exit the container and uninstall the helm chart from the cluster, but our content will remain in the mounted PV so long as we do not delete the PV or PVC.

Resize the Volume:

As of this writing, Helm does not currently have a means to resize a PVC in a volumeTemplate via helm uninstall / helm install; however, there are options for accomplishing the goal and retaining data.

Helm Chart with a kubectl Patch or Edit:

Let’s resize the PV to 2GB. First, we need to update our helm chart’s deployment.yaml as shown below. Save your edits.

Update the PVC:

Get the PVC name via:

kubectl get pvc | grep demo

As we saw above, the PVC persists after a helm uninstall. There’s currently not an option for helm to update this PVC automatically (looks like a future option). In the meantime here are two options that we’ve found.

kubectl patch PVC:

Now that we have the PVC name, and we’ve updated (but not yet deployed) our helm chart to reflect 2Gi, we need to patch the PVC as follows:

kubectl patch pvc my-data-demo-0 -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'

kubectl edit pvc:

This approach uses the edit option within kubectl which opens a terminal editor for you to manually make changes.

kubectl edit pvc my-data-demo-0

Redeploy the Chart:

Whichever approach you used (patch or edit), it’s now time to redeploy the helm chart so that kubernetes can auto resize the volume for us.

helm uninstall demo --wait;
helm install demo demo/;

Verify the Increased PVC:

Let’s verify the newly sized PVC via one or both of the following:

kubectl get pvc | grep demo
# or
kubectl get pvc my-data-demo-0 -o yaml

The PVC should now show 2Gi. Let’s verify it within the container:

Our PV is now 2 Gi and our sample data is intact!

Acknowledgements:

Thanks to Joe and Joey for their help as we looked into statefulsets and volumeTemplates!