Troubleshooting permission errors when enabling persistence

Objective

This guide will teach you how to autonomously fix an OVHcloud Managed Kubernetes Service when Permission Errors are encountered during Helm Chart deployment or deployment creation.

Issue explanation

Several Helm charts are updated with some security hardening best practices.
Using a non-root container, for example, is a new rule to follow for security reason.
But a main drawback to use non-root containers is related to mounting persistent volumes in these containers.

Indeed, processes running inside these containers do not have the necessary privileges to modify the ownership of the existing filesystem in a volume.

A solution is to use the SecurityContext provided by Kubernetes to automatically modify the ownership of the attached volumes and to provide a StorageClass which supports modifying the volume's filesystem.
However, the StorageClass used by default for the "OVHcloud Managed Kubernetes Service" didn't support the possibility to modify the volume's filesystem.

In the following documentation, we are providing some patches, prior to an update of our service.

Observed behaviors

Some pods can be marked in CrashLoopBackOff status a few seconds/minutes after being scheduled, due to insufficient write access to persistent volumes.

Example of error logs:

mariadb 18:13:27.78 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.

Provided solutions

1. We (the OVHcloud Managed Kubernetes Service team) are working on a patch to be released in early 2022. So, if you are not impacted by the issue, please do not update your Helm Chart deployment (as only recent Helm Charts seem to make use of security context, which causes this issue) and wait until a new version of your managed service is available through the OVHcloud console.

2. You are using the Bitnami Helm Charts and you want to be able to quickly fix this behavior without waiting for our patch. You can follow the instructions described in this documentation: https://docs.bitnami.com/kubernetes/faq/troubleshooting/troubleshooting-helm-chart-issues/

3. **This solution is not recommended if you don't know what you are doing and only works with clusters above `1.20` version.** You are impacted by this issue but your Helm Chart provider didn't offer a proper solution and you can't wait for our official patch.

If you are in this case, please follow these instructions at your own risk:

• Verify what is the StorageClass that you are using by default (generally the csi-cinder-high-speed):

 get storageclasses.storage.k8s.io 
NAME                              PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-cinder-classic                cinder.csi.openstack.org   Delete          Immediate           true                   12d
csi-cinder-high-speed (default)   cinder.csi.openstack.org   Delete          Immediate           true                   12d
csi-cinder-high-speed-gen2        cinder.csi.openstack.org   Delete          Immediate           true                   5h11m
csi-cinder-high-speed-luks        cinder.csi.openstack.org   Delete          Immediate           true                   5h11m
csi-cinder-classic-luks           cinder.csi.openstack.org   Delete          Immediate           true                   5h11m
csi-cinder-high-speed-gen2-luks   cinder.csi.openstack.org   Delete          Immediate           true                   5h11m

• Delete the concerned StorageClass that you are using by default

$ kubectl delete storageclasses.storage.k8s.io csi-cinder-high-speed 
storageclass.storage.k8s.io "csi-cinder-high-speed" deleted

• Create a new StorageClass with the required fix

$ kubectl apply -f https://raw.githubusercontent.com/ovh/docs/develop/pages/public_cloud/containers_orchestration/managed_kubernetes/fix-persistent-volumes-permissions/files/fixed-cinder-high-speed-storage-class.yaml 
storageclass.storage.k8s.io/csi-cinder-high-speed created

• Delete the concerned Helm Chart

For example with the Helm Chart bitnami/wordpress which is concerned by this behavior:

$ helm uninstall my-first-k8s-wordpress

And don't forget to verify if concerned PersistentVolumeClaim and PersistentVolume have been deleted before reinstalling the Helm Chart:

$ kubectl get persistentvolumeclaims -A
$ kubectl get persistentvolumes 

• Reinstall the concerned Helm Chart or deployment

For example with the Helm Chart bitnami/wordpress which is concerned by this behavior:

$ helm install my-first-k8s-wordpress bitnami/wordpress

You can see that the pods are now up and running, which means that the permission errors related to the persistentVolumes are now fixed.

$ kubectl get pods
NAME                                        READY   STATUS             RESTARTS        AGE
my-first-k8s-wordpress-2-8554886b4b-l8tnq   1/1     Running            0               21m
my-first-k8s-wordpress-2-mariadb-0          1/1     Running            0               21m

Go further

To learn more about using your Kubernetes cluster the practical way, we invite you to look at our OVHcloud Managed Kubernetes documentation.

• If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for assisting you on your specific use case of your project.

• Join our community of users.