Customizing a Ceph.conf deployed with Rook on Kubernetes
Two options for customizing the parameters of a ceph.conf file deployed by Rook. These customizations could be needed for performance or troubleshooting reasons. In this post, the debug output level of various Ceph services will be modified. Other parameters can be updated using the same process
The override settings are saved in a ConfigMap called rook-config-override within the rook-ceph namespace and applied to the pods during creation. The contents of this map are empty in a default deployment.
To list the available ConfigMaps in the rook-ceph namespace:
$ kubectl -n rook-ceph get ConfigMaps NAME DATA AGE rook-ceph-config 1 106m rook-ceph-mon-endpoints 3 106m rook-config-override 1 106m rook-crush-config 1 105m rook-test-ownerref 0 106m
To list the contents of the rook-config-override ConfigMap and display the contents in YAML format.
$ kubectl -n rook-ceph get ConfigMap rook-config-override -o yaml
apiVersion: v1
data:
config: ""
kind: ConfigMap
metadata:
creationTimestamp: "2019-02-13T21:53:51Z"
name: rook-config-override
namespace: rook-ceph
ownerReferences:
- apiVersion: v1
blockOwnerDeletion: true
kind: CephCluster
name: rook-ceph
uid: cfcb486e-2fd9-11e9-bc12-52540087c4d1
resourceVersion: "164993"
selfLink: /api/v1/namespaces/rook-ceph/configmaps/rook-config-override
uid: d9b48916-2fd9-11e9-bc12-52540087c4d1
Option 1: Setting parameters via live editing
Edit config map using EDITOR. YAML is the default format for editing
$ kubectl -n rook-ceph edit ConfigMap rook-config-override -o yaml
In this example, we will update the config from "" to setting needed to enable debugging of the OSD service. More information on available configuration settings can be found in the references below.
Display updated config map
$ kubectl -n rook-ceph get configmaps rook-config-override -o yaml
apiVersion: v1
data:
config: |
[global]
debug osd = 5
kind: ConfigMap
metadata:
creationTimestamp: "2019-02-13T21:53:51Z"
name: rook-config-override
namespace: rook-ceph
ownerReferences:
- apiVersion: v1
blockOwnerDeletion: true
kind: CephCluster
name: rook-ceph
uid: cfcb486e-2fd9-11e9-bc12-52540087c4d1
resourceVersion: "182611"
selfLink: /api/v1/namespaces/rook-ceph/configmaps/rook-config-override
uid: d9b48916-2fd9-11e9-bc12-52540087c4d1
Restart impacted pod
Once the updated settings are saved, the impacted pods will need to be restarted. In this case, the OSD pods will need to be restarted. Care must be taken to restart the OSD pods only with the cluster is healthy and all data protected.
List running OSD pods:
$ kubectl -n rook-ceph get pods --selector=app=rook-ceph-osd NAME READY STATUS RESTARTS AGE rook-ceph-osd-0-6669bdbf6d-vvsjn 1/1 Running 0 127m rook-ceph-osd-1-556bff7694-8mhq2 1/1 Running 0 127m rook-ceph-osd-2-7db64c88bc-d7sp5 1/1 Running 0 127m
Restart each pod while waiting for the cluster to recover
$ kubectl -n rook-ceph delete pod rook-ceph-osd-0-6669bdbf6d-vvsjn pod "rook-ceph-osd-0-6669bdbf6d-vvsjn" deleted
The cluster will go into a HEALTH_WARN while the OSD pod is being restarted and any data re-syncing occurs. Restart the remaining OSD pods once the ceph cluster has returned to HEALTH_OK
Verify setting
$ kubectl -n rook-ceph exec rook-ceph-osd-0-6669bdbf6d-p5msf -- cat /etc/ceph/ceph.conf | grep "debug osd" debug osd = 5
Option 2: Setting parameters during deployment
Customized ceph.conf parameters can be deployed during cluster deployment by adding a ConfigMap to the cluster.yaml file. These values will be applied to the nodes during initial and follow pod startups. In this example, we are going to set the debug level for the Ceph mon service.
Edit cluster.yaml file
Using your favorite editor, add the following lines to the cluster.yaml file:
---
apiVersion: v1
kind: ConfigMap
data:
config: |
[global]
debug mon = 5
metadata:
name: rook-config-override
namespace: rook-ceph
Deploy cluster
Deploy the ceph cluster as per previous blog posts listed below.
Verify setting
$ kubectl -n rook-ceph get pods --selector=app=rook-ceph-mon NAME READY STATUS RESTARTS AGE rook-ceph-mon-a-79b6c85f89-6rzrw 1/1 Running 0 2m50s rook-ceph-mon-b-57bc4756c9-bmg9n 1/1 Running 0 2m39s rook-ceph-mon-c-6f8bb9598d-9mlcb 1/1 Running 0 2m25s $ kubectl -n rook-ceph exec rook-ceph-mon-a-79b6c85f89-6rzrw -- cat /etc/ceph/ceph.conf | grep "debug mon" debug mon = 5
Notes
- The OSD and Mon pods should be restarted one at a time with the Ceph cluster returning to a HEALTH_OK between pod restarts
- No validation is performed on the override parameters. "mon debug" or a config option which could render the Ceph cluster inoperable can be added to the config override and applied on the next pod restart
References:
No comments:
Post a Comment