Wednesday, October 5, 2022

Using S3 storage on Open Data Framework provided by Ceph-RGW

My previous posts demonstrated how to use CephFS and Ceph RBD backed storage classes as deployed by Open Data Framework on OpenShift. 

This blog post I will demonstrate how to use the Ceph RGW backed storage class from within a pod running on the OpenShift cluster. I will extract the S3 authentication credentials, create a name space, start a pod and demonstrate how to securely interact with the S3 service. 

Background

As stated in the previous blog posts ODF deploys a Ceph cluster within the OCP cluster. The cluster master nodes are used as the Ceph monitor processes, workers are utilized as OSD and the remaining related pods are scheduled on the cluster. 

This post as been verified against OCP and ODF versions 4.11. 

Storage Class Listing

Let's start by verifying the availability of ODF storage classes. The following command will display the available storage classes.

$ oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                    kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  6d
ocs-storagecluster-ceph-nfs   openshift-storage.nfs.csi.ceph.com      Delete          Immediate              false                  6d
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   6d
ocs-storagecluster-ceph-rgw   openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  6d
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   6d
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc         Delete          Immediate              false                  6d

Verify S3 Access

To verify S3 access we will create an object bucket claim, extract the needed authentication and connection information, create a namespace with a configured POD and verify access.

Create ObjectBucketClaim

An ObjectBucketClaim is created against the CephRGW backed storage class.

$ cat objectbucketclaim.yaml 
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-bucket
  namespace: openshift-storage
spec:
  generateBucketName: ceph-bkt
  storageClassName: ocs-storagecluster-ceph-rgw
$ oc apply -f objectbucketclaim.yaml 
objectbucketclaim.objectbucket.io/ceph-bucket created
$ oc get objectbucketclaim -n openshift-storage ceph-bucket -o jsonpath='{.status.phase}{"\n"}'
Bound

Extract Secrets and Connection Information

Once the ObjectBucketClaim is phase is Bound, the S3 secrets and connection information can be extract from the OCP cluster. The following commands will extract the needed information for later usage and print the resulting information to the screen. 

$ export AWS_ACCESS_KEY_ID=`oc get secret -n openshift-storage rook-ceph-object-user-ocs-storagecluster-cephobjectstore-ocs-storagecluster-cephobjectstoreuser -o jsonpath='{.data.AccessKey}'  | base64 -d`
$ export AWS_SECRET_ACCESS_KEY=`oc get secret -n openshift-storage rook-ceph-object-user-ocs-storagecluster-cephobjectstore-ocs-storagecluster-cephobjectstoreuser -o jsonpath='{.data.SecretKey}'  | base64 -d`
$ export AWS_BUCKET=`oc get cm ceph-bucket -n openshift-storage -o jsonpath='{.data.BUCKET_NAME}'`
$ export AWS_HOST=`oc get cm ceph-bucket -n openshift-storage -o jsonpath='{.data.BUCKET_HOST}'`
$ export AWS_PORT=`oc get cm ceph-bucket -n openshift-storage -o jsonpath='{.data.BUCKET_PORT}'`
$ echo ${AWS_ACCESS_KEY_ID}
$ echo ${AWS_SECRET_ACCESS_KEY}
$ echo ${AWS_BUCKET}
$ echo ${AWS_BUCKET}
$ echo ${AWS_HOST}
$ echo ${AWS_PORT}

Update S3 pod yaml

The pod yaml file will be updated to pass the S3 parameters and applied to the cluster
$ cat 74-consume-s3.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: s3-test
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: s3-test
  labels:
    app: rook-s3
spec:
  containers:
  - name: run-pod1
    image: registry.access.redhat.com/ubi8/ubi
    imagePullPolicy: IfNotPresent
    command: ['sh', '-c', 'yum install -y wget python3 && cd /tmp && wget https://downloads.sourceforge.net/project/s3tools/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz && tar -zxf /tmp/s3cmd-2.2.0.tar.gz && ls /tmp && tail -f /dev/null' ]
    env:
    - name: AWS_ACCESS_KEY_ID
      value: VALUE_FROM_ECHO_AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      value: VALUE_FROM_ECHO_AWS_SECRET_ACCESS_KEY
    - name: AWS_HOST
      value: VALUE_FROM_ECHO_AWS_HOST
    - name: AWS_PORT
      value: VALUE_FROM_ECHO_AWS_PORT
    - name: AWS_BUCKET
      value: VALUE_FROM_ECHO_AWS_BUCKET
$ sed -e "s/VALUE_FROM_ECHO_AWS_ACCESS_KEY_ID/${AWS_ACCESS_KEY_ID}/g" \
-e "s/VALUE_FROM_ECHO_AWS_SECRET_ACCESS_KEY/${AWS_SECRET_ACCESS_KEY}/" \
-e "s/VALUE_FROM_ECHO_AWS_HOST/${AWS_HOST}/" \
-e "s/VALUE_FROM_ECHO_AWS_PORT/\"${AWS_PORT}\"/" \
-e "s/VALUE_FROM_ECHO_AWS_BUCKET/${AWS_BUCKET}/" \
-i consume-s3.yaml
$ oc apply -f consume-s3.yaml 
namespace/s3-test created
pod/test-pod created

Wait for the pod to be ready

The pod command line includes the installation commands needed to update the UBI image as needed for this demonstration. A customized image should be used in a production environment. Checking for the python3 command will be a sufficient check to ensure this demonstration pod is configured.

$ oc exec -n s3-test test-pod -- python3 -V
Python 3.6.8

NOTE: This demonstration is using the s3cmd to interface with the bucket. A customized configuration file using the needed S3 parameters and the service CA certificate is copied into to the pod for easier testing. This updating and copying of this file is left out of this blog post.

Verify S3 Environment

We can use the printenv command to verify the setting of the necessary environment credentials. These parameters can be used with a custom image to access the internal Ceph RGW storage
$ oc exec -n s3-test test-pod -- printenv | grep AWS
AWS_BUCKET=ceph-bkt-...ac49
AWS_ACCESS_KEY_ID=FAJ...1HR
AWS_SECRET_ACCESS_KEY=3z3...Vtd
AWS_HOST=rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc
AWS_PORT=443

Verify S3 Access

To verify S3 access, we will simply create and list a new bucket.

$ oc exec -n s3-test test-pod -- python3 /tmp/s3cmd-2.2.0/s3cmd mb s3://validate.${RANDOM}
Bucket 's3://validate.18454/' created
$ oc exec -n s3-test test-pod -- python3 /tmp/s3cmd-2.2.0/s3cmd ls
2022-10-05 20:54  s3://validate.18454

Conclusion

With this blog post we have demonstrated how to consume the internal S3 storage service by creating an ObjectBucketClaim, extracting needed authentication information, deploying a customized pod and running commands. This information can be extended to support the deployment and operation of customized S3 enabled applications. 

Monday, October 3, 2022

Verifying Openshift Data Foundation data access with RBD

This post is similar to my previous post where a CephFS backed storage class was verified with Openshift Data Foundation on Openshift Container Platform. 

 This blog post will demonstrate how to use the ODF provisioned RBD backed storage class by binding a persistent volume claim (PVC), binding to a pod and verifying data access.

Background

ODF deploys a Ceph cluster on top of an existing OCP cluster. Ceph features are provided to the cluster via a variety of storage classes.

This post as been verified against OCP and ODF versions 4.8 through 4.11. 

Storage Class Listing

Let's start by verifying the availability of ODF storage classes. The following command will display the available storage classes.

$ oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                    kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  40h
ocs-storagecluster-ceph-nfs   openshift-storage.nfs.csi.ceph.com      Delete          Immediate              false                  40h
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   40h
ocs-storagecluster-ceph-rgw   openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  40h
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   40h
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc         Delete          Immediate              false                  40h

Verify RBD Storage Class

To verify RBD, we will create a namespace, bind a PVC and create a POD with the PVC attached. Once the POD is running we will copy content into the mount point and verify access.

Create RBD Verification Namespace

 The namespace is created via the following yaml and command

$ cat rbd-ns.yaml
apiVersion: v1                                                                                                                                                                                                     
kind: Namespace                                                                                                                                                                                                    
metadata:                                                                                                                                                                                                          
  labels:                                                                                                                                                                                                          
    openshift.io/cluster-monitoring: "true"                                                                                                                                                                        
  name: rbd-ns                                                                                                                                                                                                     
spec: {}
$ oc apply -f rbd-ns.yaml 
namespace/rbd-ns created

Create RBD Verification PVC

The PVC is created against the RBD backed storage class.

$ cat rbd-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
  namespace: rbd-ns
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ocs-storagecluster-ceph-rbd
$ oc apply -f rbd-pvc.yaml 
persistentvolumeclaim/rbd-pvc created

Create RBD Verification POD

A pod is created in the RBD verification namespace. This pod uses the Red Hat's Apache 2.4 image based on their UBI image. This image was chosen as a convenience and to permit verification of network access to copied data. The image can be replaced with any other suitable image and related tests.

NOTE: This pod description includes setting the fsGroup in the securityContext. This will ensure the mounted volume will be accessible by the internal process. In this pod, httpd runs as UID 1001. 

$ cat rbd-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: rbd-ns
  labels:
    app: rook-ceph-block
spec:
  securityContext:
    fsGroup: 1001
  containers:
  - name: sample-pod1
    image: registry.access.redhat.com/ubi8/httpd-24
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: repo-vol
      mountPath: "/var/www/html/"
  volumes:
  - name: repo-vol
    persistentVolumeClaim:
      claimName: rbd-pvc
$ oc apply -f rbd-pod.yaml 
pod/test-pod created

Copy Static Content Into RBD Verification Pod

A simple HTML file is copied into the pod's document root. This will ensure the attached PVC can be written to and read from bia the web service.

$ $ cat index-block.html 
Here be block dragons
$ oc cp index-rbd.html rbd-ns/test-pod:/var/www/html/index.html

Create and Expose Service from Verification Pod

The RBD pod is exposed via the standard OCP service and route commands.

$ cat rbd-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: block-httpd
  namespace: rbd-ns
spec:
  ports:
  - name: http
    port: 8080
    target: 8080
    protocol: TCP
  selector:
    app: rook-ceph-block
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: block-httpd
  namespace: rbd-ns
spec:
  port:
    targetPort: http
  to:
    kind: Service
    name: block-httpd
$ oc apply -f rbd-service.yaml 
service/block-httpd created
route.route.openshift.io/block-httpd created

Verify RBD Service and Data Access

A simple curl command is used to validate access to the data written to the RBD backed PVC.

$ RBD_HOST=`oc get route -n rbd-ns block-httpd block-httpd -o jsonpath='{.items[0].spec.host}'`
$ curl ${RBD_HOST}
Here be block dragons

RBD Verification Conclusion

As with the CephFS post, this post has demonstrated how to bind a PVC to a RBD backed storage class, attach the PVC to a pod, copy data into the pod and verify access to the data via a network service. Other than setting a securityContext, no major changes are needed from the previous demo.


Friday, September 16, 2022

Verifying Openshift Data Foundation data access with CephFS

This post serves as a follow up to my previous post about deploying Openshift Data Foundation on Openshift Container Platform. 

This blog post will demonstrate how to use the ODF provisioned CepfFS backed storage class by binding a persistent volume claim (PVC), binding to a pod and verifying data access. 

Background

ODF deploys a Ceph cluster on top of an existing OCP cluster. Ceph features are provided to the cluster via a variety of storage classes.

This post as been verified against OCP and ODF versions 4.8 through 4.11. 

Storage Class Listing

Lets start by verifying the availability of ODF storage classes. The following command will display the available storage classes.

$ oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                    kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  40h
ocs-storagecluster-ceph-nfs   openshift-storage.nfs.csi.ceph.com      Delete          Immediate              false                  40h
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   40h
ocs-storagecluster-ceph-rgw   openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  40h
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   40h
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc         Delete          Immediate              false                  40h

Verify CepfFS Storage Class

To verify CephFS, we will create a namespace, bind a PVC and create a POD with the PVC attached. Once the POD is running we will copy content into the mount point and verify access.

Create CephFS Verification Namespace

 The namespace is created via the following yaml and command

$ cat cephfs-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  labels:
    openshift.io/cluster-monitoring: "true"
  name: cephfs-ns
spec: {}
$ oc apply -f cephfs-ns.yaml 
namespace/cephfs-ns created

Create CephFS Verification PVC

The PVC is created against the CephFS backed storage class.

$ cat cephfs-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
  namespace: cephfs-ns
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: ocs-storagecluster-cephfs
$ oc apply -f cephfs-pvc.yaml 
persistentvolumeclaim/cephfs-pvc created

Create CephFS Verification POD

A pod is created in the CephFS verification namespace. This pod uses the Red Hat's Apache 2.4 image based on their UBI image. This image was chosen as a convenience and to permit verification of network access to copied data. The image can be replaced with any other suitable image and related tests. 

$ cat cephfs-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: cephfs-pod
  namespace: cephfs-ns
  labels:
    app: rook-ceph-cephfs
spec:
  containers:
  - name: cephfs-pod1
    image: registry.access.redhat.com/ubi8/httpd-24
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: repo-vol
      mountPath: "/var/www/html/"
  volumes:
  - name: repo-vol
    persistentVolumeClaim:
      claimName: cephfs-pvc
$ oc apply -f cephfs-pod.yaml 
pod/cephfs-pod created

Copy Static Content Into CephFS Verification Pod

A simple HTML file is copied into the pod's document root. This will ensure the attached PVC can be written to and read from bia the web service.

$ cat index-cephfs.html 
Here be cephfs dragons
$ oc cp index-cephfs.html cephfs-ns/test-pod:/var/www/html/index.html

Create and Expose Service from Verification Pod

The CephFS pod is exposed via the standard OCP service and route commands.

 $ cat cephfs-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: cephfs-httpd
  namespace: cephfs-ns
spec:
  ports:
  - name: http
    port: 8080
    target: 8080
    protocol: TCP
  selector:
    app: rook-ceph-cephfs
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: cephfs-httpd
  namespace: cephfs-ns
spec:
  port:
    targetPort: http
  to:
    kind: Service
    name: cephfs-httpd
$ oc apply -f cephfs-service.yaml 
service/cephfs-httpd created
route.route.openshift.io/cephfs-httpd created

Verify CephFS Service and Data Access

A simple curl command is used to validate access to the data written to the CephFS backed PVC

$ CEPHFS_HOST=`oc get route -n cephfs-ns cephfs-httpd cephfs-httpd -o jsonpath='{.items[0].spec.host}'`
$ curl ${CEPHFS_HOST} Here be cephfs dragons

CephFS Verification Conclusion

This post has demonstrated how to bind a PVC to a CephFS backed storage class, attach the PVC to a pod, copy data into the pod and verify access to the data via a network service. 


Wednesday, September 14, 2022

Installing Red Hat Openshift Data Foundation via the command line

Red Hat Openshift Data Foundation (ODF) is an Openshift Container Platform (OCP) add-on for providing Ceph storage to pods running within a deployed cluster. This Red Hat product is based on the upstream Rook project. When deployed, ODF will provide CephFS, RBD and RGW backed storage classes. The Ceph cluster monitors will be installed on the OCP master nodes with the Ceph ODS processes running on the designated OCP worker nodes.

The processes in this post have been validated against OCP and ODF versions 4.8 through 4.11.

A basic understanding of Openshift, Openshift Data Foundation and Ceph are needed to understand this post. This post is not intended to replace official product documentation.

Lab Description

Hardware

This post is developed utilizing a virtual lab consisting of 7 virtual machines. These consist of the following types of hosts:

  • Provision: 2 CPU, 16G RAM, >70G system disk, RHEL 8
  • Master (x3): 6 CPU, 16G RAM, 60G system disk
  • Worker (x3): 14 CPU, 48G RAM, 60G system disk, 10G data disk (x3)

Network

The virtual lab consists of two networks.

  • Provision: Isolated, managed by the Openshift Installer
  • Baremetal: Bridged, managed by an DNS/DHCP server out of scope for this document

Additional Lab Infrastructure

  • A virtual BMC service is used to provide IMPI management of the virtual machines. This service runs on the hypervisor and is reachable from the provision host.
  • A DNS/DHCP virtual server. This service provides DHCP and DNS services to the OCP cluster. The OS provided dhcp and bind servers are used to provide IP address assignment and name resolution.

Cluster Installation

The OCP cluster is deployed using the Openshift Installer Provisioned Infrastructure (IPI) install method. This method uses of a series of Ansible playbooks executed from the provision node to deploy an OCP cluster. The details of the installation is beyond the scope of this post.

Installation Environment
The ODF installation process is performed from the provision host by a user with access to the admin level kubeconfig. This is automatically configured for the user who runs the IPI installer.

ODF Installation Process

Node Verification

Verify availability of nodes. Ensure three master and three worker nodes are available.
$ oc get nodes
NAME       STATUS   ROLES    AGE    VERSION
master-0   Ready    master   110m   v1.24.0+b62823b
master-1   Ready    master   110m   v1.24.0+b62823b
master-2   Ready    master   110m   v1.24.0+b62823b
worker-0   Ready    worker   91m    v1.24.0+b62823b
worker-1   Ready    worker   91m    v1.24.0+b62823b
worker-2   Ready    worker   90m    v1.24.0+b62823b

Verify availability of storage on the worker nodes. At least one disk needs to be available per host but three disks are used in this example. worker-0 is used as an example with worker-1 and worker-2 being similar.
$ oc debug node/worker-0

Pod IP: 192.168.122.35
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   60G  0 disk 
|-sda1   8:1    0    1M  0 part 
|-sda2   8:2    0  127M  0 part 
|-sda3   8:3    0  384M  0 part /boot
`-sda4   8:4    0 59.5G  0 part /sysroot
vda    252:0    0   10G  0 disk 
vdb    252:16   0   10G  0 disk 
vdc    252:32   0   10G  0 disk

ODF Subscription Installation and Verification

The ODF subscription needs to be added to the OCP cluster. This subscription is available via the Openshift Marketplace and is added to the cluster with the following yaml file.
$ cat odf-subscription.yaml
apiVersion: v1
kind: Namespace
metadata:
  labels:
    openshift.io/cluster-monitoring: "true"
  name: openshift-storage
spec: {}
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
  generateName: openshift-storage-
  name: openshift-storage-24mhn
  namespace: openshift-storage
spec:
  targetNamespaces:
  - openshift-storage
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator
  namespace: openshift-storage
spec:
  channel: stable-4.11
  installPlanApproval: Automatic
  name: odf-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: odf-operator.v4.11.0
  
$ oc apply -f 22-ocs-sub.yaml
namespace/openshift-storage created
operatorgroup.operators.coreos.com/openshift-storage-24mhn created
subscription.operators.coreos.com/odf-operator created
The installation can be monitored with the following command. The operator is fully installed when "Succeed" is returned as the phase.
$ watch -d "oc get csv -n openshift-storage -l operators.coreos.com/ocs-operator.openshift-storage='' -o jsonpath='{.items[0].status.phase}'"

The openshift-storage pods can be listed with the following command. All pods should be in "Running" or "Completed" state. No pods should in a "Pending" or "Error" state.
$ oc get pods -n openshift-storage

Next we will enable the ODF console in the OCP web console. This will allow the modification and monitoring of the ODF cluster via the OCP web console.
$ oc patch console.operator cluster -n openshift-storage --type json -p '[{"op": "add", "path": "/spec/plugins", "value": ["odf-console"]}]'

Local Storage Subscription Installation and Verification

The Local Storage subscription needs to be installed. This is to support the worker nodes service as Ceph OSD servers. This subscription is available via the Openshift Marketplace and is added to the cluster with the following yaml file.

$ cat local-subscription.yaml
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: openshift-local-storage
    olm.operatorgroup.uid/1b9690c6-f7d4-47f4-8046-b389b44b0612: ""
  name: openshift-local-storage
spec:
  finalizers:
  - kubernetes
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: LocalVolume.v1.local.storage.openshift.io,LocalVolumeDiscovery.v1alpha1.local.storage.openshift.io,LocalVolumeDiscoveryResult.v1alpha1.local.storage.openshift.io,LocalVolumeSet.v1alpha1.local.storage.openshift.io
  name: openshift-local-storage-operator
  namespace: openshift-local-storage
spec:
  targetNamespaces:
  - openshift-local-storage
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/local-storage-operator.openshift-local-storage: ""
  name: local-storage-operator
  namespace: openshift-local-storage
spec:
  channel: stable
  installPlanApproval: Automatic
  name: local-storage-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

The installation of the local storage operator can be monitored with a command similar to the ODF operator installation. The operator is fully installed when "Succeed" is returned as the phase.

$ watch -d "`oc get csv -n openshift-local-storage -l operators.coreos.com/local-storage-operator.openshift-local-storage="" -o=jsonpath='{.items[0].status.phase}'"

Again, the status of the pods can be checked. All pods in the openshift-local-storage namespace should be in either the "Running" or "Completed" state.

Configure Local Storage Disks

Once the ODF and local storage operators are configured, the local disks can be configured and make ready for the storage cluster deployment. This consists of two elements:
  1. Labeling nodes which are used for OSD storage
  2. Configuring the disks for usage.
Each worker node will need to be labeled with the "cluster.ocs.openshift.io/openshift-storage=" tag. The command for worker-0 is given as an example below. worker-1 and worker-2 are labeled with a similar command.

$ oc label nodes worker-0 cluster.ocs.openshift.io/openshift-storage=''

The local storage disks are provisioned with following yaml file

$ cat label-disks.yaml
apiVersion: local.storage.openshift.io/v1alpha1
kind: LocalVolumeSet
metadata:
  name: localpv
  namespace: openshift-local-storage
spec:
  deviceInclusionSpec:
    deviceTypes: #Unused disks and partitions meeting these requirements are used
    - disk
    - part
    minSize: 1Gi
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions: #Nodes with this label are used
      - key: cluster.ocs.openshift.io/openshift-storage
        operator: Exists
  storageClassName: localblock
  tolerations:
  - effect: NoSchedule
    key: node.ocs.openshift.io/storage
    operator: Equal
    value: "true"
  volumeMode: Block
$ oc apply -f label-disks.yaml
localvolumeset.local.storage.openshift.io/localpv created

The apply command will start the configuration process of the local volume disks and their related persistent volumes (PV). The progress of this process can be monitored with the following command. The process is complete when the status returns "True"

$ watch -d "oc get localvolumeset -n openshift-local-storage localpv -o jsonpath='{.status.conditions[0].status}'"

This command will provide more information while monitoring the configuration of the local disks.
$ watch -d "oc get LocalVolumeSet -A;echo; oc get pods -n openshift-local-storage; echo ; oc get pv"

There should be one PV for each disk which matches the "deviceInclusionSpec". In this lab, a total of 9 PVs should be available.

$ oc get pv | wc -l

Configure ODF Storage Cluster

The ODF storage cluster can be deployed with the following yaml file. The storageDeviceSets count should be a OSD sets configured. In this lab, 9 disks are configure for OSD usage and the storageDeviceSet count is set to 3. This value will need to be adjusted for the local environment.

$ cat deploy-storagecluster.yamlapiVersion: ocs.openshift.io/v1
kind: StorageCluster
metadata:
  name: ocs-storagecluster
  namespace: openshift-storage
spec:
  arbiter: {}
  encryption:
    kms: {}
  externalStorage: {}
  flexibleScaling: true
  resources:
    mds:
      limits:
        cpu: "3"
        memory: "8Gi"
      requests:
        cpu: "3"
        memory: "8Gi"
  monDataDirHostPath: /var/lib/rook
  managedResources:
    cephBlockPools:
      reconcileStrategy: manage   # <-- Default value is manage
    cephConfig: {}
    cephFilesystems: {}
    cephObjectStoreUsers: {}
    cephObjectStores: {}
  multiCloudGateway:
    reconcileStrategy: manage   # <-- Default value is manage
  storageDeviceSets:
  - count: 3  # <-- Modify count to desired value. For each set of 3 disks increment the count by 1.
    dataPVCTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: "100Mi"
        storageClassName: localblock
        volumeMode: Block
    name: ocs-deviceset
    placement: {}
    portable: false
    replica: 3
    resources:
      limits:
        cpu: "2"
        memory: "5Gi"
      requests:
        cpu: "2"
        memory: "5Gi"
$ oc apply -f deploy-storagecluster.yaml
storagecluster.ocs.openshift.io/ocs-storagecluster created

The storage cluster deployment start the configuration process of the Ceph monitors and OSD processes on the node. A simple monitoring of this process can be monitored with the below command. The storage cluster is deployed and ready for usage once the phase returns "Ready".

$ watch -d "oc get storagecluster -n openshift-storage ocs-storagecluster -o jsonpath={'.status.phase'}"

A more through monitoring of the configuration process can be accomplished with this command

$ watch -d "oc get storagecluster -n openshift-storage; echo ; oc get cephcluster -n openshift-storage; echo; oc get noobaa -n openshift-storage ; echo; oc get pods -n openshift-storage|tail -n 20"

Checking Available Storage Classes

Multiple Ceph storage classes are available at the completion of the storage cluster deployment. This can be viewed with the following command

$ oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                    kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  5m54s
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   2m4s
ocs-storagecluster-ceph-rgw   openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  5m36s
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   2m4s
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc         Delete          Immediate              false                  14s

PVs can now be created against this storage classes with their eventual consumption by PVCs and pods.

Deploying Ceph Toolbox Pod

A pod containing the Ceph command line tools can be added to the deploy cluster with the below command.
$ oc patch OCSInitialization ocsinit -n openshift-storage --type json --patch  '[{ "op": "replace", "path": "/spec/enableCephTools", "value": true }]'
ocsinitialization.ocs.openshift.io/ocsinit patched

The Ceph tool box can be used with the following command
$ $ oc rsh -n openshift-storage `oc get pods -n openshift-storage -l app=rook-ceph-tools -o jsonpath='{.items[0].metadata.name}'`
sh-4.4$ ceph status
  cluster:
    id:     c680a945-60bb-4da3-b419-64f017884b8f
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c (age 17m)
    mgr: a(active, since 17m)
    mds: 1/1 daemons up, 1 hot standby
    osd: 9 osds: 9 up (since 16m), 9 in (since 17m)
    rgw: 1 daemon active (1 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   12 pools, 449 pgs
    objects: 362 objects, 134 MiB
    usage:   437 MiB used, 90 GiB / 90 GiB avail
    pgs:     449 active+clean
 
  io:
    client:   3.4 KiB/s rd, 80 KiB/s wr, 4 op/s rd, 9 op/s wr

sh-4.4$ ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME                          STATUS  REWEIGHT  PRI-AFF
-1         0.08817  root default                                                
-7         0.02939      host worker-0-osp-example-com                           
 0    hdd  0.00980          osd.0                          up   1.00000  1.00000
 5    hdd  0.00980          osd.5                          up   1.00000  1.00000
 8    hdd  0.00980          osd.8                          up   1.00000  1.00000
-3         0.02939      host worker-1-osp-example-com                           
 1    hdd  0.00980          osd.1                          up   1.00000  1.00000
 3    hdd  0.00980          osd.3                          up   1.00000  1.00000
 7    hdd  0.00980          osd.7                          up   1.00000  1.00000
-5         0.02939      host worker-2-osp-example-com                           
 2    hdd  0.00980          osd.2                          up   1.00000  1.00000
 4    hdd  0.00980          osd.4                          up   1.00000  1.00000
 6    hdd  0.00980          osd.6                          up   1.00000  1.00000

Conclusion and Followup

This post has detailed the operations needed to configure OCP for ODF, configuration of the local disks and the deployment of the storage cluster. Additional posts will be made providing examples of how to consume the storage provide by ODF.

References:

Sunday, December 22, 2019

A Torrid Tail of Too Many Circles - Continued

Previously

In my earlier post, I covered how I attempted to create a Moire pattern on my pyportal screen. While the implementation was pretty simple, there were issues with excessive memory usage and dreadful refresh speed.

Moving to a faster speed


The first implementation used a lot of system memory because each rendered circle consisted of a TileGrid, Bitmap and Palette. The goal for the second implementation was to reduce the amount of memory used by using one Bitmap per group of circles. I first needed to refresh my memory on how to find the points on a circle. A quick trip down the Google results lead me to this page. I was able to quickly implement a draw_circle function to a circle to a provided Bitmap. The center and radius of the of the circle are used as the basis of the function and a SMOOTHNESS value is used to determine the size of each calculated step. The draw_circle is wrapped in two loops to generate each set of circles into a single bitmap. Parameters are provided for the number and spacing of the generated circles and how much time to wait between each set of calculations.

Journey Review


The new circle function is much faster and generates a pleasing set of circles. Memory usage is much less since only one TileGrid, Bitmap and Palette is used for the display. However, it takes quite a few seconds to generate a new set of circles in the Bitmap. However, the random placement of the circles results in display updates which can be displeasing. This issue was rectified by adding a 'wobble' to the previous generation's centers. The math still takes a long time but the updates are more pleasing.

Where to go next


Is it possible to make the updates faster?

Monday, December 16, 2019

A Torrid Tail of Too Many Circles

Where it all began....


A long time ago, I received a Adafuit pyportal from one of their mystery boxes. This device has spent most of its time collecting dust or displaying the default demonstration code. I decided to replicate some coding from the 80s and see if I could display Moire Patterns on the pyportal screen. To this end, I have started the journey of drawing too many circles on the screen.

I decided to use the CircuitPython language to code my implementations. I could use a variety of editors (vim, Mu and Atom) and run the code by simply updating the code.py file on the device. My development environment is Linux based so I eventually fell into old development habits and used vim for development, cp for deployment and minicom to monitor code execution.

And so it begins


The Adafruit maintains a list CircuitPython modules available (200!). The main modules used in this project are displayio and Display Shapes. The former is responsible for managing the display and bitmaps while the latter is used to generate shapes. Adafruit has a great information page on how to use the displayio framework. The shapes library returns a TileGrid for each shape generated.

Implementation 


For this implementation, I created a display group per set of circles and populated it with TileGrids created by Circle(). The center of the circles is randomly generated and each ring radius is incrementally larger than the previous. The code will sleep for a few seconds, delete the two display groups and regenerate a new set of circles.

Issues


There are a number of issues with this implementation

  • Memory: Each circles consists of a TileGrid, a bitmap sized to hold the new shape and a palette. This results in 60 total circles able to be drawn.
  • Display Speed: The updating of the display is dreadfully slow. It takes about 20 seconds to update the display. Maybe the Group is not flattened before being displayed. 

Moving forward


Obviously this is not an optimal solution. I continued working on this code based to make speed improvements but that is a tale for another time. 



Friday, April 5, 2019

Developing and testing patches for Rook

Backgroud


The purpose of this posting is to detail the steps needed to develop and test Rook code using the provided CI scripts. My previous blog posting did not use the CI scripting provided by rook and may not provide sufficient testing coverage to pass the upstream CI.

Pre-built infrastructure


Test Hosts Configuration
This CI development environment uses four (4) libvirt virtual machines as test hosts. The hardware configurations of the virtual machines remains the same as in the previous blog posts. These machines are built from the same clone image and configured with as follows
  • Ubuntu 16 (some commands may change for Ubuntu 18)
  • Local user with admin rights, updated to use sudo with no password
  • Pre-populated SSH host keys and StrictHostchecking set to no in the .ssh/config file
  • /etc/hosts file pre-populated with the host name and IP of each test host. 
  • Statically defined IP address as per this blog post

Hostnames and IPs

The virtual machines in this environment will be configured with the following hostnames and IPs
192.168.122.31 kube-ub-master
192.168.122.32 kube-ub-node1
192.168.122.33 kube-ub-node2
192.168.122.34 kube-ub-node3

Configuring the master server

The Rook CI scripts are designed to run from the kubernetes master host. These configuration steps are ran as the configured admin user on that host.

Update DNS resolver and hostname

On Ubuntu 16, the resolv.conf file is managed by systemd. This configuration can result in the coredns container not starting due to a lookup loop.
sudo sed -i -e 's/dns=.*/dns=default/' /etc/NetworkManager/NetworkManager.conf
sudo systemctl disable systemd-resolved.service
sudo systemctl stop  systemd-resolved.service
sudo rm /etc/resolv.conf
echo nameserver 8.8.8.8 | sudo tee /etc/resolv.conf
echo kube-ub-master | sudo tee /etc/hostname

Install go 1.11

The CI requires go version 1.11 to be available on the system in order to build the local images and to run the go based test infrastructure.
wget https://dl.google.com/go/go1.11.6.linux-amd64.tar.gz
cd /usr/local
sudo tar -zxf ~kschinck/go1.11.6.linux-amd64.tar.gz
cd ~
echo export PATH=/usr/local/go/bin:\$PATH | tee -a ~/.bashrc
source .bashrc

Install and configure docker

Configure docker to use the registry on the master node
sudo apt-get install -y docker.io git ansible curl
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "insecure-registries" : ["192.168.122.31:5000"]
}
EOF
sudo systemctl start docker
sudo systemctl enable docker

Add user to the docker group

To run the CI scripts, the docker user needs to able to run docker commands without using sudo.
sudo usermod -a -G docker myuser
The new group assignment can be picked up by either logging out and back in or running the newgrp command.

Start the docker registry container

The docker registry container needs to be running in order to host the locally build versions of the Rook images.
docker run -d -p 5000:5000 --restart=always --name registry registry:2

Verify Registry Access

The curl command can be used to verify access to the local registry.

curl http://192.168.122.31:5000/v2/_catalog
{"repositories":[]}

Configure Worker Nodes

The worker nodes need to be configure similar to the master node with the exception of needing golang support added.
for HOST in kube-ub-node1 kube-ub-node2 kube-ub-node3
do
  cat<<EOF | ssh $HOST
sudo sed -i -e 's/dns=.*/dns=default/' /etc/NetworkManager/NetworkManager.conf
sudo systemctl disable systemd-resolved.service
sudo systemctl stop  systemd-resolved.service
sudo rm /etc/resolv.conf
echo nameserver 8.8.8.8 | sudo tee /etc/resolv.conf
echo $HOST | sudo tee /etc/hostname
sudo apt-get install -y docker.io 
cat <<EOG | sudo tee /etc/docker/daemon.json
{
  "insecure-registries" : ["192.168.122.31:5000"]
}
EOG
sudo systemctl start docker
sudo systemctl enable docker
echo ‘export GOPATH=${HOME}/go’ | tee -a ~/.bashrc
source .bashrc
echo $PATH
echo $GOPATH
mkdir -p ${GOPATH}/src/github.com/rook
cd ${GOPATH}/src/github.com/rook
git clone http://github.com/myuser/rook.git
cd rook
git checkout mypatch
EOF
done

Download Source Code

At this point we are ready to prepare for and download the forked git repository.The repository is configured according to the Rook development guidelines
These commands will prepare the needed directory path, clone the git repo, switch to the development branch and build the images.
cat <<EOF>>~/.bashrc
export GOPATH=${HOME}/go
EOF
export GOPATH=${HOME}/go
mkdir -p ${GOPATH}/src/github.com/rook
cd ${GOPATH}/src/github.com/rook
git clone http://github.com/myuser/rook.git
cd rook
git checkout mypatch
make

Tag and push built image

The make command will build and upload a rook/ceph image. In order to use this image, this image will need to be tagged and pushed back to the local registry.
docker tag `docker images | grep "rook/ceph" | awk '{print $1":"$2}'` 192.168.122.31:5000/rook/ceph:latest
docker push 192.168.122.31:5000/rook/ceph:latest
The local test framework files need to be update to use the new image. The following command will update the image reference to the local registry.
sed -i -e 's/image: rook.*$/image: 192.168.122.31:5000\/rook\/ceph:latest/' ./tests/framework/installer/ceph_manifests.go

Deploy master  node using CI script

cd tests/scripts
./kubeadm.sh up
The end of this command will display the kubectl command used to configure additional worker nodes. This needs to be copied and used in below worker node configuration setp

Deploy worker nodes using CI script

This command will add each worker host to the kubernetes cluster.
for HOST in kube-ub-node1 kube-ub-node2 kube-ub-node3
do
  cat<<EOF | ssh $HOST
  cd cd ${GOPATH}/src/github.com/rook/rook/tests/scripts
  ./kubeadm.sh install node TEXT_TAKEN_FROM_THE_MASTER_INSTALL_OUTPUT
EOF


Setup the kubectl command for debugging


mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

At this point, the admin will be able to use the kubectl command to interact with the deployed kubernetes cluster.

Run CI Test

Rook uses the provide test framework of go to run the validation commands. The test command is ran from the top of the rook repository directory. It will deploy a variety of configurations and validate the operation of the cluster. Output is saved under the _output directory.
This command will run the standard tests called SmokeSuite and save the output to a file in the /tmp directory.


time go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration 2>&1 | tee /tmp/integration.log

The results can be reviewed from /tmp/integration.log file.

The kubernetes cluster should be in a clean state if the tests finish successfully.

Repeating the CI Execution


If an operator is to be update updated, the previous uploaded images should be deleted from the local repository prior to making updates, rebuilding the image, tagging and pushing.

If the CI configuration is updated, it is sufficient to rerun the go test command line.