Using S3 storage on Open Data Framework provided by Ceph-RGW

My previous posts demonstrated how to use CephFS and Ceph RBD backed storage classes as deployed by Open Data Framework on OpenShift. 

This blog post I will demonstrate how to use the Ceph RGW backed storage class from within a pod running on the OpenShift cluster. I will extract the S3 authentication credentials, create a name space, start a pod and demonstrate how to securely interact with the S3 service. 

Background

As stated in the previous blog posts ODF deploys a Ceph cluster within the OCP cluster. The cluster master nodes are used as the Ceph monitor processes, workers are utilized as OSD and the remaining related pods are scheduled on the cluster. 

This post as been verified against OCP and ODF versions 4.11. 

Storage Class Listing

Let's start by verifying the availability of ODF storage classes. The following command will display the available storage classes.

$ oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                    kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  6d
ocs-storagecluster-ceph-nfs   openshift-storage.nfs.csi.ceph.com      Delete          Immediate              false                  6d
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   6d
ocs-storagecluster-ceph-rgw   openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  6d
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   6d
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc         Delete          Immediate              false                  6d

Verify S3 Access

To verify S3 access we will create an object bucket claim, extract the needed authentication and connection information, create a namespace with a configured POD and verify access.

Create ObjectBucketClaim

An ObjectBucketClaim is created against the CephRGW backed storage class.

$ cat objectbucketclaim.yaml 
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-bucket
  namespace: openshift-storage
spec:
  generateBucketName: ceph-bkt
  storageClassName: ocs-storagecluster-ceph-rgw
$ oc apply -f objectbucketclaim.yaml 
objectbucketclaim.objectbucket.io/ceph-bucket created
$ oc get objectbucketclaim -n openshift-storage ceph-bucket -o jsonpath='{.status.phase}{"\n"}'
Bound

Extract Secrets and Connection Information

Once the ObjectBucketClaim is phase is Bound, the S3 secrets and connection information can be extract from the OCP cluster. The following commands will extract the needed information for later usage and print the resulting information to the screen. 

$ export AWS_ACCESS_KEY_ID=`oc get secret -n openshift-storage rook-ceph-object-user-ocs-storagecluster-cephobjectstore-ocs-storagecluster-cephobjectstoreuser -o jsonpath='{.data.AccessKey}'  | base64 -d`
$ export AWS_SECRET_ACCESS_KEY=`oc get secret -n openshift-storage rook-ceph-object-user-ocs-storagecluster-cephobjectstore-ocs-storagecluster-cephobjectstoreuser -o jsonpath='{.data.SecretKey}'  | base64 -d`
$ export AWS_BUCKET=`oc get cm ceph-bucket -n openshift-storage -o jsonpath='{.data.BUCKET_NAME}'`
$ export AWS_HOST=`oc get cm ceph-bucket -n openshift-storage -o jsonpath='{.data.BUCKET_HOST}'`
$ export AWS_PORT=`oc get cm ceph-bucket -n openshift-storage -o jsonpath='{.data.BUCKET_PORT}'`
$ echo ${AWS_ACCESS_KEY_ID}
$ echo ${AWS_SECRET_ACCESS_KEY}
$ echo ${AWS_BUCKET}
$ echo ${AWS_BUCKET}
$ echo ${AWS_HOST}
$ echo ${AWS_PORT}

Update S3 pod yaml

The pod yaml file will be updated to pass the S3 parameters and applied to the cluster

$ cat 74-consume-s3.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: s3-test
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: s3-test
  labels:
    app: rook-s3
spec:
  containers:
  - name: run-pod1
    image: registry.access.redhat.com/ubi8/ubi
    imagePullPolicy: IfNotPresent
    command: ['sh', '-c', 'yum install -y wget python3 && cd /tmp && wget https://downloads.sourceforge.net/project/s3tools/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz && tar -zxf /tmp/s3cmd-2.2.0.tar.gz && ls /tmp && tail -f /dev/null' ]
    env:
    - name: AWS_ACCESS_KEY_ID
      value: VALUE_FROM_ECHO_AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      value: VALUE_FROM_ECHO_AWS_SECRET_ACCESS_KEY
    - name: AWS_HOST
      value: VALUE_FROM_ECHO_AWS_HOST
    - name: AWS_PORT
      value: VALUE_FROM_ECHO_AWS_PORT
    - name: AWS_BUCKET
      value: VALUE_FROM_ECHO_AWS_BUCKET
$ sed -e "s/VALUE_FROM_ECHO_AWS_ACCESS_KEY_ID/${AWS_ACCESS_KEY_ID}/g" \
-e "s/VALUE_FROM_ECHO_AWS_SECRET_ACCESS_KEY/${AWS_SECRET_ACCESS_KEY}/" \
-e "s/VALUE_FROM_ECHO_AWS_HOST/${AWS_HOST}/" \
-e "s/VALUE_FROM_ECHO_AWS_PORT/\"${AWS_PORT}\"/" \
-e "s/VALUE_FROM_ECHO_AWS_BUCKET/${AWS_BUCKET}/" \
-i consume-s3.yaml
$ oc apply -f consume-s3.yaml 
namespace/s3-test created
pod/test-pod created

Wait for the pod to be ready

The pod command line includes the installation commands needed to update the UBI image as needed for this demonstration. A customized image should be used in a production environment. Checking for the python3 command will be a sufficient check to ensure this demonstration pod is configured.

$ oc exec -n s3-test test-pod -- python3 -V
Python 3.6.8

NOTE: This demonstration is using the s3cmd to interface with the bucket. A customized configuration file using the needed S3 parameters and the service CA certificate is copied into to the pod for easier testing. This updating and copying of this file is left out of this blog post.

Verify S3 Environment

We can use the printenv command to verify the setting of the necessary environment credentials. These parameters can be used with a custom image to access the internal Ceph RGW storage

$ oc exec -n s3-test test-pod -- printenv | grep AWS
AWS_BUCKET=ceph-bkt-...ac49
AWS_ACCESS_KEY_ID=FAJ...1HR
AWS_SECRET_ACCESS_KEY=3z3...Vtd
AWS_HOST=rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc
AWS_PORT=443

Verify S3 Access

To verify S3 access, we will simply create and list a new bucket.

$ oc exec -n s3-test test-pod -- python3 /tmp/s3cmd-2.2.0/s3cmd mb s3://validate.${RANDOM}
Bucket 's3://validate.18454/' created
$ oc exec -n s3-test test-pod -- python3 /tmp/s3cmd-2.2.0/s3cmd ls
2022-10-05 20:54  s3://validate.18454

Conclusion

With this blog post we have demonstrated how to consume the internal S3 storage service by creating an ObjectBucketClaim, extracting needed authentication information, deploying a customized pod and running commands. This information can be extended to support the deployment and operation of customized S3 enabled applications. 

Verifying Openshift Data Foundation data access with RBD

This post is similar to my previous post where a CephFS backed storage class was verified with Openshift Data Foundation on Openshift Container Platform. 

 This blog post will demonstrate how to use the ODF provisioned RBD backed storage class by binding a persistent volume claim (PVC), binding to a pod and verifying data access.

Background

ODF deploys a Ceph cluster on top of an existing OCP cluster. Ceph features are provided to the cluster via a variety of storage classes.

This post as been verified against OCP and ODF versions 4.8 through 4.11. 

Storage Class Listing

Let's start by verifying the availability of ODF storage classes. The following command will display the available storage classes.

$ oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                    kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  40h
ocs-storagecluster-ceph-nfs   openshift-storage.nfs.csi.ceph.com      Delete          Immediate              false                  40h
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   40h
ocs-storagecluster-ceph-rgw   openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  40h
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   40h
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc         Delete          Immediate              false                  40h

Verify RBD Storage Class

To verify RBD, we will create a namespace, bind a PVC and create a POD with the PVC attached. Once the POD is running we will copy content into the mount point and verify access.

Create RBD Verification Namespace

 The namespace is created via the following yaml and command

$ cat rbd-ns.yaml
apiVersion: v1                                                                                                                                                                                                     
kind: Namespace                                                                                                                                                                                                    
metadata:                                                                                                                                                                                                          
  labels:                                                                                                                                                                                                          
    openshift.io/cluster-monitoring: "true"                                                                                                                                                                        
  name: rbd-ns                                                                                                                                                                                                     
spec: {}
$ oc apply -f rbd-ns.yaml 
namespace/rbd-ns created

Create RBD Verification PVC

The PVC is created against the RBD backed storage class.

$ cat rbd-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
  namespace: rbd-ns
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ocs-storagecluster-ceph-rbd

$ oc apply -f rbd-pvc.yaml 
persistentvolumeclaim/rbd-pvc created

Create RBD Verification POD

A pod is created in the RBD verification namespace. This pod uses the Red Hat's Apache 2.4 image based on their UBI image. This image was chosen as a convenience and to permit verification of network access to copied data. The image can be replaced with any other suitable image and related tests.

NOTE: This pod description includes setting the fsGroup in the securityContext. This will ensure the mounted volume will be accessible by the internal process. In this pod, httpd runs as UID 1001. 

$ cat rbd-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: rbd-ns
  labels:
    app: rook-ceph-block
spec:
  securityContext:
    fsGroup: 1001
  containers:
  - name: sample-pod1
    image: registry.access.redhat.com/ubi8/httpd-24
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: repo-vol
      mountPath: "/var/www/html/"
  volumes:
  - name: repo-vol
    persistentVolumeClaim:
      claimName: rbd-pvc

$ oc apply -f rbd-pod.yaml 
pod/test-pod created

Copy Static Content Into RBD Verification Pod

A simple HTML file is copied into the pod's document root. This will ensure the attached PVC can be written to and read from bia the web service.

$ $ cat index-block.html 
Here be block dragons
$ oc cp index-rbd.html rbd-ns/test-pod:/var/www/html/index.html

Create and Expose Service from Verification Pod

The RBD pod is exposed via the standard OCP service and route commands.

$ cat rbd-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: block-httpd
  namespace: rbd-ns
spec:
  ports:
  - name: http
    port: 8080
    target: 8080
    protocol: TCP
  selector:
    app: rook-ceph-block
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: block-httpd
  namespace: rbd-ns
spec:
  port:
    targetPort: http
  to:
    kind: Service
    name: block-httpd
$ oc apply -f rbd-service.yaml 
service/block-httpd created
route.route.openshift.io/block-httpd created

Verify RBD Service and Data Access

A simple curl command is used to validate access to the data written to the RBD backed PVC.

$ RBD_HOST=`oc get route -n rbd-ns block-httpd block-httpd -o jsonpath='{.items[0].spec.host}'`
$ curl ${RBD_HOST}
Here be block dragons

RBD Verification Conclusion

As with the CephFS post, this post has demonstrated how to bind a PVC to a RBD backed storage class, attach the PVC to a pod, copy data into the pod and verify access to the data via a network service. Other than setting a securityContext, no major changes are needed from the previous demo.